Skip to content
This repository has been archived by the owner on Jun 10, 2019. It is now read-only.

Elasticsearch: Some index files failed to download. #476

Closed
kevin-olbrich opened this issue May 18, 2018 · 7 comments
Closed

Elasticsearch: Some index files failed to download. #476

kevin-olbrich opened this issue May 18, 2018 · 7 comments

Comments

@kevin-olbrich
Copy link
Contributor

Hi!

Executing: chroot /target/ee083aa0/root apt-get update
Get:1 http://packages.icinga.com/debian icinga-stretch InRelease [6,969 B]
Get:2 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:3 http://ftp.de.debian.org/debian stretch InRelease
Get:4 http://ftp.de.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:5 http://packages.icinga.com/debian icinga-stretch/main Sources [9,008 B]
Get:6 http://packages.icinga.com/debian icinga-stretch/main amd64 Packages [28.4 kB]
Get:7 http://ftp.de.debian.org/debian stretch Release [118 kB]
Get:8 http://ftp.de.debian.org/debian stretch Release.gpg [2,434 B]
Get:9 http://security.debian.org stretch/updates/main amd64 Packages [351 kB]
Get:10 http://security.debian.org stretch/updates/main Translation-en [163 kB]
Get:11 http://security.debian.org stretch/updates/contrib amd64 Packages [1,776 B]
Get:12 http://security.debian.org stretch/updates/contrib Translation-en [1,759 B]
Get:13 http://security.debian.org stretch/updates/non-free amd64 Packages [1,268 B]
Get:14 http://security.debian.org stretch/updates/non-free Translation-en [481 B]
Get:15 http://ftp.de.debian.org/debian stretch-updates/main amd64 Packages [11.0 kB]
Get:16 http://ftp.de.debian.org/debian stretch-updates/main Translation-en [6,782 B]
Ign:17 https://artifacts.elastic.co/packages/6.x/apt stable InRelease
Get:18 http://ftp.de.debian.org/debian stretch/main amd64 Packages [7,122 kB]
Ign:19 https://artifacts.elastic.co/packages/6.x/apt stable Release
Ign:20 https://artifacts.elastic.co/packages/6.x/apt stable/main all Packages
Get:21 http://ftp.de.debian.org/debian stretch/main Translation-en [5,394 kB]
Get:22 http://ftp.de.debian.org/debian stretch/contrib amd64 Packages [50.9 kB]
Get:23 http://ftp.de.debian.org/debian stretch/contrib Translation-en [45.9 kB]
Get:24 http://ftp.de.debian.org/debian stretch/non-free amd64 Packages [78.0 kB]
Get:25 http://ftp.de.debian.org/debian stretch/non-free Translation-en [80.1 kB]
Ign:26 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
Ign:27 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en
Ign:28 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en_US
Ign:20 https://artifacts.elastic.co/packages/6.x/apt stable/main all Packages
Ign:26 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
Ign:27 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en
Ign:28 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en_US
Ign:20 https://artifacts.elastic.co/packages/6.x/apt stable/main all Packages
Ign:26 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
Ign:27 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en
Ign:28 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en_US
Ign:20 https://artifacts.elastic.co/packages/6.x/apt stable/main all Packages
Ign:26 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
Ign:27 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en
Ign:28 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en_US
Ign:20 https://artifacts.elastic.co/packages/6.x/apt stable/main all Packages
Ign:26 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
Ign:27 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en
Ign:28 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en_US
Ign:20 https://artifacts.elastic.co/packages/6.x/apt stable/main all Packages
Err:26 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
Ign:27 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en
Ign:28 https://artifacts.elastic.co/packages/6.x/apt stable/main Translation-en_US
Fetched 13.7 MB in 3s (4,283 kB/s)
Reading package lists...
W: The repository 'https://artifacts.elastic.co/packages/6.x/apt stable Release' does not have a Release file.
E: Failed to fetch https://artifacts.elastic.co/packages/6.x/apt/dists/stable/main/binary-amd64/Packages
E: Some index files failed to download. They have been ignored, or old ones used instead.
Command 'chroot /target/ee083aa0/root apt-get update' returned non-zero exit status 100

This is an issue because Elastic repos use a dynamic generation method for repo files (not file based).

Adding the same repo to an already installed system works flawless.

Any ideas?

Kind regards,
Kevin
@kevin-olbrich
Copy link
Contributor Author

https://artifacts.elastic.co/packages/6.x/apt/dists/stable/main/binary-amd64/Packages
This file exists. Wget from the building server is also working correctly.

@kevin-olbrich
Copy link
Contributor Author 

  sources:
    # mariadb:
      # - deb http://mirror2.hs-esslingen.de/mariadb/repo/10.2/debian stretch main
      # - deb-src http://mirror2.hs-esslingen.de/mariadb/repo/10.2/debian stretch main
    icinga:
      - deb http://packages.icinga.com/debian icinga-stretch main
      - deb-src http://packages.icinga.com/debian icinga-stretch main
    elastic:
      - deb https://artifacts.elastic.co/packages/6.x/apt stable main

@andsens
Copy link
Owner

andsens commented May 18, 2018

This is an issue because Elastic repos use a dynamic generation method for repo files (not file based).
Adding the same repo to an already installed system works flawless.

I don't get that part, could you elaborate? What is different during bootstrapping that causes this error?

@kevin-olbrich
Copy link
Contributor Author

kevin-olbrich commented May 18, 2018 •
edited
Loading

I don't get that part, could you elaborate? What is different during bootstrapping that causes this error?

The Elastic Repo is generated on the fly, it's not static as most repos are (files exposed with index).
This is just an idea, not sure if this is the root of the problem. I just never had issues with regular repos.

I am not sure why this issue exists. The same repo works fine on installed systems but failed during bootstrap.

@kevin-olbrich
Copy link
Contributor Author

kevin-olbrich commented Jun 13, 2018 •
edited
Loading

I got the same problem with a repo from microsoft:

W: The repository 'https://packages.microsoft.com/debian/9/prod stretch Release' does not have a Release file.
E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages
E: Some index files failed to download. They have been ignored, or old ones used instead.
Command 'chroot /target/abf44eb5/root apt-get update' returned non-zero exit status 100

The build system does not use or define any proxy. Other build jobs which download files to the local machine also have no problem in receiving files over https.

Is there any known issue in using https repos? "apt-transport-https" is listed under "include_packages".

Full log using an alternative repo url from microsoft:

Installing trusted keys
Writing aptitude sources to disk
Configuring apt to always clean everything out when it's done
Updating the package cache
Executing: chroot /target/2a72a9d9/root apt-get update
Get:1 http://packages.icinga.com/debian icinga-stretch InRelease [6,969 B]
Ign:2 http://ftp.de.debian.org/debian stretch InRelease
Get:3 http://security.debian.org stretch/updates InRelease [94.3 kB]
Get:4 http://ftp.de.debian.org/debian stretch-updates InRelease [91.0 kB]
Ign:5 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch InRelease
Ign:6 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch Release
Ign:7 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main all Packages

Ign:8 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main amd64 Packages
Ign:9 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en_US
Ign:10 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en
Get:11 http://ftp.de.debian.org/debian stretch Release [118 kB]
Ign:7 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main all Packages
Ign:8 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main amd64 Packages
Ign:9 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en_US
Get:12 http://packages.icinga.com/debian icinga-stretch/main Sources [9,008 B]
Ign:10 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en
Get:13 http://ftp.de.debian.org/debian stretch Release.gpg [2,434 B]
Ign:7 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main all Packages
Get:14 http://packages.icinga.com/debian icinga-stretch/main amd64 Packages [28.4 kB]
Ign:8 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main amd64 Packages
Ign:9 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en_US
Ign:10 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en
Ign:7 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main all Packages
Ign:8 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main amd64 Packages
Ign:9 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en_US
Get:15 http://security.debian.org stretch/updates/main amd64 Packages [360 kB]
Ign:10 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en
Ign:7 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main all Packages
Ign:8 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main amd64 Packages
Ign:9 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en_US
Ign:10 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en
Ign:7 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main all Packages
Err:8 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main amd64 Packages

Get:16 http://security.debian.org stretch/updates/main Translation-en [169 kB]
Ign:9 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en_US
Ign:10 https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch/main Translation-en
Get:17 http://security.debian.org stretch/updates/contrib amd64 Packages [1,776 B]
Get:18 http://security.debian.org stretch/updates/contrib Translation-en [1,759 B]
Get:19 http://security.debian.org stretch/updates/non-free amd64 Packages [1,268 B]
Get:20 http://security.debian.org stretch/updates/non-free Translation-en [481 B]
Get:21 http://ftp.de.debian.org/debian stretch-updates/main amd64 Packages [11.0 kB]
Get:22 http://ftp.de.debian.org/debian stretch-updates/main Translation-en [6,782 B]
Get:23 http://ftp.de.debian.org/debian stretch/main amd64 Packages [7,122 kB]
Get:24 http://ftp.de.debian.org/debian stretch/main Translation-en [5,394 kB]
Get:25 http://ftp.de.debian.org/debian stretch/contrib amd64 Packages [50.9 kB]
Get:26 http://ftp.de.debian.org/debian stretch/contrib Translation-en [45.9 kB]
Get:27 http://ftp.de.debian.org/debian stretch/non-free amd64 Packages [78.0 kB]
Get:28 http://ftp.de.debian.org/debian stretch/non-free Translation-en [80.1 kB]

Fetched 13.7 MB in 3s (4,189 kB/s)
Reading package lists...
W: The repository 'https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch Release' does not have a Release file.
E: Failed to fetch https://packages.microsoft.com/repos/microsoft-debian-stretch-prod/dists/stretch/main/binary-amd64/Packages
E: Some index files failed to download. They have been ignored, or old ones used instead.
Command 'chroot /target/2a72a9d9/root apt-get update' returned non-zero exit status 100
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/example-appserver_build_trunk/workspace/bvz-git/bootstrapvz/base/main.py", line 111, in run
    tasklist.run(info=bootstrap_info, dry_run=dry_run)
  File "/var/lib/jenkins/jobs/example-appserver_build_trunk/workspace/bvz-git/bootstrapvz/base/tasklist.py", line 44, in run
    task.run(info)
  File "/var/lib/jenkins/jobs/example-appserver_build_trunk/workspace/bvz-git/bootstrapvz/common/tasks/apt.py", line 200, in run
    'apt-get', 'update'])
  File "/var/lib/jenkins/jobs/example-appserver_build_trunk/workspace/bvz-git/bootstrapvz/common/tools.py", line 14, in log_check_call
    raise e
CalledProcessError: Command 'chroot /target/2a72a9d9/root apt-get update' returned non-zero exit status 100
Rolling back

Tasklist:
	bootstrapvz.plugins.minimize_size.tasks.mounts.RemoveFolderMounts
	bootstrapvz.common.tasks.filesystem.UnmountRoot
	bootstrapvz.common.tasks.partitioning.UnmapPartitions
	bootstrapvz.common.tasks.volume.Detach
	bootstrapvz.common.tasks.filesystem.DeleteMountDir
	bootstrapvz.common.tasks.volume.Delete
	bootstrapvz.common.tasks.workspace.DeleteWorkspace

This works perfectly:
root@slde0005:~# curl -s https://packages.microsoft.com/repos/microsoft-debian-stretch-prod/dists/stretch/main/binary-amd64/Packages

The packages are also installed on the build machine, which is also Debian stretch.

@kevin-olbrich
Copy link
Contributor Author

Found the issue!

bootstrapper:
  workspace: /target
  mirror: http://ftp.de.debian.org/debian/
  include_packages:
   - apt-transport-https
   - ca-certificates

The package "ca-certificates" must be installed for "apt-transport-https" to work properly.
Apt could be more verbose in this case (like CA validation failed).

Should I add these details to the docs?

@kevin-olbrich kevin-olbrich mentioned this issue Jun 13, 2018
Merged
@andsens
Copy link
Owner

andsens commented Jun 13, 2018

Wow. Yeah, agreed on the unhelpful error message. PR merged and closing issue. Good job figuring it out :-)

@andsens andsens closed this as completed Jun 13, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andsens @kevin-olbrich