Page MenuHomeVyOS Platform

VyOS build failing due to repo.saltstack.com
Closed, ResolvedPublicBUG

Description

Seeing the following when building our fork or upstream:

Err:7 https://repo.saltstack.com/py3/debian/10/amd64/archive/3003 buster Release
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 65.8.192.71 443]
...
Reading package lists...
W: https://repo.saltstack.com/py3/debian/10/amd64/archive/3002.5/dists/buster/InRelease: No system certificates available. Try installing ca-certificates.
W: https://repo.saltstack.com/py3/debian/10/amd64/archive/3002.5/dists/buster/Release: No system certificates available. Try installing ca-certificates.
E: The repository 'https://repo.saltstack.com/py3/debian/10/amd64/archive/3002.5 buster Release' does not have a Release file.
E: An unexpected failure occurred, exiting...
P: Begin unmounting filesystems...
P: Saving caches...

with an immediate fail after.

Details

Difficulty level
Unknown (require assessment)
Version
1.4
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Event Timeline

sempervictus created this task.
sempervictus created this object in space S1 VyOS Public.

I've cleaned the build space and rebuilt the Docker container - no dice locally or in the CI stack, still fails the same way.
It looks like they're using an AWS CA in the cert chain:

Certificate chain
 0 s:CN = repo.saltstack.com
   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
   i:C = US, O = Amazon, CN = Amazon Root CA 1
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority

i do recall the starfield certs being a problem back in the day as well, but guessing its quite unlikely for the base build environment to have an AWS root CA in its trusted list.

Unhelpfully it looks like Salt has changed repo: https://repo.saltproject.io/#debian

I suspect this is because VMware acquired them.

Thank you for pointing that out - updated defaults.json and it seems to have made that issue go away.
For some reason its now breaking on using our internal repo (no TLS there, inside the datacenter), but i suspect its got something to do with the repo itself or some change in Debian since we started using it.

After cleaning the chroot and retrying, it now fails utterly with the '#' in there:

[2021-08-06 21:16:21] lb bootstrap_archives 
P: Configuring file /etc/apt/sources.list
E: Malformed entry 1 in list file /etc/apt/sources.list.d/custom.list (Suite)
E: The list of sources could not be read.
E: An unexpected failure occurred, exiting...
P: Begin unmounting filesystems...
P: Saving caches...
E: Malformed entry 1 in list file /etc/apt/sources.list.d/custom.list (Suite)
E: The list of sources could not be read.
make: *** [Makefile:32: iso] Error 1
vyos_bld@2bf3fc4ac845:/vyos$ cat build/chroot/etc/apt/sources.list.d/custom.list
deb https://repo.saltproject.io/#debian buster main
deb http://repo.powerdns.com/debian buster-rec-43 main
deb http://our.internal.repo/vyos buster main

Trying to use their instructions from https://repo.saltproject.io/#debian i'm back to the certificate issue - repo is set to https://repo.saltproject.io/py3/debian/10/amd64/latest buster main and the custom GPG key has been added, but certificate checks still fail hard:

Reading package lists...
W: https://repo.saltproject.io/py3/debian/10/amd64/latest/dists/buster/InRelease: No system certificates available. Try installing ca-certificates.
W: https://repo.saltproject.io/py3/debian/10/amd64/latest/dists/buster/Release: No system certificates available. Try installing ca-certificates.
E: The repository 'https://repo.saltproject.io/py3/debian/10/amd64/latest buster Release' does not have a Release file.
E: An unexpected failure occurred, exiting...
P: Begin unmounting filesystems...
P: Saving caches...
Reading package lists...
Building dependency tree...
Del nftables 0.9.6-1 [66.8 kB]

https://repo.saltproject.io/py3/debian/10/amd64/latest buster Release looks wrong - shouldn't it read main at the end, instead of Release?

The procedure I usually end up using:

curl -o - /usr/share/keyrings/salt-archive-keyring.gpg | apt-key add -
echo "deb https://repo.saltproject.io/py3/debian/10/amd64/latest buster main" > /etc/apt/sources.list.d/saltstack.list

But not everyone wants Salt's apt-key able to sign any package.

What packages are we actually pulling from there? Any reason they're not in the VyOS repo itself?
I removed their repo entirely from the JSON config and my image built fine (apparently i now have to add a /debian suffix for all packages in our repo, but that's weirdness in the repo management stack):

Reading package lists...
Building dependency tree...
Reading state information...
[2021-08-06 21:39:40] lb source 
P: Source stage disabled, skipping
P: Build completed successfully

salt-minion which depends on salt-common which may depend on a couple of other things:

Depends: python3-apt, python3-dateutil, python3-jinja2, python3-msgpack (>= 0.4), python3-pkg-resources, python3-requests, python3-yaml, python3-systemd, python3-psutil, python3-distro, python3-pycryptodome, python3-zmq (>= 17.0.0), python3-markupsafe, python3:any

Most of those are fulfilled by buster, but there might be a couple of others in the salt repo.

Seems like the repo's not needed anymore as my iso just built without it, twice, after a clean, and with a bunch of added stuff (tor, docker, systemd-nspawn, xtables-addons, hardened-malloc, a grsec kernel, etc) for which dependencies are also available without it.
Either way, probably a good idea to keep deps for anything third-party in the vyos repo itself since third parties can become hostile through buyouts or license changes any time and anywhere in these post-FOSS times.

salt-minion in the debian buster tree is version 2016.11.2+ds-1+deb9u4

It's ancient and buggy, and lacks a lot of support for network automation.

syncer changed the subtype of this task from "Task" to "Bug".Aug 27 2021, 10:26 AM

I can not reproduce the issue for 1.4 and 1.3 release. Maybe a temporary image build issue with the upstream repositories.

VyOS 1.3 (Debian Buster based) uses: 3002.5+ds-1 and VyOS 1.4 (Debian Bullseye based) uses: 3002.6+dfsg1-4

I still think we should upgrade both to 3003.2

Once i dropped that repo everything started to work again... and while it may be a temporary thing, i think that this sort of illustrates the problem with reliance on external repos. Its probably safer to mirror their content into something that can be relied upon to stay live as companies are bought and sold all the time, resources vanish, and licenses change from real FOSS to semi-permissive .

Viacheslav claimed this task.
Viacheslav moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.