Page MenuHomeVyOS Platform

Container registry with authentication prevents config load (section container) after reboot
Closed, ResolvedPublicBUG

Description

Let's assume that you have a private Harbor instance with a public project which proxies Dockerhub.
When you have a container configuration like the following everything works fine after a reboot:

vyos@vyos# show container
 name alpine {
     allow-host-networks
     image registry.example.org/dockerhub/library/alpine:latest
     restart always
 }
 registry registry.example.org {
 }

It still works when you add authentication to it:

vyos@vyos# show container
 name alpine {
     allow-host-networks
     image registry.example.org/dockerhub/library/alpine:latest
     restart always
 }
 registry registry.example.org {
	 password "my-password"
	 username "my-username"
 }

But then, after a reboot the configuration is not loaded:

vyos@vyos:~$ configure
WARNING: There was a config error on boot: saving the configuration now could overwrite data.
You may want to check and reload the boot config
[edit]
vyos@vyos# load
Loading configuration from 'config.boot'
Load complete. Use 'commit' to make changes effective.
[edit]
vyos@vyos# compare
+ container {
+     name alpine {
+         allow-host-networks
+         image "registry.example.org/dockerhub/library/alpine:latest"
+         restart "always"
+     }
+     registry registry.example.org {
+         authentication {
+             password "my-password"
+             username "my-username"
+         }
+     }
+ }

Details

Difficulty level
Easy (less than an hour)
Version
1.4 rc1
Why the issue appeared?
Implementation mistake
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)

Event Timeline

The first thing could be that the container cannot connect to the registry as it happens before static routing (not sure).

vyos@r4# /opt/vyatta/sbin/priority.pl | match "container|static"
450 container
480 protocols/static
481 vrf/name/node.tag/protocols/static
[edit]
vyos@r4#

I think I found the problem.

The order of what get loaded from the configuration is sub-optimal.
The container part get loaded before the routing part (static, bgp).

I checked the boot log and filtered it for config and router.
(The real registry name I use is replaced with registry.example.org)

vyos@vyos:~$ show log | grep -E "vyos-config|vyos-router"
Jan 08 16:16:50 systemd[1]: Started vyos-configd.service - VyOS configuration daemon.
Jan 08 16:17:01 systemd[1]: Started vyos-router.service - VyOS Router.
Jan 08 16:17:05 vyos-router[1148]: Waiting for NICs to settle down: settled in 0sec..
Jan 08 16:17:12 vyos-router[1148]: Mounting VyOS Config...done.
Jan 08 16:17:17 vyos-configd[769]: Received message: {"type": "init"}
Jan 08 16:17:18 vyos-configd[769]: config session pid is 1592
Jan 08 16:17:18 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/host_name.py"}
Jan 08 16:17:18 vyos-configd[769]: Sending response 1
Jan 08 16:17:18 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/system_console.py"}
Jan 08 16:17:18 vyos-configd[769]: Sending response 1
Jan 08 16:17:18 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/conntrack.py"}
Jan 08 16:17:19 vyos-configd[769]: Sending response 1
Jan 08 16:17:19 vyos-configd[769]: Received message: {"type": "node", "data": "VYOS_TAGNODE_VALUE=lo/usr/libexec/vyos/conf_mode/interfaces-loopback.py"}
Jan 08 16:17:19 vyos-configd[769]: Sending response 1
Jan 08 16:17:19 vyos-configd[769]: Received message: {"type": "node", "data": "VYOS_TAGNODE_VALUE=eth0/usr/libexec/vyos/conf_mode/interfaces-ethernet.py"}
Jan 08 16:17:19 vyos-configd[769]: Sending response 1
Jan 08 16:17:20 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/system-syslog.py"}
Jan 08 16:17:20 vyos-configd[769]: Sending response 1
Jan 08 16:17:21 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/system-login.py"}
Jan 08 16:17:21 vyos-configd[769]: Sending response 8
Jan 08 16:17:23 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/host_name.py"}
Jan 08 16:17:24 vyos-configd[769]: Sending response 1
Jan 08 16:17:24 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/host_name.py"}
Jan 08 16:17:24 vyos-configd[769]: Sending response 1
Jan 08 16:17:24 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/config_mgmt.py"}
Jan 08 16:17:24 vyos-configd[769]: Sending response 8
Jan 08 16:17:25 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/container.py"}
Jan 08 16:17:25 vyos-configd[769]: Error: authenticating creds for "registry.example.org": pinging container
Jan 08 16:17:25 vyos-configd[769]: registry registry.example.org: Get "https://registry.example.org/v2/": dial
Jan 08 16:17:25 vyos-configd[769]: tcp: lookup registry.example.org on 9.9.9.9:53: dial udp 9.9.9.9:53:
Jan 08 16:17:25 vyos-configd[769]: connect: network is unreachable
Jan 08 16:17:25 vyos-configd[769]: Sending response 2
Jan 08 16:17:25 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/protocols_static.py"}
Jan 08 16:17:26 vyos-configd[769]: Sending response 1
Jan 08 16:17:26 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/protocols_bgp.py"}
Jan 08 16:17:27 vyos-configd[769]: Sending response 1
Jan 08 16:17:27 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/ntp.py"}
Jan 08 16:17:28 vyos-configd[769]: Sending response 1
Jan 08 16:17:28 vyos-configd[769]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/ssh.py"}
Jan 08 16:17:29 vyos-configd[769]: Sending response 1
Jan 08 16:17:31 vyos-router[1148]: Starting VyOS router: migrate configure failed!
Jan 08 16:17:31 vyos-config[1154]: Configuration error

So in my opinion either the check must not be made on boot but only on configuration changes.
Or the order must be changed so that container become the last part.

But then there is the problem if your only connectivity is via BGP/OSPF etc.
You need to wait for sessions to establish.

The same problem (container config does not get loaded) occurs, when DNS is not available for whatever reason.
This must not prevent the current container settings to be online.

Viacheslav suggested the following change which worked for me:

sudo nano /opt/vyatta/share/vyatta-cfg/templates/container/node.def

and change priority from 450 to 902

Warning would be much better because it would solve the problem.
When you have the image already loaded and the system was rebooted, the image should still exist and therefore the user/pass is not required (for the moment).

syncer changed the task status from In progress to Backport candidate.Feb 2 2024, 1:25 PM
c-po changed the task status from Backport candidate to In progress.Feb 23 2024, 8:54 PM
c-po edited projects, added VyOS 1.4 Sagitta (1.4.0); removed VyOS 1.4 Sagitta.
c-po changed Difficulty level from Unknown (require assessment) to Easy (less than an hour).
c-po moved this task from Need Triage to Finished on the VyOS 1.5 Circinus board.
c-po moved this task from In Progress to Finished on the VyOS 1.4 Sagitta (1.4.0-epa1) board.
dmbaturin changed Why the issue appeared? from Will be filled on close to Implementation mistake.
dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.