Page MenuHomeVyOS Platform

keepalived-fifo.py unable to load config
Closed, ResolvedPublicBUG

Description

Hi,

I'm in the process of upgrading VyOS on our firewalls and currently testing the new image. I have an issue with keepalived-fifo.py that can't load the config when VRRP is initially configured. If I reboot vyos or restart keepalived it works correctly. This breaks executing transition-scripts.
Looks like the new image runs newer keepalived and keepalived-fifo.py file has been updated since VyOS 1.4-rolling-202108111528 we currently use.

0 14:54:48 [email protected]:~$ show version

Version: VyOS 1.4-rolling-202206201610

Jul 11 14:29:33 vyos systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Jul 11 14:29:33 vyos Keepalived[61689]: Starting Keepalived v2.2.4 (12/22,2021), git commit v2.2.4-227-g8af889bc
Jul 11 14:29:33 vyos Keepalived[61689]: Running on Linux 5.10.123-amd64-vyos #1 SMP Sat Jun 18 06:08:47 UTC 2022 (built for Linux 5.10.113)
Jul 11 14:29:33 vyos Keepalived[61689]: Command line: '/usr/sbin/keepalived' '--use-file' '/run/keepalived/keepalived.conf' '--pid'
Jul 11 14:29:33 vyos Keepalived[61689]: '/run/keepalived/keepalived.pid' '--dont-fork' '--snmp'
Jul 11 14:29:33 vyos Keepalived[61689]: Configuration file /run/keepalived/keepalived.conf
Jul 11 14:29:33 vyos Keepalived[61689]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jul 11 14:29:33 vyos Keepalived[61689]: Starting VRRP child process, pid=61703
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: Starting SNMP subagent
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: Created directory: /var/lib/snmp/cert_indexes
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: Warning: Failed to connect to the agentx master agent ([NIL]):
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jul 11 14:29:33 vyos Keepalived[61689]: Startup complete
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (12) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (99) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (104) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (106) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (108) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (109) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (111) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (112) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (113) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (114) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (118) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (120) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (121) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (122) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (124) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (125) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (126) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos conntrack-tools[61751]: vyatta-vrrp-conntracksync invoked at Mon 11 Jul 2022 02:29:33 PM UTC
Jul 11 14:29:33 vyos conntrack-tools[61763]: vyos transitioning to BACKUP state for VRRP sync-group [sync-group]
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Starting FIFO pipe for Keepalived
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Unable to load configuration:
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Message reading start
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Message processing start
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Received message: GROUP "sync-group" BACKUP 0
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: GROUP sync-group changed state to BACKUP
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (126) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: VRRP_Group(sync-group) Syncing instances to MASTER state
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (99) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (113) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (106) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (121) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (122) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (104) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (108) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (109) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (111) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (112) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (125) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (114) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (118) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (120) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (124) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (12) Entering MASTER STATE
Jul 11 14:29:36 vyos conntrack-tools[62325]: vyatta-vrrp-conntracksync invoked at Mon 11 Jul 2022 02:29:36 PM UTC
Jul 11 14:29:36 vyos conntrack-tools[62329]: vyos transitioning to MASTER state for VRRP sync-group [sync-group]
Jul 11 14:29:36 vyos keepalived-fifo.py[61713]: Received message: INSTANCE "12" BACKUP 200
Jul 11 14:29:36 vyos keepalived-fifo.py[61713]: INSTANCE 12 changed state to BACKUP
Jul 11 14:29:36 vyos keepalived-fifo.py[61713]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'
...
\-- Boot 987396fe63f74609b58a60a9bf6bacb6 --
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Starting FIFO pipe for Keepalived
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Loaded configuration: {'group': {'12': {'address': {'10.193.255.190/26': {}}, 'authentication': {'password': 'xxx', 'type': 'ah'}, 'description':>
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Message reading start
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Message processing start
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Received message: GROUP "sync-group" BACKUP 0
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: GROUP sync-group changed state to BACKUP

Thanks,
Damian

Details

Difficulty level
Unknown (require assessment)
Version
1.4-rolling-202206201610
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

The problem here seems to be that keepalived is started before the complete commit is finished. So conf.get_config_dict() fails to get the config.

I've tested with a simple sleep in the keepalived-fifo.py, and a sleep of 4 seconds seems sufficient to make sure it works correctly.

A better solution might be retrying the conf.get_config_dict() a couple of times.

This is what I did (forgot to write it here) with the difference that my sleep timer is 60s as my config has many lines.
Would be good to have this fixed properly.

I've create a PR which does the retry part. It retries 10 time every 0.5 seconds until it succeeds or it's out of retries.

https://github.com/vyos/vyos-1x/pull/1485

Viacheslav changed the task status from Open to Needs testing.Mon, Aug 29, 10:51 AM
Viacheslav assigned this task to roedie.

I've tested this and it seems to work correctly.