keepalived-fifo.py unable to load config
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	krox2
	Jul 11 2022, 3:07 PM

Description

Hi,

I'm in the process of upgrading VyOS on our firewalls and currently testing the new image. I have an issue with keepalived-fifo.py that can't load the config when VRRP is initially configured. If I reboot vyos or restart keepalived it works correctly. This breaks executing transition-scripts.
Looks like the new image runs newer keepalived and keepalived-fifo.py file has been updated since VyOS 1.4-rolling-202108111528 we currently use.

0 14:54:48 vagrant@vyos:~$ show version

Version: VyOS 1.4-rolling-202206201610

Jul 11 14:29:33 vyos systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Jul 11 14:29:33 vyos Keepalived[61689]: Starting Keepalived v2.2.4 (12/22,2021), git commit v2.2.4-227-g8af889bc
Jul 11 14:29:33 vyos Keepalived[61689]: Running on Linux 5.10.123-amd64-vyos #1 SMP Sat Jun 18 06:08:47 UTC 2022 (built for Linux 5.10.113)
Jul 11 14:29:33 vyos Keepalived[61689]: Command line: '/usr/sbin/keepalived' '--use-file' '/run/keepalived/keepalived.conf' '--pid'
Jul 11 14:29:33 vyos Keepalived[61689]: '/run/keepalived/keepalived.pid' '--dont-fork' '--snmp'
Jul 11 14:29:33 vyos Keepalived[61689]: Configuration file /run/keepalived/keepalived.conf
Jul 11 14:29:33 vyos Keepalived[61689]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jul 11 14:29:33 vyos Keepalived[61689]: Starting VRRP child process, pid=61703
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: Starting SNMP subagent
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: Created directory: /var/lib/snmp/cert_indexes
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: Warning: Failed to connect to the agentx master agent ([NIL]):
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jul 11 14:29:33 vyos Keepalived[61689]: Startup complete
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (12) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (99) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (104) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (106) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (108) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (109) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (111) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (112) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (113) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (114) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (118) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (120) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (121) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (122) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (124) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (125) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos Keepalived_vrrp[61703]: (126) Entering BACKUP STATE (init)
Jul 11 14:29:33 vyos conntrack-tools[61751]: vyatta-vrrp-conntracksync invoked at Mon 11 Jul 2022 02:29:33 PM UTC
Jul 11 14:29:33 vyos conntrack-tools[61763]: vyos transitioning to BACKUP state for VRRP sync-group [sync-group]
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Starting FIFO pipe for Keepalived
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Unable to load configuration:
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Message reading start
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Message processing start
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Received message: GROUP "sync-group" BACKUP 0
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: GROUP sync-group changed state to BACKUP
Jul 11 14:29:33 vyos keepalived-fifo.py[61713]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (126) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: VRRP_Group(sync-group) Syncing instances to MASTER state
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (99) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (113) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (106) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (121) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (122) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (104) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (108) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (109) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (111) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (112) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (125) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (114) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (118) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (120) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (124) Entering MASTER STATE
Jul 11 14:29:36 vyos Keepalived_vrrp[61703]: (12) Entering MASTER STATE
Jul 11 14:29:36 vyos conntrack-tools[62325]: vyatta-vrrp-conntracksync invoked at Mon 11 Jul 2022 02:29:36 PM UTC
Jul 11 14:29:36 vyos conntrack-tools[62329]: vyos transitioning to MASTER state for VRRP sync-group [sync-group]
Jul 11 14:29:36 vyos keepalived-fifo.py[61713]: Received message: INSTANCE "12" BACKUP 200
Jul 11 14:29:36 vyos keepalived-fifo.py[61713]: INSTANCE 12 changed state to BACKUP
Jul 11 14:29:36 vyos keepalived-fifo.py[61713]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'
...
\-- Boot 987396fe63f74609b58a60a9bf6bacb6 --
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Starting FIFO pipe for Keepalived
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Loaded configuration: {'group': {'12': {'address': {'10.193.255.190/26': {}}, 'authentication': {'password': 'xxx', 'type': 'ah'}, 'description':>
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Message reading start
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Message processing start
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: Received message: GROUP "sync-group" BACKUP 0
Jul 11 14:44:32 vyos keepalived-fifo.py[11637]: GROUP sync-group changed state to BACKUP

Thanks,
Damian

Details

Difficulty level: Unknown (require assessment)
Version: 1.4-rolling-202206201610
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Bug (incorrect behavior)

Event Timeline

krox2 created this task.Jul 11 2022, 3:07 PM

pasik added a subscriber: pasik.Jul 11 2022, 6:50 PM

The problem here seems to be that keepalived is started before the complete commit is finished. So conf.get_config_dict() fails to get the config.

I've tested with a simple sleep in the keepalived-fifo.py, and a sleep of 4 seconds seems sufficient to make sure it works correctly.

A better solution might be retrying the conf.get_config_dict() a couple of times.

This is what I did (forgot to write it here) with the difference that my sleep timer is 60s as my config has many lines.
Would be good to have this fixed properly.

I've create a PR which does the retry part. It retries 10 time every 0.5 seconds until it succeeds or it's out of retries.

https://github.com/vyos/vyos-1x/pull/1485

https://github.com/vyos/vyos-1x/pull/1486 try#2

Viacheslav changed the task status from Open to Needs testing.Aug 29 2022, 10:51 AM

Viacheslav assigned this task to roedie.

I've tested this and it seems to work correctly.

roedie moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.Sep 17 2022, 8:34 PM

I think this needs to be backported to 1.3 too

@florin If this is needed I'll make a pull request coming week.

roedie reopened this task as Backport pending.Oct 10 2022, 9:18 AM

Done: https://github.com/vyos/vyos-1x/pull/1630

roedie closed this task as Resolved.Nov 23 2022, 6:23 PM