Page MenuHomeVyOS Platform
Create Task
Maniphest T4041

"transition-script" doesn't work on "sync-group"
Closed, ResolvedPublicSR
Actions

Assigned To
Unknown Object (User)
Authored By
m.korobeinikov
Dec 3 2021, 12:50 AM
Tags
Referenced Files
F2205310: image.png
Dec 6 2021, 11:46 PM
F2205314: image.png
Dec 6 2021, 11:46 PM
F2205316: image.png
Dec 6 2021, 11:46 PM
Subscribers
c-po
m.korobeinikov
Maintainers

Description

"transition-script" doesn't work on "sync-group"

set high-availability vrrp sync-group SYN transition-script fault /config/scripts/*
set high-availability vrrp sync-group SYN transition-script backup /config/scripts/*
set high-availability vrrp sync-group SYN transition-script master /config/scripts/***

LOG VRRP when scripn on "sync-group"

Dec 02 09:33:42 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec 02 09:33:43 Keepalived[7328]: Starting Keepalived v2.1.5 (07/13,2020)
Dec 02 09:33:43 Keepalived[7328]: Running on Linux 5.4.162-amd64-vyos #1 SMP Fri Nov 26 20:35:17 UTC 2021 (built for Linux 4.19.160)
Dec 02 09:33:43 Keepalived[7328]: Command line: '/usr/sbin/keepalived' '--use-file' '/run/keepalived/keepalived.conf' '--pid'
Dec 02 09:33:43 Keepalived[7328]: '/run/keepalived/keepalived.pid' '--dont-fork' '--snmp'
Dec 02 09:33:43 Keepalived[7328]: Opening file '/run/keepalived/keepalived.conf'.
Dec 02 09:33:43 Keepalived[7328]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Dec 02 09:33:43 Keepalived[7328]: Starting VRRP child process, pid=7329
Dec 02 09:33:43 Keepalived_vrrp[7329]: Registering Kernel netlink reflector
Dec 02 09:33:43 Keepalived_vrrp[7329]: Registering Kernel netlink command channel
Dec 02 09:33:43 Keepalived_vrrp[7329]: Opening file '/run/keepalived/keepalived.conf'.
Dec 02 09:33:43 Keepalived_vrrp[7329]: Starting SNMP subagent
Dec 02 09:33:43 Keepalived_vrrp[7329]: Warning: Failed to connect to the agentx master agent ([NIL]):
Dec 02 09:33:43 Keepalived_vrrp[7329]: Unsafe permissions found for script '/config/scripts/vrrp-check.sh' - disabling.
Dec 02 09:33:43 Keepalived_vrrp[7329]: Disabling track script healthcheck_outside due to insecure
Dec 02 09:33:43 Keepalived_vrrp[7329]: Registering gratuitous ARP shared channel
Dec 02 09:33:43 Keepalived_vrrp[7329]: (inside) Entering BACKUP STATE (init)
Dec 02 09:33:43 Keepalived_vrrp[7329]: (outside) Entering BACKUP STATE (init)
Dec 02 09:33:43 Keepalived_vrrp[7329]: (inside) received lower priority (90) advert from 10.55.0.3 - discarding
Dec 02 09:33:43 Keepalived_vrrp[7329]: (outside) received lower priority (90) advert from 108.175.223.236 - discarding
Dec 02 09:33:43 keepalived-fifo.py[7330]: Starting FIFO pipe for Keepalived
Dec 02 09:33:43 keepalived-fifo.py[7330]: Loaded configuration: {'vrrp_groups': {'SYN': {'STOP': None, 'FAULT': '/config/scripts/ipsec-stop.sh', 'BACKUP': '/config/scripts/ipsec-stop.sh', 'MASTER': '/config/scripts/ipsec-restart.sh'}}, 'sync_groups': {}}
Dec 02 09:33:43 keepalived-fifo.py[7330]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Dec 02 09:33:43 keepalived-fifo.py[7330]: Message reading start
Dec 02 09:33:43 keepalived-fifo.py[7330]: Message processing start
Dec 02 09:33:44 keepalived-fifo.py[7330]: Received message: GROUP "SYN" BACKUP 0
Dec 02 09:33:44 keepalived-fifo.py[7330]: GROUP SYN changed state to BACKUP
Dec 02 09:33:44 keepalived-fifo.py[7330]: Received message: INSTANCE "inside" BACKUP 95
Dec 02 09:33:44 keepalived-fifo.py[7330]: INSTANCE inside changed state to BACKUP
Dec 02 09:33:44 keepalived-fifo.py[7330]: Received message: INSTANCE "outside" BACKUP 95
Dec 02 09:33:44 keepalived-fifo.py[7330]: INSTANCE outside changed state to BACKUP

vyos@vyos:~$ cat /run/keepalived/keepalived.conf

  1. Autogenerated by VyOS
  2. Do not edit this file, all your changes will be lost
  3. on next commit or reboot

global_defs {

dynamic_interfaces
script_user root
# Don't run scripts configured to be run as root if any part of the path
# is writable by a non-root user.
enable_script_security
notify_fifo /run/keepalived/keepalived_notify_fifo
notify_fifo_script /usr/libexec/vyos/system/keepalived-fifo.py

}

vrrp_instance inside {

state BACKUP
interface eth8.712
virtual_router_id 3
priority 95
advert_int 1
preempt_delay 0
virtual_ipaddress {
    10.55.0.1/29
}

}
vrrp_script healthcheck_outside {

script "/config/scripts/vrrp-check.sh"
interval 1
fall 1
rise 1

}
vrrp_instance outside {

state BACKUP
interface eth10.711
virtual_router_id 2
priority 95
advert_int 1
preempt_delay 0
virtual_ipaddress {
    *.175.223.238/29
}
track_script {
    healthcheck_outside
}

}

vrrp_sync_group SYN {

group {
    outside
    inside
}

}

but works fine on "vrrp group"

set high-availability vrrp group outside transition-script fault /config/scripts/*
set high-availability vrrp group outside transition-script backup /config/scripts/*
set high-availability vrrp group outside transition-script master /config/scripts/***

LOG VRRP when scripn on "vrrp group"

Dec 02 09:18:02 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec 02 09:18:02 Keepalived[4113]: Starting Keepalived v2.1.5 (07/13,2020)
Dec 02 09:18:02 Keepalived[4113]: Running on Linux 5.4.156-amd64-vyos #1 SMP Thu Oct 28 18:19:14 UTC 2021 (built for Linux 4.19.160)
Dec 02 09:18:02 Keepalived[4113]: Command line: '/usr/sbin/keepalived' '--use-file' '/run/keepalived/keepalived.conf' '--pid'
Dec 02 09:18:02 Keepalived[4113]: '/run/keepalived/keepalived.pid' '--dont-fork' '--snmp'
Dec 02 09:18:02 Keepalived[4113]: Opening file '/run/keepalived/keepalived.conf'.
Dec 02 09:18:02 Keepalived[4113]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Dec 02 09:18:02 Keepalived[4113]: Starting VRRP child process, pid=4114
Dec 02 09:18:02 Keepalived_vrrp[4114]: Registering Kernel netlink reflector
Dec 02 09:18:02 Keepalived_vrrp[4114]: Registering Kernel netlink command channel
Dec 02 09:18:02 Keepalived_vrrp[4114]: Opening file '/run/keepalived/keepalived.conf'.
Dec 02 09:18:02 Keepalived_vrrp[4114]: Starting SNMP subagent
Dec 02 09:18:02 Keepalived_vrrp[4114]: Warning: Failed to connect to the agentx master agent ([NIL]):
Dec 02 09:18:02 Keepalived_vrrp[4114]: Unsafe permissions found for script '/config/scripts/vrrp-check.sh' - disabling.
Dec 02 09:18:02 Keepalived_vrrp[4114]: Disabling track script healthcheck_outside due to insecure
Dec 02 09:18:02 Keepalived_vrrp[4114]: Registering gratuitous ARP shared channel
Dec 02 09:18:02 Keepalived_vrrp[4114]: (inside) Entering BACKUP STATE (init)
Dec 02 09:18:02 Keepalived_vrrp[4114]: (outside) Entering BACKUP STATE (init)
Dec 02 09:18:03 keepalived-fifo.py[4115]: Starting FIFO pipe for Keepalived
Dec 02 09:18:03 keepalived-fifo.py[4115]: Loaded configuration: {'vrrp_groups': {'inside': {'STOP': None, 'FAULT': None, 'BACKUP': None, 'MASTER': None}, 'outside': {'STOP': None, 'FAULT': '/config/scripts/ipsec-stop.sh', 'BACKUP': '/config/scripts/ipsec-stop.sh', 'MASTER': '/config/scripts/ipsec-restart.sh'}, 'SYN': {'STOP': None, 'FAULT': None, 'BACKUP': None, 'MASTER': None}}, 'sync_groups': {}}
Dec 02 09:18:03 keepalived-fifo.py[4115]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Dec 02 09:18:03 keepalived-fifo.py[4115]: Message reading start
Dec 02 09:18:03 keepalived-fifo.py[4115]: Message processing start
Dec 02 09:18:03 keepalived-fifo.py[4115]: Received message: GROUP "SYN" BACKUP 0
Dec 02 09:18:03 keepalived-fifo.py[4115]: GROUP SYN changed state to BACKUP
Dec 02 09:18:03 keepalived-fifo.py[4115]: Received message: INSTANCE "inside" BACKUP 90
Dec 02 09:18:03 keepalived-fifo.py[4115]: INSTANCE inside changed state to BACKUP
Dec 02 09:18:03 keepalived-fifo.py[4115]: Received message: INSTANCE "outside" BACKUP 90
Dec 02 09:18:03 keepalived-fifo.py[4115]: INSTANCE outside changed state to BACKUP
Dec 02 09:18:03 keepalived-fifo.py[4115]: Running the command: /config/scripts/ipsec-stop.sh

vyos@vyos:~$ cat /run/keepalived/keepalived.conf

  1. Autogenerated by VyOS
  2. Do not edit this file, all your changes will be lost
  3. on next commit or reboot

global_defs {

dynamic_interfaces
script_user root
# Don't run scripts configured to be run as root if any part of the path
# is writable by a non-root user.
enable_script_security
notify_fifo /run/keepalived/keepalived_notify_fifo
notify_fifo_script /usr/libexec/vyos/system/keepalived-fifo.py

}

vrrp_instance inside {

state BACKUP
interface eth1.712
virtual_router_id 3
priority 90
advert_int 1
preempt_delay 0
virtual_ipaddress {
    10.55.0.1/29
}

}
vrrp_script healthcheck_outside {

script "/config/scripts/vrrp-check.sh"
interval 30
fall 3
rise 1

}
vrrp_instance outside {

state BACKUP
interface eth0.711
virtual_router_id 2
priority 90
advert_int 1
preempt_delay 0
virtual_ipaddress {
    *.175.223.238/29
}
track_script {
    healthcheck_outside
}

}

vrrp_sync_group SYN {

group {
    outside
    inside
}

}

Details

Difficulty level
Easy (less than an hour)
Version
VyOS 1.3.0-epa3, VyOS 1.3-beta-202112010443 , VyOS 1.4-rolling-202112021432
Why the issue appeared?
Implementation mistake
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects

Event Timeline

m.korobeinikov created this task.Dec 3 2021, 12:50 AM
m.korobeinikov added a comment.Dec 3 2021, 12:55 AM

I checked it on these versions of VYOS. (VyOS 1.3.0-epa3, VyOS 1.3-beta-202112010443 , VyOS 1.4-rolling-202112021432)

pasik added a subscriber: pasik.Dec 3 2021, 1:23 PM
n.fort added a subscriber: n.fort.Dec 4 2021, 11:40 AM

It may be related to T4033 ???

c-po claimed this task.Dec 5 2021, 8:22 AM
c-po changed Difficulty level from Normal (likely a few hours) to Easy (less than an hour).Dec 5 2021, 7:49 PM
c-po changed Why the issue appeared? from Will be filled on close to Implementation mistake.
c-po closed this task as Resolved.Dec 5 2021, 8:26 PM
c-po triaged this task as High priority.
m.korobeinikov reopened this task as Open.Dec 6 2021, 11:46 PM

I tested this bug on "vyos-1.3-beta-202112060443".

image.png (261×460 px, 18 KB)

The problem has been partially resolved. If you restart VYOS, scripts on syn-groups don't work. After using the "vrrp restart" command, everything is ok.

LOG VRRP after REBOOT and before "VRRP RESTART" in state "MASTER"

Dec 06 23:20:20 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec 06 23:20:20 Keepalived[1819]: Starting Keepalived v2.1.5 (07/13,2020)
Dec 06 23:20:20 Keepalived[1819]: Running on Linux 5.4.163-amd64-vyos #1 SMP Thu Dec 2 07:05:05 UTC 2021 (built for Linux 4.19.160)
Dec 06 23:20:20 Keepalived[1819]: Command line: '/usr/sbin/keepalived' '--use-file' '/run/keepalived/keepalived.conf' '--pid'
Dec 06 23:20:20 Keepalived[1819]: '/run/keepalived/keepalived.pid' '--dont-fork' '--snmp'
Dec 06 23:20:20 Keepalived[1819]: Opening file '/run/keepalived/keepalived.conf'.
Dec 06 23:20:20 Keepalived[1819]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Dec 06 23:20:20 Keepalived[1819]: Starting VRRP child process, pid=1829
Dec 06 23:20:20 Keepalived_vrrp[1829]: Registering Kernel netlink reflector
Dec 06 23:20:20 Keepalived_vrrp[1829]: Registering Kernel netlink command channel
Dec 06 23:20:20 Keepalived_vrrp[1829]: Opening file '/run/keepalived/keepalived.conf'.
Dec 06 23:20:20 Keepalived_vrrp[1829]: Starting SNMP subagent
Dec 06 23:20:21 Keepalived_vrrp[1829]: Warning: Failed to connect to the agentx master agent ([NIL]):
Dec 06 23:20:21 Keepalived_vrrp[1829]: Unsafe permissions found for script '/config/scripts/vrrp-check.sh' - disabling.
Dec 06 23:20:21 Keepalived_vrrp[1829]: Disabling track script healthcheck_outside due to insecure
Dec 06 23:20:21 Keepalived_vrrp[1829]: Registering gratuitous ARP shared channel
Dec 06 23:20:21 Keepalived_vrrp[1829]: (inside) Entering BACKUP STATE (init)
Dec 06 23:20:21 Keepalived_vrrp[1829]: (outside) Entering BACKUP STATE (init)
Dec 06 23:20:22 keepalived-fifo.py[1834]: Starting FIFO pipe for Keepalived
Dec 06 23:20:22 keepalived-fifo.py[1834]: Unable to load configuration:
Dec 06 23:20:22 keepalived-fifo.py[1834]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Dec 06 23:20:22 keepalived-fifo.py[1834]: Message reading start
Dec 06 23:20:22 keepalived-fifo.py[1834]: Message processing start
Dec 06 23:20:22 keepalived-fifo.py[1834]: Received message: GROUP "SYN" BACKUP 0
Dec 06 23:20:22 keepalived-fifo.py[1834]: GROUP SYN changed state to BACKUP
Dec 06 23:20:22 keepalivmonospaced texted-fifo.py[1834]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'
Dec 06 23:20:24 Keepalived_vrrp[1829]: (outside) Entering MASTER STATE
Dec 06 23:20:24 Keepalived_vrrp[1829]: VRRP_Group(SYN) Syncing instances to MASTER state
Dec 06 23:20:24 Keepalived_vrrp[1829]: (inside) Entering MASTER STATE
Dec 06 23:20:25 keepalived-fifo.py[1834]: Received message: INSTANCE "inside" BACKUP 95
Dec 06 23:20:25 keepalived-fifo.py[1834]: INSTANCE inside changed state to BACKUP
Dec 06 23:20:25 keepalived-fifo.py[1834]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'

image.png (73×553 px, 5 KB)

Sync-Group scripts don't work

sync-group SYN {

member outside
member inside
transition-script {
    backup /config/scripts/ipsec-stop.sh
    fault /config/scripts/ipsec-stop.sh
    master /config/scripts/ipsec-restart.sh

LOG VRRP after REBOOT and before "VRRP RESTART" in state "BACKUP"

Dec 06 23:33:07 Keepalived_vrrp[1829]: Netlink reports eth10.711 down
Dec 06 23:33:07 Keepalived_vrrp[1829]: (outside) Entering FAULT STATE
Dec 06 23:33:07 Keepalived_vrrp[1829]: (outside) sent 0 priority
Dec 06 23:33:07 Keepalived_vrrp[1829]: VRRP_Group(SYN) Syncing instances to FAULT state
Dec 06 23:33:07 Keepalived_vrrp[1829]: (inside) Entering FAULT STATE
Dec 06 23:33:07 keepalived-fifo.py[1834]: Received message: INSTANCE "outside" BACKUP 95
Dec 06 23:33:07 keepalived-fifo.py[1834]: INSTANCE outside changed state to BACKUP
Dec 06 23:33:07 keepalived-fifo.py[1834]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'

image.png (74×590 px, 5 KB)

After reboot, VRRP doesn't run scripts on sync-groups. An error appears in the logs.

keepalived-fifo.py[1834]: Error processing message: 'KeepalivedFifo' object has no attribute 'vrrp_config_dict'

LOG VRRP after REBOOT and "VRRP RESTART"

Dec 06 23:41:06 Keepalived_vrrp[1829]: Stopped
Dec 06 23:41:06 Keepalived[1819]: Stopped Keepalived v2.1.5 (07/13,2020)
Dec 06 23:41:06 systemd[1]: keepalived.service: Succeeded.
Dec 06 23:41:06 systemd[1]: Stopped Keepalive Daemon (LVS and VRRP).
Dec 06 23:41:06 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec 06 23:41:06 Keepalived[2916]: Starting Keepalived v2.1.5 (07/13,2020)
Dec 06 23:41:06 Keepalived[2916]: Running on Linux 5.4.163-amd64-vyos #1 SMP Thu Dec 2 07:05:05 UTC 2021 (built for Linux 4.19.160)
Dec 06 23:41:06 Keepalived[2916]: Command line: '/usr/sbin/keepalived' '--use-file' '/run/keepalived/keepalived.conf' '--pid'
Dec 06 23:41:06 Keepalived[2916]: '/run/keepalived/keepalived.pid' '--dont-fork' '--snmp'
Dec 06 23:41:06 Keepalived[2916]: Opening file '/run/keepalived/keepalived.conf'.
Dec 06 23:41:06 Keepalived[2916]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Dec 06 23:41:06 Keepalived[2916]: Starting VRRP child process, pid=2917
Dec 06 23:41:06 Keepalived_vrrp[2917]: Registering Kernel netlink reflector
Dec 06 23:41:06 Keepalived_vrrp[2917]: Registering Kernel netlink command channel
Dec 06 23:41:06 Keepalived_vrrp[2917]: Opening file '/run/keepalived/keepalived.conf'.
Dec 06 23:41:06 Keepalived_vrrp[2917]: Starting SNMP subagent
Dec 06 23:41:06 Keepalived_vrrp[2917]: Warning: Failed to connect to the agentx master agent ([NIL]):
Dec 06 23:41:06 Keepalived_vrrp[2917]: Unsafe permissions found for script '/config/scripts/vrrp-check.sh' - disabling.
Dec 06 23:41:06 Keepalived_vrrp[2917]: Disabling track script healthcheck_outside due to insecure
Dec 06 23:41:06 Keepalived_vrrp[2917]: Registering gratuitous ARP shared channel
Dec 06 23:41:06 Keepalived_vrrp[2917]: (inside) Entering BACKUP STATE (init)
Dec 06 23:41:06 Keepalived_vrrp[2917]: (outside) Entering BACKUP STATE (init)
Dec 06 23:41:06 keepalived-fifo.py[2918]: Starting FIFO pipe for Keepalived
Dec 06 23:41:06 keepalived-fifo.py[2918]: Loaded configuration: {'group': {'inside': {'interface': 'eth8.712', 'priority': '95', 'virtual_address': ['10.55.0.1/29'], 'vrid': '3'}, 'outside': {'hea}
Dec 06 23:41:06 keepalived-fifo.py[2918]: PIPE already exist: /run/keepalived/keepalived_notify_fifo
Dec 06 23:41:06 keepalived-fifo.py[2918]: Message reading start
Dec 06 23:41:06 keepalived-fifo.py[2918]: Message processing start
Dec 06 23:41:07 keepalived-fifo.py[2918]: Received message: GROUP "SYN" BACKUP 0
Dec 06 23:41:07 keepalived-fifo.py[2918]: GROUP SYN changed state to BACKUP
Dec 06 23:41:07 keepalived-fifo.py[2918]: Running the command: /config/scripts/ipsec-stop.sh
Dec 06 23:41:07 keepalived-fifo.py[2918]: Received message: INSTANCE "inside" BACKUP 95
Dec 06 23:41:07 keepalived-fifo.py[2918]: INSTANCE inside changed state to BACKUP
Dec 06 23:41:07 keepalived-fifo.py[2918]: Received message: INSTANCE "outside" BACKUP 95
Dec 06 23:41:07 keepalived-fifo.py[2918]: INSTANCE outside changed state to BACKUP
Dec 06 23:41:10 Keepalived_vrrp[2917]: (outside) Entering MASTER STATE
Dec 06 23:41:10 Keepalived_vrrp[2917]: VRRP_Group(SYN) Syncing instances to MASTER state
Dec 06 23:41:10 Keepalived_vrrp[2917]: (inside) Entering MASTER STATE
Dec 06 23:41:10 keepalived-fifo.py[2918]: Received message: INSTANCE "outside" MASTER 95
Dec 06 23:41:10 keepalived-fifo.py[2918]: INSTANCE outside changed state to MASTER
Dec 06 23:41:10 keepalived-fifo.py[2918]: Received message: INSTANCE "inside" MASTER 95
Dec 06 23:41:10 keepalived-fifo.py[2918]: INSTANCE inside changed state to MASTER
Dec 06 23:41:10 keepalived-fifo.py[2918]: Received message: GROUP "SYN" MASTER 0
Dec 06 23:41:10 keepalived-fifo.py[2918]: GROUP SYN changed state to MASTER
Dec 06 23:41:10 keepalived-fifo.py[2918]: Running the command: /config/scripts/ipsec-restart.sh

Only after the vrrp restart command is applied, the scripts begin to be applied.

Unknown Object (User) signed these changes with MFA.Dec 7 2021, 12:34 AM
Unknown Object (User) closed this task as Resolved.
Unknown Object (User) claimed this task.
Unknown Object (User) changed the subtype of this task from "Bug" to "SR".
Unknown Object (User) edited subscribers, added: c-po, Unknown Object (User); removed: n.fort, pasik.

There is a task with VRRP scripts problem on reboot:

https://phabricator.vyos.net/T4059
VRRP sync-group transition script does not persist after reboot

m.korobeinikov mentioned this in T4059: VRRP sync-group transition script does not persist after reboot.Dec 7 2021, 5:26 AM
c-po moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.Dec 9 2021, 10:33 PM
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.Aug 29 2022, 12:15 PM
syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus (1.3.0) board.