Page MenuHomeVyOS Platform

VRRP IPSEC-AH : sequence number xxxxxxx already processed. Packet dropped. Local(xxxxxxx)
Needs testing, Requires assessmentPublicBUG

Description

Hi guys,

Seems there is a bug in VRRP, both nodes entered MASTER state rendering the VRRP address unavailable.
In the log there are two main lines that popout

Dec 09 02:07:27 Keepalived_vrrp[3090]: (IPv4_VLAN99) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 09 07:28:19 Keepalived_vrrp[3090]: (IPv4_VLAN10) Entering MASTER STATE
Dec 09 07:28:19 Keepalived_vrrp[3090]: (IPv4_VLAN10) IPSEC-AH : sequence number 7626336 already processed. Packet dropped. Local(7626336)

As soon as I reboot the first node everything started working again on node 2 as expected and the master was able to take over again.

[email protected]:~$ show version

Version:          VyOS 1.3.0-epa3
Release train:    equuleus

Built by:         Sentrium S.L.
Built on:         Sun 31 Oct 2021 17:38 UTC
Build UUID:       383e45ad-b32a-4359-8183-9baacc8e69d9
Build commit ID:  bb511522cc3bb2-dirty

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-42 39 f0 01 50 fb b9 8f-6d 44 a4 9d be a0 66 17
Hardware UUID:    4239f001-50fb-b98f-6d44-a49dbea06617

Copyright:        VyOS maintainers and contributors

VRRP config

[email protected]:~$ show configuration commands | strip-private | match vrrp
set high-availability vrrp group IPv4_VLAN10 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN10 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN10 interface 'eth1.10'
set high-availability vrrp group IPv4_VLAN10 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN10 priority '200'
set high-availability vrrp group IPv4_VLAN10 virtual-address 'xxx.xxx.10.1/24'
set high-availability vrrp group IPv4_VLAN10 vrid '10'
set high-availability vrrp group IPv4_VLAN75 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN75 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN75 interface 'eth1.75'
set high-availability vrrp group IPv4_VLAN75 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN75 priority '200'
set high-availability vrrp group IPv4_VLAN75 virtual-address 'xxx.xxx.75.1/24'
set high-availability vrrp group IPv4_VLAN75 vrid '75'
set high-availability vrrp group IPv4_VLAN98 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN98 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN98 interface 'eth1.98'
set high-availability vrrp group IPv4_VLAN98 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN98 priority '200'
set high-availability vrrp group IPv4_VLAN98 virtual-address 'xxx.xxx.98.1/24'
set high-availability vrrp group IPv4_VLAN98 vrid '98'
set high-availability vrrp group IPv4_VLAN99 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN99 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN99 interface 'eth1.99'
set high-availability vrrp group IPv4_VLAN99 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN99 priority '200'
set high-availability vrrp group IPv4_VLAN99 virtual-address 'xxx.xxx.99.1/24'
set high-availability vrrp group IPv4_VLAN99 vrid '99'
set high-availability vrrp group IPv4_WAN authentication password xxxxxx
set high-availability vrrp group IPv4_WAN authentication type 'ah'
set high-availability vrrp group IPv4_WAN interface 'eth0'
set high-availability vrrp group IPv4_WAN preempt-delay '180'
set high-availability vrrp group IPv4_WAN priority '200'
set high-availability vrrp group IPv4_WAN virtual-address 'xxx.xxx.84.192/25'
set high-availability vrrp group IPv4_WAN vrid '1'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN10'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN75'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN98'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN99'
set high-availability vrrp sync-group VLAN member 'IPv4_WAN'{F2220989}

Greetings,

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.3.0-epa3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

m.korobeinikov changed the task status from Open to Needs testing.Dec 10 2021, 12:51 AM

@Viacheslav the only way is by letting it run.
As adviced in the slack I upgraed to differt version, just now it dropped again.
This time it's differtent as the backup still sayes it still the backup node but all traffic to the VRRP address is offline.

Master: Version:          VyOS 1.3-beta-202112090443
Slave: Version:          VyOS 1.4-rolling-202112090318

In the logging of the Master node:

[email protected]:~$ show log vrrp
-- Logs begin at Wed 2021-12-22 08:46:04 CET, end at Wed 2021-12-22 16:43:56 CET. --
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN99) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780985)
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN10) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780985)
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN75) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780985)
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN98) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780985)

Log of the slave

Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN75) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN10) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN98) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN99) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 16:14:46 Keepalived_vrrp[3130]: A thread timer expired 4.842168 seconds ago
Dec 22 16:14:46 Keepalived_vrrp[3130]: (IPv4_VLAN99) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: VRRP_Group(VLAN) Syncing instances to MASTER state
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN10) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) Entering MASTER STATE
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN99" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN99 changed state to MASTER
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN98) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_WAN) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780969)
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) Master received advert from 10.10.75.253 with higher priority 200, ours 100
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: VRRP_Group(VLAN) Syncing instances to BACKUP state
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN10) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN98) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN99) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_WAN) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN99) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780969)
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN10) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780966)
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN98) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780966)
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN10" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN10 changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN75" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN75 changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN98" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN98 changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_WAN" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_XS4ALL changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: GROUP "VLAN" MASTER 0
Dec 22 16:14:56 keepalived-fifo.py[3144]: GROUP VLAN changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN75" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN75 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN10" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN10 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN98" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN98 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN99" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN99 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_WAN" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_XS4ALL changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: GROUP "VLAN" BACKUP 0
Dec 22 16:14:56 keepalived-fifo.py[3144]: GROUP VLAN changed state to BACKUP