Page MenuHomeVyOS Platform

conntrack-sync no longer works with VRRP/high-availability
Open, NormalPublicBUG

Description

When trying to assign conntrack-sync to a new 'high-availability' vrrp group, it says the group does not exist.

Example:

set high-availability vrrp group int1 hello-source-address '10.60.1.3'
set high-availability vrrp group int1 interface 'eth1.200'
set high-availability vrrp group int1 peer-address '10.60.1.2'
set high-availability vrrp group int1 preempt-delay '600'
set high-availability vrrp group int1 priority '200'
set high-availability vrrp group int1 virtual-address '10.60.1.1/24'
set high-availability vrrp group int1 vrid '200'
set service conntrack-sync accept-protocol 'tcp,udp,icmp'
set service conntrack-sync event-listen-queue-size '8'
set service conntrack-sync failover-mechanism vrrp sync-group int1
set service conntrack-sync interface 'eth1.200'
set service conntrack-sync mcast-group '225.0.0.50'
set service conntrack-sync sync-queue-size '8'

When you commit it, you get this error:

vyos@fw1# commit
[ service conntrack-sync ]
conntrack-sync error: VRRP sync-group int1 is not configured

[[service conntrack-sync]] failed
Commit failed
[edit]
vyos@fw1#

I assume it works with the old style of vrrp assigned to an interface, with an explicit sync-group tag, but I didn't test it, as I didn't want to rebuild everything using the older style!

Details

Difficulty level
Unknown (require assessment)
Version
vyos-1.3-rolling-202003151839
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change

Event Timeline

xrobau created this task.Jul 22 2019, 1:46 PM
xrobau updated the task description. (Show Details)Jul 22 2019, 1:48 PM

Confirmed still present in VyOS 1.3-rolling-201911030242

pasik added a subscriber: pasik.Nov 4 2019, 9:51 AM
Merijn added a subscriber: Merijn.Nov 4 2019, 11:54 AM

You have to add a sync-group.
set high-availability vrrp sync-group intgroup member int1
set service conntrack-sync failover-mechanism vrrp sync-group intgroup

After adding the missing command set high-availability vrrp sync-group sync member int1, we have a new error when starting conntrackd

[Tue Nov 5 00:07:41 2019] (pid=30976) [ERROR] unknown layer 3 protocol

This is an issue back from April - https://www.spinics.net/lists/netfilter/msg58628.html - which was fixed in this commit: http://patchwork.ozlabs.org/patch/1107919/

At a guess, pulling the latest build should fix this.

Merijn added a comment.Nov 5 2019, 6:35 AM

In 1.2.3 build this error does not appear and it seems to work correctly

xrobau added a comment.Nov 5 2019, 8:08 PM

Confirmed fixed in

vyos@mke-fw1:~$ show version
Version:          VyOS 1.2-rolling-201911051339
Built by:         autobuild@vyos.net
Built on:         Tue 05 Nov 2019 13:39 UTC
Build UUID:       3863567b-039d-4fdd-90cc-eda2e1b11bc6
Build Commit ID:  33c865b2ada281
xrobau closed this task as Resolved.Nov 5 2019, 8:18 PM
xrobau claimed this task.

This issue is still present in 1.3-rolling-202001240217

Confirming that I also report this on 1.3-rolling-202001240217. Just upgraded this morning and I see the same unknown layer 3 protocol error as reported.

Confirmed here as well, I had a working config back on 1.2.3 and it broke when I upgraded to 1.3. This is what happens when I try to commit:

vyos@cr01a-vyos# commit
[ service conntrack-sync ]
conntrack-sync error: /etc/init.d/conntrackd failed to start /usr/sbin/conntrackd!

[[service conntrack-sync]] failed
Commit failed
[edit service conntrack-sync]
vyos@cr01a-vyos# sudo journalctl | grep -iP 'conntrack.*error'
Feb 01 02:16:08 cr01a-vyos conntrackd[8927]: [Sat Feb  1 02:16:08 2020] (pid=8927) [ERROR] unknown layer 3 protocol
Feb 01 02:16:08 cr01a-vyos conntrackd[8940]: [Sat Feb  1 02:16:08 2020] (pid=8940) [ERROR] unknown layer 3 protocol
Feb 01 02:16:09 cr01a-vyos conntrackd[8941]: [Sat Feb  1 02:16:09 2020] (pid=8941) [ERROR] unknown layer 3 protocol
Feb 01 02:16:09 cr01a-vyos conntrackd[8943]: [Sat Feb  1 02:16:09 2020] (pid=8943) [ERROR] unknown layer 3 protocol
Feb 01 02:16:09 cr01a-vyos conntrackd[8946]: [Sat Feb  1 02:16:09 2020] (pid=8946) [ERROR] unknown layer 3 protocol
Feb 01 22:04:53 cr01a-vyos conntrackd[10083]: [Sat Feb  1 22:04:53 2020] (pid=10083) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10096]: [Sat Feb  1 22:04:54 2020] (pid=10096) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10098]: [Sat Feb  1 22:04:54 2020] (pid=10098) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10100]: [Sat Feb  1 22:04:54 2020] (pid=10100) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10110]: [Sat Feb  1 22:04:54 2020] (pid=10110) [ERROR] unknown layer 3 protocol
Feb 01 22:07:13 cr01a-vyos conntrackd[10170]: [Sat Feb  1 22:07:13 2020] (pid=10170) [ERROR] unknown layer 3 protocol
Feb 01 22:07:13 cr01a-vyos conntrackd[10183]: [Sat Feb  1 22:07:13 2020] (pid=10183) [ERROR] unknown layer 3 protocol
Feb 01 22:07:14 cr01a-vyos conntrackd[10186]: [Sat Feb  1 22:07:14 2020] (pid=10186) [ERROR] unknown layer 3 protocol
Feb 01 22:07:14 cr01a-vyos conntrackd[10187]: [Sat Feb  1 22:07:14 2020] (pid=10187) [ERROR] unknown layer 3 protocol
Feb 01 22:07:14 cr01a-vyos conntrackd[10196]: [Sat Feb  1 22:07:14 2020] (pid=10196) [ERROR] unknown layer 3 protocol
[edit service conntrack-sync]
vyos@cr01a-vyos# run show ver | grep Ver
Version:          VyOS 1.3-rolling-202001311637

One thing to note, this is broken both on clustered devices and using VRRP.

jjakob added a subscriber: jjakob.Feb 28 2020, 9:57 PM

We don't build conntrack-tools in 1.3 (current/equuleus) any more, upstream Debian Buster conntrack and conntrackd packages are used. So as upstream gets patched, we'll pull in those patches automatically.
If I see things correctly, there are references to conntrack-tools in the build scripts that still need to be removed.

Upstream still hasn't made a release with this patch: https://git.netfilter.org/conntrack-tools/commit/?id=c12fa8df76752b0a011430f069677b52e4dad164
So we could wait on upstream to release it and debian to package it, or build our own as we used to in 1.2.
It would be better to ask upstream to make a release as there's less work for us.

That's bad, because debian stable (=buster) is fixing security bugs only. They will not fix/add this patches to conntrack package, they leave conntrack buggy. So you sould build an own conntrack-tools package for 1.3 too :( If not, vyos will be less good software.

@cpo I think you need to add it to CI in addition to vyos-build

jjakob added a comment.Mar 1 2020, 4:51 PM

https://github.com/jjakob/vyos-build/tree/conntrack-tools-wip builds conntrack-tools from upstream git snapshot 20200301.

You can test it by doing:

git clone https://github.com/jjakob/vyos-build.git
git checkout conntrack-tools-wip
docker build -t vyos-build/current docker
docker run --rm -it -v $(pwd):/vyos -w /vyos vyos-build/current scripts/build-packages -vv -c -b conntrack-tools

then copy the deb packages from packages/ to a running 1.3 system and install them with dpkg -i, or make the iso:

./configure
sudo make iso
xrobau reopened this task as Open.Mon, Mar 16, 1:41 AM
xrobau set Is it a breaking change? to Unspecified (possibly destroys the router).

Reopened, confirmed broken again.

xrobau triaged this task as Normal priority.Mon, Mar 16, 1:43 AM
xrobau changed Version from vyos-1.2.0-rolling+201907210337 to vyos-1.3-rolling-202003151839.
xrobau changed Is it a breaking change? from Unspecified (possibly destroys the router) to Behavior change.

I opened the PR for our custom build of the package in vyos-build as well: https://github.com/vyos/vyos-build/pulls. I was waiting on testing results from anyone, but I went and tested it myself. The basic functionality works, I couldn't test the above bug. If it's merged and the new package build is added to CI, the above debian PR isn't needed (or our custom build isn't).