Page MenuHomeVyOS Platform

Update conntrack-sync packages to fix VRRP issues
Closed, ResolvedPublicBUG

Description

When trying to assign conntrack-sync to a new 'high-availability' vrrp group, it says the group does not exist.

Example:

set high-availability vrrp group int1 hello-source-address '10.60.1.3'
set high-availability vrrp group int1 interface 'eth1.200'
set high-availability vrrp group int1 peer-address '10.60.1.2'
set high-availability vrrp group int1 preempt-delay '600'
set high-availability vrrp group int1 priority '200'
set high-availability vrrp group int1 virtual-address '10.60.1.1/24'
set high-availability vrrp group int1 vrid '200'
set service conntrack-sync accept-protocol 'tcp,udp,icmp'
set service conntrack-sync event-listen-queue-size '8'
set service conntrack-sync failover-mechanism vrrp sync-group int1
set service conntrack-sync interface 'eth1.200'
set service conntrack-sync mcast-group '225.0.0.50'
set service conntrack-sync sync-queue-size '8'

When you commit it, you get this error:

vyos@fw1# commit
[ service conntrack-sync ]
conntrack-sync error: VRRP sync-group int1 is not configured

[[service conntrack-sync]] failed
Commit failed
[edit]
vyos@fw1#

I assume it works with the old style of vrrp assigned to an interface, with an explicit sync-group tag, but I didn't test it, as I didn't want to rebuild everything using the older style!

Details

Difficulty level
Hard (possibly days)
Version
vyos-1.3-rolling-202003151839
Why the issue appeared?
Issues in third-party code
Is it a breaking change?
Perfectly compatible

Event Timeline

xrobau created this task.Jul 22 2019, 1:46 PM
xrobau updated the task description. (Show Details)Jul 22 2019, 1:48 PM

Confirmed still present in VyOS 1.3-rolling-201911030242

pasik added a subscriber: pasik.Nov 4 2019, 9:51 AM
Merijn added a subscriber: Merijn.Nov 4 2019, 11:54 AM

You have to add a sync-group.
set high-availability vrrp sync-group intgroup member int1
set service conntrack-sync failover-mechanism vrrp sync-group intgroup

After adding the missing command set high-availability vrrp sync-group sync member int1, we have a new error when starting conntrackd

[Tue Nov 5 00:07:41 2019] (pid=30976) [ERROR] unknown layer 3 protocol

This is an issue back from April - https://www.spinics.net/lists/netfilter/msg58628.html - which was fixed in this commit: http://patchwork.ozlabs.org/patch/1107919/

At a guess, pulling the latest build should fix this.

Merijn added a comment.Nov 5 2019, 6:35 AM

In 1.2.3 build this error does not appear and it seems to work correctly

xrobau added a comment.Nov 5 2019, 8:08 PM

Confirmed fixed in

vyos@mke-fw1:~$ show version
Version:          VyOS 1.2-rolling-201911051339
Built by:         autobuild@vyos.net
Built on:         Tue 05 Nov 2019 13:39 UTC
Build UUID:       3863567b-039d-4fdd-90cc-eda2e1b11bc6
Build Commit ID:  33c865b2ada281
xrobau closed this task as Resolved.Nov 5 2019, 8:18 PM
xrobau claimed this task.

This issue is still present in 1.3-rolling-202001240217

Confirming that I also report this on 1.3-rolling-202001240217. Just upgraded this morning and I see the same unknown layer 3 protocol error as reported.

Confirmed here as well, I had a working config back on 1.2.3 and it broke when I upgraded to 1.3. This is what happens when I try to commit:

vyos@cr01a-vyos# commit
[ service conntrack-sync ]
conntrack-sync error: /etc/init.d/conntrackd failed to start /usr/sbin/conntrackd!

[[service conntrack-sync]] failed
Commit failed
[edit service conntrack-sync]
vyos@cr01a-vyos# sudo journalctl | grep -iP 'conntrack.*error'
Feb 01 02:16:08 cr01a-vyos conntrackd[8927]: [Sat Feb  1 02:16:08 2020] (pid=8927) [ERROR] unknown layer 3 protocol
Feb 01 02:16:08 cr01a-vyos conntrackd[8940]: [Sat Feb  1 02:16:08 2020] (pid=8940) [ERROR] unknown layer 3 protocol
Feb 01 02:16:09 cr01a-vyos conntrackd[8941]: [Sat Feb  1 02:16:09 2020] (pid=8941) [ERROR] unknown layer 3 protocol
Feb 01 02:16:09 cr01a-vyos conntrackd[8943]: [Sat Feb  1 02:16:09 2020] (pid=8943) [ERROR] unknown layer 3 protocol
Feb 01 02:16:09 cr01a-vyos conntrackd[8946]: [Sat Feb  1 02:16:09 2020] (pid=8946) [ERROR] unknown layer 3 protocol
Feb 01 22:04:53 cr01a-vyos conntrackd[10083]: [Sat Feb  1 22:04:53 2020] (pid=10083) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10096]: [Sat Feb  1 22:04:54 2020] (pid=10096) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10098]: [Sat Feb  1 22:04:54 2020] (pid=10098) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10100]: [Sat Feb  1 22:04:54 2020] (pid=10100) [ERROR] unknown layer 3 protocol
Feb 01 22:04:54 cr01a-vyos conntrackd[10110]: [Sat Feb  1 22:04:54 2020] (pid=10110) [ERROR] unknown layer 3 protocol
Feb 01 22:07:13 cr01a-vyos conntrackd[10170]: [Sat Feb  1 22:07:13 2020] (pid=10170) [ERROR] unknown layer 3 protocol
Feb 01 22:07:13 cr01a-vyos conntrackd[10183]: [Sat Feb  1 22:07:13 2020] (pid=10183) [ERROR] unknown layer 3 protocol
Feb 01 22:07:14 cr01a-vyos conntrackd[10186]: [Sat Feb  1 22:07:14 2020] (pid=10186) [ERROR] unknown layer 3 protocol
Feb 01 22:07:14 cr01a-vyos conntrackd[10187]: [Sat Feb  1 22:07:14 2020] (pid=10187) [ERROR] unknown layer 3 protocol
Feb 01 22:07:14 cr01a-vyos conntrackd[10196]: [Sat Feb  1 22:07:14 2020] (pid=10196) [ERROR] unknown layer 3 protocol
[edit service conntrack-sync]
vyos@cr01a-vyos# run show ver | grep Ver
Version:          VyOS 1.3-rolling-202001311637

One thing to note, this is broken both on clustered devices and using VRRP.

jjakob added a subscriber: jjakob.Feb 28 2020, 9:57 PM

We don't build conntrack-tools in 1.3 (current/equuleus) any more, upstream Debian Buster conntrack and conntrackd packages are used. So as upstream gets patched, we'll pull in those patches automatically.
If I see things correctly, there are references to conntrack-tools in the build scripts that still need to be removed.

Upstream still hasn't made a release with this patch: https://git.netfilter.org/conntrack-tools/commit/?id=c12fa8df76752b0a011430f069677b52e4dad164
So we could wait on upstream to release it and debian to package it, or build our own as we used to in 1.2.
It would be better to ask upstream to make a release as there's less work for us.

That's bad, because debian stable (=buster) is fixing security bugs only. They will not fix/add this patches to conntrack package, they leave conntrack buggy. So you sould build an own conntrack-tools package for 1.3 too :( If not, vyos will be less good software.

@cpo I think you need to add it to CI in addition to vyos-build

jjakob added a comment.Mar 1 2020, 4:51 PM

https://github.com/jjakob/vyos-build/tree/conntrack-tools-wip builds conntrack-tools from upstream git snapshot 20200301.

You can test it by doing:

git clone https://github.com/jjakob/vyos-build.git
git checkout conntrack-tools-wip
docker build -t vyos-build/current docker
docker run --rm -it -v $(pwd):/vyos -w /vyos vyos-build/current scripts/build-packages -vv -c -b conntrack-tools

then copy the deb packages from packages/ to a running 1.3 system and install them with dpkg -i, or make the iso:

./configure
sudo make iso
xrobau reopened this task as Open.Mar 16 2020, 1:41 AM
xrobau set Is it a breaking change? to Unspecified (possibly destroys the router).

Reopened, confirmed broken again.

xrobau triaged this task as Normal priority.Mar 16 2020, 1:43 AM
xrobau changed Version from vyos-1.2.0-rolling+201907210337 to vyos-1.3-rolling-202003151839.
xrobau changed Is it a breaking change? from Unspecified (possibly destroys the router) to Behavior change.

I opened the PR for our custom build of the package in vyos-build as well: https://github.com/vyos/vyos-build/pulls. I was waiting on testing results from anyone, but I went and tested it myself. The basic functionality works, I couldn't test the above bug. If it's merged and the new package build is added to CI, the above debian PR isn't needed (or our custom build isn't).

jjakob added a comment.EditedApr 8 2020, 10:28 PM

There was a new upstream release 1.4.6 7 days ago, but that shouldn't make it to debian stable (buster). Only the patch done by elbandi via PR could get released as 1.4.5-3, but it hasn't been yet. We could make a backport of 1.4.6 into buster-backports and add a custom apt pin for the package. (I'd rather not go the backport route, as that means the backporter needs to always update the upload for security fixes, rather I'd add all patches for bugs into 1.4.5 for buster and ask for a new buster release).

The new conntract package depend in newer libnetfilter. but you dont need to rebuild the package, just download the debs.

It's best if we just use packages targeted for buster, not another debian release. I suggest you create PRs for all pathches needed (in addition to the one you already submitted) in Debian's PTS for buster's conntrack-tools, and then ask them to make a new release with those patches included.

xrobau removed xrobau as the assignee of this task.Jun 14 2020, 8:02 PM
c-po added a subscriber: c-po.Jun 14 2020, 8:10 PM
This comment was removed by c-po.
c-po claimed this task.Jun 14 2020, 8:19 PM
c-po added a comment.Jun 20 2020, 11:58 AM

Picking up on the build issue

c-po renamed this task from conntrack-sync no longer works with VRRP/high-availability to Update conntrack-sync packages to fix VRRP issues.Jun 20 2020, 11:59 AM
c-po changed the task status from Open to In progress.
c-po changed Difficulty level from Unknown (require assessment) to Hard (possibly days).
c-po changed Why the issue appeared? from Will be filled on close to Issues in third-party code.
c-po changed Is it a breaking change? from Behavior change to Perfectly compatible.
c-po removed projects: vyatta-vrrp, conntrack-tools.
c-po changed the task status from In progress to Needs testing.Jun 23 2020, 7:19 PM
c-po added a comment.Jun 25 2020, 6:42 PM

Updated versions:

  • libnetfilter-conntrack 1.0.8-1
  • conntrack 1.4.6

Will be included in the next rolling release. Closing this!

c-po closed this task as Resolved.Jun 25 2020, 6:42 PM
c-po moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.Aug 3 2020, 2:15 PM