Page MenuHomeVyOS Platform

conntrack-tools flooding logs
Needs testing, Requires assessmentPublic

Description

Hi,

I was running 1.2.5 and have upgraded to 1.2.6-epa1.
I have not changed any configuration, but now my logs are filled with the following:

Aug 16 08:33:50 ferrari conntrack-tools[6386]: tcp      6 src=37.48.65.66 dst=202.137.243.17 sport=46330 dport=49371 src=192.168.0.5 dst=37.48.65.66 sport=49371 dport=46330 [ASSURED]
Aug 16 08:33:50 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.104 dst=216.58.199.46 sport=59625 dport=443 src=216.58.199.46 dst=202.137.243.17 sport=443 dport=59625 [ASSURED]
Aug 16 08:33:51 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.252 dst=192.168.10.2 sport=59104 dport=161 src=192.168.10.2 dst=192.168.0.252 sport=161 dport=59104 [ASSURED]
Aug 16 08:33:51 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.252 dst=192.168.10.2 sport=59122 dport=161 src=192.168.10.2 dst=192.168.0.252 sport=161 dport=59122 [ASSURED]
Aug 16 08:33:51 ferrari conntrack-tools[6386]: tcp      6 src=184.171.210.143 dst=202.137.243.17 sport=33325 dport=49371 src=192.168.0.5 dst=184.171.210.143 sport=49371 dport=33325 [ASSURED]
Aug 16 08:33:52 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.106 dst=192.168.10.2 sport=43479 dport=993 src=192.168.10.2 dst=192.168.0.106 sport=993 dport=43479 [ASSURED]
Aug 16 08:33:52 ferrari conntrack-tools[6386]: tcp      6 src=65.60.150.40 dst=202.137.243.17 sport=42888 dport=49371 src=192.168.0.5 dst=65.60.150.40 sport=49371 dport=42888 [ASSURED]
Aug 16 08:33:52 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.5 dst=37.48.65.66 sport=49086 dport=80 src=37.48.65.66 dst=202.137.243.17 sport=80 dport=49086 [ASSURED]
Aug 16 08:33:54 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.7 dst=34.206.114.4 sport=44826 dport=443 src=34.206.114.4 dst=202.137.243.17 sport=443 dport=44826 [ASSURED]
Aug 16 08:33:54 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.5 dst=37.48.65.66 sport=49084 dport=80 src=37.48.65.66 dst=202.137.243.17 sport=80 dport=49084 [ASSURED]
Aug 16 08:33:55 ferrari conntrack-tools[6386]: tcp      6 src=37.48.65.66 dst=202.137.243.17 sport=46328 dport=49371 src=192.168.0.5 dst=37.48.65.66 sport=49371 dport=46328 [ASSURED]
Aug 16 08:33:57 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.106 dst=192.168.10.2 sport=43478 dport=993 src=192.168.10.2 dst=192.168.0.106 sport=993 dport=43478 [ASSURED]
Aug 16 08:33:57 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.120 dst=216.58.199.46 sport=12526 dport=443 src=216.58.199.46 dst=202.137.243.17 sport=443 dport=12526 [ASSURED]
Aug 16 08:33:57 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.5 dst=37.48.65.66 sport=49088 dport=80 src=37.48.65.66 dst=202.137.243.17 sport=80 dport=49088 [ASSURED]

I have two routers in my small network, they use VRRP and conntrack-sync between them so that on failover state can be maintained.
Config of my conntrack sync on my primary router (the one from which the logs above are collected)

[edit service conntrack-sync]
tim@ferrari# show
 accept-protocol tcp,udp,icmp
 event-listen-queue-size 8
 expect-sync all
 failover-mechanism {
     vrrp {
         sync-group failover-group
     }
 }
 interface eth1 {
 }
 mcast-group 225.0.0.50
 sync-queue-size 8
[edit service conntrack-sync]

There are so many conntrack logs that "show log" is essentially useless unless I do " show log | no-match conntrack-tools"

Thanks!

Details

Difficulty level
Unknown (require assessment)
Version
1.2.6-epa1
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change

Event Timeline

tjh created this task.Aug 15 2020, 8:42 PM
tjh created this object in space S1 VyOS Public.
pasik added a subscriber: pasik.Aug 16 2020, 6:27 PM
tjh added a comment.Aug 19 2020, 11:44 PM

So I fixed this on my setup by kill -9 conntrackd
and then sudo /etc/init.d/conntrackd start

It seems that maybe on first boot (maybe from an upgrade, which this box has been) that there's some logging going on?

It seems to have fixed it.
Prior to doing the above, I'd also turned off protocol debug in syslog, which I realised was on, but even restarting syslog didn't stop the flood of logs.

(This is what was turned on when the box booted)

global {
    facility all {
        level info
    }
    facility protocols {
        level debug
    }
}

So I don't know if it's something where the config didn't honour the disable of protocol debug, or if conntrackd didn't see the updated disable of protocol debug until it was killed, or something else that's setup on a first time bootup since a 1.2.5-1.2.6 upgrade, but one of those things has resolved the problem.

syncer changed the task status from Open to Needs testing.Sep 8 2020, 10:42 PM
tjh added a comment.EditedSep 23 2020, 10:00 AM

So I just hit this bug again upgrading from 1.2.6-epa1 to 1.2.6.

This time I checked further and I found this when checking what processes were running:

root      7161  0.2  0.4  19784  2268 ?        Ss   16:40   0:01 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf -d
root      8478  0.8  1.0  26648  5276 ?        S<s  16:40   0:03 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf

On bootup, I had two conntrackd's running, one of which had "-d" appended to it.

I thought at first -d meant "debug" but it actually means "Run in daemon mode"

Anyway, the same fix as previous was applied - I killed both conntrackd's and then restarted conntrackd.

Now I'm not being spammed with logging.

Something isn't right when restart VyOS.

tjh added a comment.Sep 23 2020, 5:19 PM

Additionally, it only happens after a system image upgrade - it doesn't seem to happen if you reboot again after that.