WAN load-balancing can't flush connections when conntrack-sync is enabled
Open, NormalPublicBUG
Actions

Assigned To

Authored By

	zsdc
	Mar 21 2019, 5:22 PM

Description

When flush-connections is enabled in load-balancing, then when interface state is changing daemon wan_lb execute:

conntrack -F
conntrack -F expect

But, when conntrackd is running, the first command can't finish up and hangs. As I see, conntrack sending the command to a kernel, but don't receive any answers: Also, this leads to a continuous CPU usage by conntrack and conntrackd.

root      7792 81.7  1.0  24416  5332 ?        S<s  17:00   0:33 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf
root      7829  7.0  0.1  12652   888 pts/1    R+   17:00   0:02 conntrack -F

This problem breaks load-balancing functional.
Instead of flushing table, we can delete its content. This works without problems. I propose to change command to the followed to avoid problems:

conntrack -D

Otherwise, we need to block the ability to enable both options at the same time.

Details

Difficulty level: Unknown (require assessment)
Version: 1.2.0-rolling+201903210337
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Unspecified (please specify)

Event Timeline

zsdc created this task.Mar 21 2019, 5:22 PM

pasik added a subscriber: pasik.Mar 23 2019, 9:20 PM

syncer changed the task status from Open to Confirmed.Apr 17 2019, 7:41 PM

syncer assigned this task to hagbard.

syncer triaged this task as Normal priority.

syncer edited projects, added VyOS 1.3 Equuleus, VyOS 1.2 Crux (VyOS 1.2.2); removed VyOS 1.2 Crux.

@zsdc Can you please share some config data or clarify what you mean? thx

hagbard changed the task status from Confirmed to On hold.Apr 29 2019, 4:19 PM

hagbard reassigned this task from hagbard to zsdc.May 13 2019, 4:19 PM

hagbard added a subscriber: hagbard.

dmbaturin edited projects, added VyOS 1.2 Crux (VyOS 1.2.3); removed VyOS 1.2 Crux (VyOS 1.2.2).Jul 10 2019, 6:10 PM

@zsdc can you follow up on this

@syncer
Sorry to dredge up an old bug, but I believe I've hit this today on 1.2.7-LTS myself. Per @zsdc's original description, It seems that when you configure:

service {
    conntrack-sync {

and also configure:

load-balancing {
    wan {
        flush-connections

conntrackd will peg a CPU at 100% utilization immediately after commit, forever, and this also prevents wan load-balancing from starting. The only way I could see to recover was to delete the load-balance flush-connections configuration entry and reboot, which took me a minute, due to the system being very slow from a locked-up CPU core.

Wan load-balancing and conntrack-sync seem to cooperate together fine as long as you do *not* enable flush-connections. But, I *really* would also like flush-connections to work along with my conntrack-sync. Is it possible?

dmbaturin added a project: test.Jul 29 2021, 12:20 PM

dmbaturin set Is it a breaking change? to Unspecified (possibly destroys the router).

syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.Nov 6 2021, 11:25 AM

masterit added a subscriber: masterit.May 25 2022, 11:38 PM

angelnu added a subscriber: angelnu.Jun 16 2022, 10:08 AM

I have also hit this into the latest rolling version:

vyos@vyos:~$ show version

Version:          VyOS 1.4-rolling-202206130217
Release train:    sagitta

Built by:         [email protected]
Built on:         Mon 13 Jun 2022 02:17 UTC
Build UUID:       6c04f7fd-b1c8-4e5d-ab90-310ccfb016d5
Build commit ID:  001451a9c514e7

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:     
Hardware UUID:    70fb4a55-e1f4-4593-83d0-189dcd06ec78

Copyright:        VyOS maintainers and contributors

I have to remove flush-connections as suggested by @klipz but this results on my slow backup internet connection being used longer than needed after the primary is recovered.

Viacheslav added a project: VyOS 1.4 Sagitta.Jul 11 2022, 2:58 PM

PR https://github.com/vyos/vyatta-wanloadbalance/pull/13

syncer edited projects, added VyOS 1.3 Equuleus (1.3.3); removed VyOS 1.3 Equuleus (1.3.0).Aug 29 2022, 7:05 AM

Viacheslav changed the task status from On hold to Needs testing.Jan 9 2023, 8:32 AM

Harliff added a subscriber: Harliff.Apr 2 2023, 4:06 PM

syncer edited projects, added VyOS 1.3 Equuleus (1.3.4); removed VyOS 1.3 Equuleus (1.3.3).Jul 12 2023, 9:44 PM

syncer edited projects, added VyOS 1.3 Equuleus (1.3.5); removed VyOS 1.3 Equuleus (1.3.4).Aug 25 2023, 9:31 PM

Viacheslav moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.Aug 28 2023, 10:08 AM

@Viacheslav will you backport this to 1.3 ?