Page MenuHomeVyOS Platform

Performance Degradation 1.2 --> 1.3 | High ksoftirqd CPU usage
Closed, ResolvedPublicBUG

Description

There appears to be a bug in both version 1.3 and 1.4 that results in massive CPU interrupts (ksoftirqd/x) under high throughput, and thus a significant reduction in throughput capabilities over what was possible on 1.2.x.

Most extreme example for me is a VYOS VM running on Proxmox:
v1.2.7:
2.90 Gbits/sec
ksoftirqd/0 using ~1% CPU
Likely limited by my host

v1.3.0-rc4:
306 Mbits/sec
ksoftirqd/0 using ~98% CPU

This bug is also present on 1.3 rolling and 1.4 rolling as of yesterday, and was also present when I tested on bare metal (although less obvious as the total throughput was already limited to the 1GB NICs):
v1.2.7:
938 Mbits/sec
ksoftirqd/0 using ~3% CPU

v1.3.0-rc4:
905 Mbits/sec
ksoftirqd/0 using ~95% CPU

See the below thread for more discussion:
https://forum.vyos.io/t/1-3-rc1-performance-regression-high-cpu-usage/6749/24

Details

Difficulty level
Hard (possibly days)
Version
1.3.0-rc4
Why the issue appeared?
Design mistake
Is it a breaking change?
Config syntax change (migratable)
Issue type
Bug (incorrect behavior)

Event Timeline

c-po added a subscriber: c-po.

Hi @srnoth,

can you tell us a bot more of your configuration? Like is PPPoE used?

It would be best if you could share the output of: show configuration commands | strip-private

@c-po for simplicity I created a barebones vyos VM to route between two subnets.

IPERF3 results for 1.2.7:
18.4 Gbits/sec
ksoftirqd average 1.3% CPU

2.0 RC4:
3.14 Gbits/sec
ksoftirqd average 85% CPU

Bone stock config. Routing is between eth1 and eth2 (pure routing, no NAT):

vyos@vyos:~$ show configuration commands | strip-private
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 address 'xxx.xxx.210.1/24'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth2 address 'xxx.xxx.211.1/24'
set interfaces ethernet eth2 duplex 'auto'
set interfaces ethernet eth2 speed 'auto'
set interfaces loopback lo
set nat source rule 5010 outbound-interface 'eth0'
set nat source rule 5010 translation address 'masquerade'
set system config-management commit-revisions '100'
set system console device ttyS0 speed '9600'
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system name-server 'xxx.xxx.1.1'
set system name-server 'xxx.xxx.8.8'
set system name-servers-dhcp 'eth0'
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system time-zone 'UTC'

@Viacheslav per your suggest in the discussion thread I ran the following:

sudo ethtool -K eth1 gro on
sudo ethtool -K eth1 sg on
sudo ethtool -K eth1 tso on
sudo ethtool -K eth1 lro on

sudo ethtool -K eth2 gro on
sudo ethtool -K eth2 sg on
sudo ethtool -K eth2 tso on
sudo ethtool -K eth2 lro on

Which got me almost the same performance on 2.0 RC4 as on 1.2.7:
17.5 Gbits/sec
ksoftirqd average 3-4%

I do have similar problems with a VyOS VM inside Proxmox.

This VM has a Mellanox Connect-X 3 via PCI-E pass-through receiving full-tables via BGP and a virtio NIC towards other VMs. I can barely get to a few hundred mbit/s. Which is quite a bit low imo for 4 CPU cores of a Xeon X5670.

I toyed around with various offloading options via ethtool -K, it brought the ksoftirqd down a bit, but did not bring much in terms performance.
I also tried to give the VM dedicated CPU cores on the host, hasn't helped a lot.

Meanwhile anohter VM on stock Debian is doing the same throughput (w/o a full table - that's what the VyOS VM is for) - while also doing NAT + wireguard + VXLAN + BATMAN to parts of the traffic.

Let me know if I can do anything to debug this performance issue.

c-po changed the task status from Open to In progress.Sun, Aug 29, 7:39 PM
c-po claimed this task.
c-po triaged this task as High priority.
c-po changed Difficulty level from Unknown (require assessment) to Hard (possibly days).
c-po changed Why the issue appeared? from Will be filled on close to Design mistake.
c-po changed Is it a breaking change? from Unspecified (possibly destroys the router) to Config syntax change (migratable).
c-po set Issue type to Bug (incorrect behavior).

Analyis

An analysis of the code base from VyOS 1.2 -> 1.3 -> 1.4 revealed the following "root-cause"

VyOS 1.2 uses the "old" node.def file format

So if any of the above settings is available on the configuration CLI, the node.def file will be executed.

By default, this is not enabled in VyOS 1.2 - but the Linux Kernel enables offloading "under the hood" by default for GRO, GSO... which will boost the performance for users magically.

With the rewrite in VyOS 1.3 of all the interface related code T1579, and especially T1637 this was moved to a new approach. There is now only one handler script (Example for Ethernet interfaces) which is called whenever a user changes something under the interfaces ethernet tree. The Full CLI configuration is assembled by get_interface_dict() - a wrapper for get_config_dict() which abstracts and works for all of our interface types - single source design.

The problem now comes into play when the gathered configuration is actually written to the hardware, as there is no GSO, GRO or foo-offloading setting defined - we behave as instructed and disable the offloading.

So the real bug originates from VyOS 1.2 and the old Vyatta codebase, but the recent XML Python rewrites brought that one up to light.

Solution

A configuration migration script will be provided starting with VyOS 1.3 which will read in the CLI configuration of the ethernet interfaces and if not enabled, will query the adapter if offloading is supported at all, and if so, will enable the CLI nodes.

One might say that this will "blow" the CLI configuration but it only represents the truth - which was masked in VyOS 1.2.

c-po changed the task status from In progress to Needs testing.Mon, Aug 30, 7:46 PM
c-po added a project: VyOS 1.4 Sagitta.
c-po moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.