Hi @srnoth,

can you tell us a bot more of your configuration? Like is PPPoE used?

It would be best if you could share the output of: show configuration commands | strip-private

Flame graph for 1.3-rc4, proxmox/virtio

https://github.com/torvalds/linux/blob/master/arch/x86/lib/memcpy_64.S

@c-po for simplicity I created a barebones vyos VM to route between two subnets.

IPERF3 results for 1.2.7:
18.4 Gbits/sec
ksoftirqd average 1.3% CPU

2.0 RC4:
3.14 Gbits/sec
ksoftirqd average 85% CPU

Bone stock config. Routing is between eth1 and eth2 (pure routing, no NAT):

vyos@vyos:~$ show configuration commands | strip-private
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 address 'xxx.xxx.210.1/24'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth2 address 'xxx.xxx.211.1/24'
set interfaces ethernet eth2 duplex 'auto'
set interfaces ethernet eth2 speed 'auto'
set interfaces loopback lo
set nat source rule 5010 outbound-interface 'eth0'
set nat source rule 5010 translation address 'masquerade'
set system config-management commit-revisions '100'
set system console device ttyS0 speed '9600'
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system name-server 'xxx.xxx.1.1'
set system name-server 'xxx.xxx.8.8'
set system name-servers-dhcp 'eth0'
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system time-zone 'UTC'

@Viacheslav per your suggest in the discussion thread I ran the following:

sudo ethtool -K eth1 gro on
sudo ethtool -K eth1 sg on
sudo ethtool -K eth1 tso on
sudo ethtool -K eth1 lro on

sudo ethtool -K eth2 gro on
sudo ethtool -K eth2 sg on
sudo ethtool -K eth2 tso on
sudo ethtool -K eth2 lro on

Which got me almost the same performance on 2.0 RC4 as on 1.2.7:
17.5 Gbits/sec
ksoftirqd average 3-4%

I do have similar problems with a VyOS VM inside Proxmox.

This VM has a Mellanox Connect-X 3 via PCI-E pass-through receiving full-tables via BGP and a virtio NIC towards other VMs. I can barely get to a few hundred mbit/s. Which is quite a bit low imo for 4 CPU cores of a Xeon X5670.

I toyed around with various offloading options via ethtool -K, it brought the ksoftirqd down a bit, but did not bring much in terms performance.
I also tried to give the VM dedicated CPU cores on the host, hasn't helped a lot.

Meanwhile anohter VM on stock Debian is doing the same throughput (w/o a full table - that's what the VyOS VM is for) - while also doing NAT + wireguard + VXLAN + BATMAN to parts of the traffic.

Let me know if I can do anything to debug this performance issue.

Analyis

An analysis of the code base from VyOS 1.2 -> 1.3 -> 1.4 revealed the following "root-cause"

VyOS 1.2 uses the "old" node.def file format

• Generic Segmentation Offloading
• Generic Receive Offloading

So if any of the above settings is available on the configuration CLI, the node.def file will be executed.

By default, this is not enabled in VyOS 1.2 - but the Linux Kernel enables offloading "under the hood" by default for GRO, GSO... which will boost the performance for users magically.

With the rewrite in VyOS 1.3 of all the interface related code T1579, and especially T1637 this was moved to a new approach. There is now only one handler script (Example for Ethernet interfaces) which is called whenever a user changes something under the interfaces ethernet tree. The Full CLI configuration is assembled by get_interface_dict() - a wrapper for get_config_dict() which abstracts and works for all of our interface types - single source design.

The problem now comes into play when the gathered configuration is actually written to the hardware, as there is no GSO, GRO or foo-offloading setting defined - we behave as instructed and disable the offloading.

So the real bug originates from VyOS 1.2 and the old Vyatta codebase, but the recent XML Python rewrites brought that one up to light.

Solution

A configuration migration script will be provided starting with VyOS 1.3 which will read in the CLI configuration of the ethernet interfaces and if not enabled, will query the adapter if offloading is supported at all, and if so, will enable the CLI nodes.

One might say that this will "blow" the CLI configuration but it only represents the truth - which was masked in VyOS 1.2.