https://phabricator.vyos.net/T935 Here’s the same thing happening in the past. I think it was resolved by doing kernel updates? Can someone do a kernel update in the rolling build?

There is no newer kernel then 4.19.124 on the 4.19x train. Newer Kernels do not work as the out-of-tree Intel drivers for the NICs and QAT won‘t compile for Kernel >5.3 and that is bot an LTS one.

In T2505#64889, @c-po wrote:

There is no newer kernel then 4.19.124 on the 4.19x train. Newer Kernels do not work as the out-of-tree Intel drivers for the NICs and QAT won‘t compile for Kernel >5.3 and that is bot an LTS one.

So if this is a kernel issue I should have the same problem with the same kernel under Debian 10 right

If this can be solved by a kernel update, there was talk about maybe having different build "flavors" in the past - one with all the hardware nic drivers, one without. The minimal image could then have the latest (5.x) kernel.
There's T2085 which prevents us from testing any newer kernel ourselves as it's built by Jenkinsfiles in the CI, we'd need to manually do the steps the CI does to build a kernel. I proposed a shared script solution for these repositories in that task that could be called from both the CI and vyos-build, this would allow anyone to build all packages, including the kernel, through vyos-build, just for cases like this.

@Sonicbx @jjakob I also created https://phabricator.vyos.net/T2504 - I think we duplicated the issue here. You can close whichever issue you want.

In T2505#64896, @jjakob wrote:

If this can be solved by a kernel update, there was talk about maybe having different build "flavors" in the past - one with all the hardware nic drivers, one without. The minimal image could then have the latest (5.x) kernel.
There's T2085 which prevents us from testing any newer kernel ourselves as it's built by Jenkinsfiles in the CI, we'd need to manually do the steps the CI does to build a kernel. I proposed a shared script solution for these repositories in that task that could be called from both the CI and vyos-build, this would allow anyone to build all packages, including the kernel, through vyos-build, just for cases like this.

vyos-build-kernel comes with dedicated build scripts for some time now - this should no longer be an issue. I do not support the different falvour idea as it will be a nightmare to maintain. Just give it some time when Intel decides to update their stuff.

I replaced the distributed guest utilities (vyos-xe-guest-utilities) with the ones that come with xcp-ng. But this changed nothing regarding the packet loss. Tho, now they get properly recognized by xcp-ng :-)

Does anyone have some idea on how to test with different kernels? For now this is a deal breaker while using the 1.3.x branch. Tho I would really love to keep using bleeding edge in order to help testing things :-)

I don't see problems with Debian Buster, kernel "4.19.0-9"
Need to check this patch. Ref. https://patchwork.kernel.org/patch/9293785/

With 4.19.123-amd64-vyos I am having the same problems. I would assume, that the patch from 2016 is already in this kernel?

Also happens with 4.19.131-amd64-vyos - I guess that patch mentioned by @Viacheslav is either not included or not solving the problem.

@fetzerms the mentioned patch is not included in the mainline kernel!

@c-po Thank you for clarifying. I guess I misinterpreted what i read on patchwork. I'd be eager to test a kernel with the patch!

Oh - I'm sorry - I mixed up the lines in the kernel. The patch is actually in VyOS.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=fd07160bb7180cdd0afeb089d8cdfd66002f17e6

(21:51) cpo lnx01:~/vyos-build/packages/linux-kernel/linux # git tag --contains fd07160bb7180 | grep 4.19.131
v4.19.131

Hi,
I've been experiencing issues similar to what is being mentioned here. From my experience, I cant find any indication that the packet loss is related to packet size, it seems random.
Im running 1.4-rolling-202101270854 on XCP-NG 8.1.0. Confirmed on both Intel(R) Atom(TM) CPU C2758 and Intel(R) Xeon(R) CPU E5-2630 v3 machines. HW-checksumming turned off in XCP-NG makes no difference.

As mentioned by others, ethtool shows almost no output for the ethernet devices, and they are barely mentioned in dmesg.

# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500  7946569      0    278 0      14275236      0 162847      0 BMRU
eth1      1500  8859462      0      0 0      11365142      0 159164      0 BMRU

# dmesg | grep vif
[    2.817709] xenbus_probe_frontend: Device with no driver: device/vif/0
[    2.825728] xenbus_probe_frontend: Device with no driver: device/vif/1

# ethtool eth0
Settings for eth0:
	Link detected: yes

Im also not able to set smp_affinity on this release. Not sure if that is a symptom of something else (this was possible in 1.3 rolling)

I will gladly do any form of testing. Im not too experienced with patching kernels, but I'm not afraid of trying if that is what is needed.

We don't find any solution right now. We test with different kernels/offloads/sysctl params but without result.
Additional topic
https://xcp-ng.org/forum/topic/2956/tx-dropped-in-pv-vm

Looks like upgrading to 1.4-rolling-202102060218 finally fixed the issue for me. Now netstat shows 0 dropped TX, and my SSH connections are much more responsive.

Kernel 5.10.12

So after a week of running, and comparing to performance with the LTS, i know that something is wrong.

In this latest 1.4 rolling using kernel 5.10.12, and some of the earlier versions (that experienced packet loss), running top shows that ksoftirqd/0 runs at around 100% when saturating the link. For one of my machines, this means that packet loss starts appearing at 250mb/s, as ksoftirqd/0 reaches 100% cpu use.

Downgrading to the 1.2.6 LTS gives me almost twice the performance with the exact same config. ksoftirqd barely reaches 1% cpu load.

I start to wonder how the VyOS XCP-NG partnership is working, when Xen support is lacking so much?

@mathiashedberg could you try and enable RPS set interfaces ethernet eth0 offload rps and see if this does any good on utilisation / drop rate? I had a similar issue with a PPPoE link which behaved super bad under preasure.

@c-po That seemed to more than double my RX!

Here are some quick iperf3 results from vyos to a host connected over DAC SFP+

Baseline 1.2.6 LTS:

TX: 8.59 Gbits/sec    Retr: 0
RX: 4.24 Gbits/sec    Retr: 43

1.4-rolling-202102060218:

Without offloading:
TX: 3.16 Gbits/sec    Retr: 0
RX: 1.26 Gbits/sec    Retr: 86

With offloading:
TX: 3.00 Gbits/sec    Retr: 0
RX: 3.64 Gbits/sec    Retr: 487

netstat -i shows me 0 TX/RX errors both with and without offloading.

Im quite happy with this result as ksoftirqd cpu use is much less than before, and i can now finally saturate my wan link.

@Sonicbx Is it an actual bug?

In T2505#157736, @Viacheslav wrote:

@Sonicbx Is it an actual bug?

I haven't tried testing this in over 2 years. I no longer have a place to run XCPNG but I do have a HyperV host I can try this out on. I'll get back to you.

@Sonicbx As I remember, HyperV is not affected.
But thanks anyway,

What is the resolution? How was it resolved? @Viacheslav

There have been no reports since 2021

If it is still a bug, we'll reopen it. @Sonicbx can you re-check with the 1.3.x or 1.4-rc3?