Page MenuHomeVyOS Platform

NIC flap at any interface change
On hold, HighPublicBUG

Description

HW: X710-DA2, installed directly, no hypervisor etc.

Hi

I have a problem where any interface change, even description, results in the nic being brought down and then up again. Possibly being related to not being able to set flowcontrol.Unfortunately I had to rollback to 1.2.7 on the specified server so I'm unable to debug any further.

Here is some logs,

zib@vyos# set interfaces ethernet eth5 description Test
[edit]
zib@vyos# commit
could not set flowcontrol for eth5
[edit]
zib@vyos#


May 3 08:16:11 vyos vyos-configd[757]: config session pid is 5761
May 3 08:16:11 vyos vyos-configd[757]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/interfaces-ethernet.pyVYOS_TAGNODE_VALUE=eth5"}
May 3 08:16:11 vyos kernel: [ 863.781620] i40e 0000:05:00.1 eth5: To change autoneg please use: ethtool -s <dev> autoneg <on|off>
May 3 08:16:12 vyos kernel: [ 863.817177] i40e 0000:05:00.1 eth5: NIC Link is Down
May 3 08:16:12 vyos netplugd[1091]: eth5: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[1091]: eth5.58: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[5926]: /etc/netplug/netplug eth5 out -> pid 5926
May 3 08:16:12 vyos netplugd[1091]: eth5.80: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[1091]: eth5.81: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[5927]: /etc/netplug/netplug eth5.58 out -> pid 5927
May 3 08:16:12 vyos netplugd[5928]: /etc/netplug/netplug eth5.80 out -> pid 5928
May 3 08:16:12 vyos netplugd[5929]: /etc/netplug/netplug eth5.81 out -> pid 5929
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos netplugd[1091]: eth5.81: state OUTING pid 5929 exited status 0
May 3 08:16:12 vyos netplugd[1091]: eth5: state OUTING pid 5926 exited status 0
May 3 08:16:12 vyos netplugd[1091]: eth5.58: state OUTING pid 5927 exited status 0
May 3 08:16:12 vyos netplugd[1091]: eth5.80: state OUTING pid 5928 exited status 0
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos vyos-configd[757]: Sending response 1
May 3 08:16:12 vyos systemd[1]: opt-vyatta-config-tmp-new_config_5761.mount: Succeeded.
May 3 08:16:12 vyos systemd[3195]: opt-vyatta-config-tmp-new_config_5761.mount: Succeeded.
May 3 08:16:13 vyos kernel: [ 864.961037] i40e 0000:05:00.1 eth5: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
May 3 08:16:13 vyos netplugd[1091]: eth5: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[1091]: eth5.58: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[1091]: eth5.80: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[6018]: /etc/netplug/netplug eth5 in -> pid 6018
May 3 08:16:13 vyos netplugd[6019]: /etc/netplug/netplug eth5.58 in -> pid 6019
May 3 08:16:13 vyos netplugd[1091]: eth5.81: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[6020]: /etc/netplug/netplug eth5.80 in -> pid 6020
May 3 08:16:13 vyos netplugd[6021]: /etc/netplug/netplug eth5.81 in -> pid 6021
May 3 08:16:13 vyos commit: Successful change to active configuration by user zib on /dev/pts/2

Details

Difficulty level
Normal (likely a few hours)
Version
vyos-1.3-rolling-202105011026-amd64 and vyos-1.4-rolling-202105020417-amd64
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Event Timeline

set interfaces ethernet eth0 hw-id 'ac:1f:6b:76:95:98'
set interfaces ethernet eth1 hw-id 'ac:1f:6b:76:95:99'
set interfaces ethernet eth2 hw-id 'ac:1f:6b:76:98:7a'
set interfaces ethernet eth3 hw-id 'ac:1f:6b:76:98:7b'
set interfaces ethernet eth4 hw-id '3c:fd:fe:dd:bd:88'
set interfaces ethernet eth5 hw-id '3c:fd:fe:dd:bd:89'
set interfaces ethernet eth5 vif 58 address 'x.x.x.x'
set interfaces ethernet eth5 vif 80 address 'x.x.x.x'
set interfaces ethernet eth5 vif 80 ip ospf cost '500'
set interfaces ethernet eth5 vif 80 ip ospf dead-interval '30'
set interfaces ethernet eth5 vif 81 address 'x.x.x.x'
set interfaces ethernet eth5 vif 81 ip ospf cost '1'
set interfaces ethernet eth5 vif 81 ip ospf dead-interval '30'
set interfaces ethernet eth5 vif 81 ip ospf network 'point-to-point'

c-po changed the task status from Open to In progress.Tue, May 4, 5:56 PM
c-po claimed this task.

I have boiled down your commands to the following:

set interfaces ethernet eth2 vif 58 address '10.0.58.254/24'
set interfaces ethernet eth2 vif 80 address '10.0.80.254/24'
set interfaces ethernet eth2 vif 81 address '10.0.81.254/24'

and after applying this interface configuration I changed the description of eth2 set interfaces ethernet eth2 description test2 which resulted in the following happening on the system:

May  4 19:50:19 vyos DEBUG/IFCONFIG returned (err):
May  4 19:50:19 vyos Cannot change udp-fragmentation-offload
May  4 19:50:19 vyos isisd[889]: circuit already connected
May  4 19:50:19 vyos isisd[889]: circuit already connected
May  4 19:50:19 vyos isisd[889]: circuit already connected
May  4 19:50:20 vyos isisd[889]: circuit already connected
May  4 19:50:20 vyos systemd[1824]: opt-vyatta-config-tmp-new_config_4244.mount: Succeeded.
May  4 19:50:20 vyos systemd[1]: opt-vyatta-config-tmp-new_config_4244.mount: Succeeded.
May  4 19:50:21 vyos commit: Successful change to active configuration by user cpo on /dev/pts/0

I assume the problem could be with the famous i40e NIC driver.

Please re-run your commands but before typing commit enable debugging by running: touch /tmp/vyos.ifconfig.debug.

All was done using: VyOS 1.4 rolling 202105020417

c-po changed the task status from In progress to On hold.Tue, May 4, 7:06 PM
c-po triaged this task as High priority.
c-po changed Difficulty level from Unknown (require assessment) to Normal (likely a few hours).

vyos@vyos# set interfaces ethernet eth3 description Test
[edit]
vyos@vyos# commit
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT ","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu ":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_ id":"3cfdfeddbd89","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT ","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu ":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_ id":"3cfdfeddbd89","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ethtool --show-pause eth3'
DEBUG/IFCONFIG returned (out):
Pause parameters for eth3:
Autonegotiate: off
RX: off
TX: off
DEBUG/IFCONFIG cmd 'ethtool --pause eth3 autoneg on tx on rx on'
could not set flowcontrol for eth3
DEBUG/IFCONFIG cmd 'ethtool -K eth3 gro off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 gso off'
DEBUG/IFCONFIG write '0' > '/sys/class/net/eth3/queues/rx-0/rps_cpus'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 sg off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 tso off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 ufo off'
DEBUG/IFCONFIG cmd 'ethtool eth3'
DEBUG/IFCONFIG returned (out):
Settings for eth3:

Supported ports: [ FIBRE ]
Supported link modes:   1000baseX/Full
                        10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes:  1000baseX/Full
                        10000baseSR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
                       drv probe link
Link detected: yes

DEBUG/IFCONFIG cmd 'ethtool -s eth3 autoneg on'
{'description': 'Test',
'duplex': 'auto',
'ifname': 'eth3',
'ip': {'arp_cache_timeout': '30'},
'mtu': '1500',
'speed': 'auto',
'vif': {'80': {'address': ['1.3.3.7/29'],

'ip': {'arp_cache_timeout': '30'},
'mtu': '1500'}}}

DEBUG/IFCONFIG cmd 'ip link set dev eth3 alias "Test"'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/link_filter'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhclient@eth3.service'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhcp6c@eth3.service'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEF AULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min _mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_p ort_id":"3cfdfeddbd89","ifalias":"Test","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3 nomaster'
DEBUG/IFCONFIG write '30000' > '/proc/sys/net/ipv4/neigh/eth3/base_reachable_time_ms'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/arp_filter'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_accept'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_announce'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_ignore'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/proxy_arp'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/proxy_arp_pvlan'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/forwarding'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/rp_filter'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/forwarding'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/accept_ra'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv6/conf/eth3/autoconf'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/dad_transmits'
DEBUG/IFCONFIG cmd 'ip link set dev eth3 mtu 1500'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEF AULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min _mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_p ort_id":"3cfdfeddbd89","ifalias":"Test","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'xdp_loader -d eth3 -U --auto-mode'
DEBUG/IFCONFIG returned (out):
INFO: xdp_link_detach() no curr XDP prog on ifindex:5
DEBUG/IFCONFIG cmd 'tc qdisc del dev eth3 handle ffff: ingress 2> /dev/null;tc qdisc del dev eth3 handle 1: root prio 2> /dev/null;set $?=0'
DEBUG/IFCONFIG cmd 'ip link set dev eth3 up'
{'address': ['1.3.3.7/29'],
'ifname': 'eth3.80',
'ip': {'arp_cache_timeout': '30'},
'mtu': '1500'}
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 alias ""'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/link_filter'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhclient@eth3.80.service'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhcp6c@eth3.80.service'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3.80'
DEBUG/IFCONFIG returned (out):
[{"ifindex":8,"link":"eth3","ifname":"eth3.80","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"noqueue","operstate": "LOWERLAYERDOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:f f:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":65535,"linkinfo":{"info_kind":"vlan","info_data":{"protocol":"802.1Q","id":80,"flags":["RE ORDER_HDR"]}},"inet6_addr_gen_mode":"none","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 nomaster'
DEBUG/IFCONFIG write '30000' > '/proc/sys/net/ipv4/neigh/eth3.80/base_reachable_time_ms'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/arp_filter'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_accept'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_announce'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_ignore'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/proxy_arp'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/proxy_arp_pvlan'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/forwarding'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/rp_filter'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/forwarding'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/accept_ra'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv6/conf/eth3.80/autoconf'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/dad_transmits'
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 mtu 1500'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3.80'
DEBUG/IFCONFIG returned (out):
[{"ifindex":8,"link":"eth3","ifname":"eth3.80","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"noqueue","operstate": "LOWERLAYERDOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:f f:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":65535,"linkinfo":{"info_kind":"vlan","info_data":{"protocol":"802.1Q","id":80,"flags":["RE ORDER_HDR"]}},"inet6_addr_gen_mode":"none","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'xdp_loader -d eth3.80 -U --auto-mode'
DEBUG/IFCONFIG returned (out):
INFO: xdp_link_detach() no curr XDP prog on ifindex:8
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 up'

Don't think this is related to the i40e driver. I think it's because vyos sees

Auto-negotiation: off

when running ethtool and tries to set it to on because it's not off in the config.

DEBUG/IFCONFIG cmd 'ethtool -s eth3 autoneg on' <--- nic flap occurs here.

If I set speed 10000 and duplex full the error does not occur anymore.

The nic is using DACs so autoneg is not supported so the driver is correct.

ethtool however does siganl "Supports auto-negotiation: Yes" which is wrong. I've seen this with other nics also not using i40e so not sure how to work around it.