Smoketests fail for vyos:current (test_netns.py)
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	unity
	Sep 8 2023, 8:31 PM

Description

VyOS rolling ISO buildings fails during smoketests.

10_Run smoketests.txt.zip204 KBDownload

Details

Difficulty level: Unknown (require assessment)
Version: 1.5
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Perfectly compatible
Issue type: Bug (incorrect behavior)

Related Objects

Mentioned Here: T5518: Add MLD protocol support

Event Timeline

unity created this task.Sep 8 2023, 8:31 PM

unity attached a referenced file: F3848966: 10_Run smoketests.txt.zip. (Show Details)

unity renamed this task from Smoketests fail for vyos:current to Smoketests fail for vyos:current (test_netns.py).Sep 8 2023, 8:39 PM

Some observations:

2023-09-08T18:08:40.9131605Z DEBUG - Executing command: qemu-system-x86_64         -name "TESTVM-UEFI"         -smp sockets=1,cpus=3,cores=1         -cpu host         -bios /usr/share/OVMF/OVMF_CODE.fd         -m 3G         -vga none         -nographic         -machine accel=kvm         -uuid d27cf29e-4419-4407-8f82-dc73d1acd184         -cpu host          -boot d -drive file=build/live-image-amd64.hybrid.iso,format=raw,if=none,media=cdrom,id=drive-cd1,readonly=on -device ahci,id=achi0 -device ide-cd,bus=achi0.0,drive=drive-cd1,id=cd1,bootindex=1         -enable-kvm         -netdev user,id=n0,net=192.0.2.0/24,dhcpstart=192.0.2.101,dns=192.0.2.10 -device virtio-net-pci,netdev=n0,mac=52:54:00:00:00:00,romfile=""         -netdev user,id=n1 -device virtio-net-pci,netdev=n1,mac=52:54:00:00:00:01,romfile=""         -netdev user,id=n2 -device virtio-net-pci,netdev=n2,mac=52:54:00:00:00:02,romfile=""         -netdev user,id=n3 -device virtio-net-pci,netdev=n3,mac=52:54:00:00:00:03,romfile=""         -netdev user,id=n4 -device virtio-net-pci,netdev=n4,mac=52:54:00:00:00:04,romfile=""         -netdev user,id=n5 -device virtio-net-pci,netdev=n5,mac=52:54:00:00:00:05,romfile=""         -netdev user,id=n6 -device virtio-net-pci,netdev=n6,mac=52:54:00:00:00:06,romfile=""         -netdev user,id=n7 -device virtio-net-pci,netdev=n7,mac=52:54:00:00:00:07,romfile=""         -device virtio-scsi-pci,id=scsi0         -drive format=raw,file=testinstall-20230908-171217-cbb7.img,if=none,media=disk,id=drive-hd1,readonly=off         -device scsi-hd,bus=scsi0.0,drive=drive-hd1,id=hd1,bootindex=2
2023-09-08T18:08:40.9152507Z DEBUG - qemu-system-x86_64: warning: This family of AMD CPU doesn't support hyperthreading(3)
2023-09-08T18:08:40.9153908Z DEBUG - Please configure -smp options properly or try enabling topoext feature.

I think the above:

-smp sockets=1,cpus=3,cores=1

is supposed to be written as:

-smp 3,sockets=1,cores=3,threads=1

And regarding fails/errors:

1.
2023-09-08T18:08:41.0761215Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_load_balancing_wan.py
2023-09-08T18:08:41.0761763Z DEBUG - test_check_chains (__main__.TestLoadBalancingWan.test_check_chains) ... Cannot find device "ceth0"
2023-09-08T18:08:41.0762059Z DEBUG - Cannot find device "ceth1"
2023-09-08T18:08:41.0762350Z DEBUG - Cannot find device "ceth2"
2023-09-08T18:08:41.0762579Z DEBUG - ok
2023-09-08T18:08:41.0763118Z DEBUG - test_table_routes (__main__.TestLoadBalancingWan.test_table_routes) ... Cannot find device "ceth0"
2023-09-08T18:08:41.0763410Z DEBUG - Cannot find device "ceth1"
2023-09-08T18:08:41.0763697Z DEBUG - Cannot find device "ceth2"
2023-09-08T18:08:41.0763912Z DEBUG - ok

2.
2023-09-08T18:08:41.0777997Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_netns.py
2023-09-08T18:08:41.0778408Z DEBUG - test_netns_create (__main__.NetNSTest.test_netns_create) ... FAIL
2023-09-08T18:08:41.0778972Z DEBUG - test_netns_interface (__main__.NetNSTest.test_netns_interface) ... FAIL
2023-09-08T18:08:41.0779177Z DEBUG - 
2023-09-08T18:08:41.0779541Z DEBUG - ======================================================================
2023-09-08T18:08:41.0779943Z DEBUG - FAIL: test_netns_create (__main__.NetNSTest.test_netns_create)
2023-09-08T18:08:41.0780324Z DEBUG - ----------------------------------------------------------------------
2023-09-08T18:08:41.0780634Z DEBUG - Traceback (most recent call last):
2023-09-08T18:08:41.0781083Z DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_netns.py", line 40, in tearDown
2023-09-08T18:08:41.0781379Z DEBUG -     self.assertFalse(tmp)
2023-09-08T18:08:41.0781745Z DEBUG - AssertionError: 'ns203 (id: 3)\nnsC (id: 2)' is not false
2023-09-08T18:08:41.0781962Z DEBUG - 
2023-09-08T18:08:41.0782331Z DEBUG - ======================================================================
2023-09-08T18:08:41.0782752Z DEBUG - FAIL: test_netns_interface (__main__.NetNSTest.test_netns_interface)
2023-09-08T18:08:41.0783132Z DEBUG - ----------------------------------------------------------------------
2023-09-08T18:08:41.0783520Z DEBUG - Traceback (most recent call last):
2023-09-08T18:08:41.0783967Z DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_netns.py", line 40, in tearDown
2023-09-08T18:08:41.0784258Z DEBUG -     self.assertFalse(tmp)
2023-09-08T18:08:41.0784635Z DEBUG - AssertionError: 'ns203 (id: 3)\nnsC (id: 2)' is not false

3.
2023-09-08T18:08:41.0909390Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_service_https.py
2023-09-08T18:08:41.0909809Z DEBUG - test_api_auth (__main__.TestHTTPSService.test_api_auth) ... ERROR
2023-09-08T18:08:41.0910243Z DEBUG - test_certificate (__main__.TestHTTPSService.test_certificate) ... ok
2023-09-08T18:08:41.0910637Z DEBUG - test_default (__main__.TestHTTPSService.test_default) ... ok
2023-09-08T18:08:41.0911067Z DEBUG - test_server_block (__main__.TestHTTPSService.test_server_block) ... ok
2023-09-08T18:08:41.0911265Z DEBUG - 
2023-09-08T18:08:41.0911630Z DEBUG - ======================================================================
2023-09-08T18:08:41.0912034Z DEBUG - ERROR: test_api_auth (__main__.TestHTTPSService.test_api_auth)
2023-09-08T18:08:41.0912464Z DEBUG - ----------------------------------------------------------------------
2023-09-08T18:08:41.0912786Z DEBUG - Traceback (most recent call last):
2023-09-08T18:08:41.0913287Z DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 114, in wrapped
2023-09-08T18:08:41.0913581Z DEBUG -     return f(*args, **kwargs)
2023-09-08T18:08:41.0913842Z DEBUG -            ^^^^^^^^^^^^^^^^^^
2023-09-08T18:08:41.0914361Z DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_service_https.py", line 164, in test_api_auth
2023-09-08T18:08:41.0914736Z DEBUG -     success = r.json()['data']['SystemStatus']['success']
2023-09-08T18:08:41.0915005Z DEBUG -               ~~~~~~~~^^^^^^^^
2023-09-08T18:08:41.0915266Z DEBUG - KeyError: 'data'

4.
2023-09-08T18:08:41.0972158Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_system_conntrack.py
2023-09-08T18:08:41.0972647Z DEBUG - test_conntrack_hash_size (__main__.TestSystemConntrack.test_conntrack_hash_size) ... ok
2023-09-08T18:08:41.0973143Z DEBUG - test_conntrack_ignore (__main__.TestSystemConntrack.test_conntrack_ignore) ... ERROR
2023-09-08T18:08:41.0973686Z DEBUG - test_conntrack_module_enable (__main__.TestSystemConntrack.test_conntrack_module_enable) ... ok
2023-09-08T18:08:41.0974172Z DEBUG - test_conntrack_options (__main__.TestSystemConntrack.test_conntrack_options) ... ok
2023-09-08T18:08:41.0974374Z DEBUG - 
2023-09-08T18:08:41.0974735Z DEBUG - ======================================================================
2023-09-08T18:08:41.0975208Z DEBUG - ERROR: test_conntrack_ignore (__main__.TestSystemConntrack.test_conntrack_ignore)
2023-09-08T18:08:41.0975576Z DEBUG - ----------------------------------------------------------------------
2023-09-08T18:08:41.0975890Z DEBUG - Traceback (most recent call last):
2023-09-08T18:08:41.0976444Z DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_system_conntrack.py", line 274, in test_conntrack_ignore
2023-09-08T18:08:41.0976713Z DEBUG -     self.cli_commit()
2023-09-08T18:08:41.0977317Z DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 76, in cli_commit
2023-09-08T18:08:41.0977615Z DEBUG -     self._session.commit()
2023-09-08T18:08:41.0978089Z DEBUG -   File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 183, in commit
2023-09-08T18:08:41.0978395Z DEBUG -     out = self.__run_command([COMMIT])
2023-09-08T18:08:41.0978692Z DEBUG -           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-08T18:08:41.0979204Z DEBUG -   File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 139, in __run_command
2023-09-08T18:08:41.0979528Z DEBUG -     raise ConfigSessionError(output)
2023-09-08T18:08:41.0979930Z DEBUG - vyos.configsession.ConfigSessionError: [ firewall ]
2023-09-08T18:08:41.0980324Z DEBUG - ConfigError('Failed to apply configuration: /run/nftables-
2023-09-08T18:08:41.0980759Z DEBUG - ct.conf:15:38-53: Error: No\nsuch file or directory; did you mean set
2023-09-08T18:08:41.0981296Z DEBUG - ‘A_conntracktest’ in table ip\n‘vyos_filter’?        ip saddr  192.0.2.1
2023-09-08T18:08:41.0981683Z DEBUG - ip daddr  @A_conntracktest\ncounter notrack comment
2023-09-08T18:08:41.0982122Z DEBUG - "ignore-2"\n^^^^^^^^^^^^^^^^ /run/nftables-ct.conf:33:38-55: Error: No
2023-09-08T18:08:41.0982652Z DEBUG - such file or\ndirectory; did you mean set ‘A6_conntracktest6’ in table
2023-09-08T18:08:41.0983137Z DEBUG - ip6\n‘vyos_filter’?        ip6 saddr  fe80::1 ip6 daddr
2023-09-08T18:08:41.0983472Z DEBUG - @A6_conntracktest6\ncounter notrack comment
2023-09-08T18:08:41.0983762Z DEBUG - "ignore-12"\n^^^^^^^^^^^^^^^^^^')
2023-09-08T18:08:41.0983966Z DEBUG - 
2023-09-08T18:08:41.0984220Z DEBUG - [[firewall]] failed
2023-09-08T18:08:41.0984486Z DEBUG - [ system conntrack ]
2023-09-08T18:08:41.0984927Z DEBUG - Failed to apply configuration: /run/nftables-ct.conf:15:38-53: Error: No
2023-09-08T18:08:41.0985373Z DEBUG - such file or directory; did you mean set ‘A_conntracktest’ in table ip
2023-09-08T18:08:41.0985793Z DEBUG - ‘vyos_filter’?        ip saddr  192.0.2.1 ip daddr  @A_conntracktest
2023-09-08T18:08:41.0986111Z DEBUG - counter notrack comment "ignore-2"
2023-09-08T18:08:41.0986587Z DEBUG - ^^^^^^^^^^^^^^^^ /run/nftables-ct.conf:33:38-55: Error: No such file or
2023-09-08T18:08:41.0986999Z DEBUG - directory; did you mean set ‘A6_conntracktest6’ in table ip6
2023-09-08T18:08:41.0987457Z DEBUG - ‘vyos_filter’?        ip6 saddr  fe80::1 ip6 daddr  @A6_conntracktest6
2023-09-08T18:08:41.0987769Z DEBUG - counter notrack comment "ignore-12"
2023-09-08T18:08:41.0988018Z DEBUG - ^^^^^^^^^^^^^^^^^^
2023-09-08T18:08:41.0988222Z DEBUG - 
2023-09-08T18:08:41.0988493Z DEBUG - [[system conntrack]] failed
2023-09-08T18:08:41.0988744Z DEBUG - Commit failed

5.
2023-09-08T18:08:41.1035900Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_vrf.py
2023-09-08T18:08:41.1036351Z DEBUG - test_vrf_assign_interface (__main__.VRFTest.test_vrf_assign_interface) ... ok
2023-09-08T18:08:41.1036720Z DEBUG - test_vrf_bind_all (__main__.VRFTest.test_vrf_bind_all) ... ok
2023-09-08T18:08:41.1037167Z DEBUG - test_vrf_disable_forwarding (__main__.VRFTest.test_vrf_disable_forwarding) ... ok
2023-09-08T18:08:41.1037770Z DEBUG - test_vrf_ip_ipv6_protocol_non_existing_route_map (__main__.VRFTest.test_vrf_ip_ipv6_protocol_non_existing_route_map) ... ok
2023-09-08T18:08:41.1038309Z DEBUG - test_vrf_ip_protocol_route_map (__main__.VRFTest.test_vrf_ip_protocol_route_map) ... ok
2023-09-08T18:08:41.1038805Z DEBUG - test_vrf_ipv6_protocol_route_map (__main__.VRFTest.test_vrf_ipv6_protocol_route_map) ... ok
2023-09-08T18:08:41.1039300Z DEBUG - test_vrf_link_local_ip_addresses (__main__.VRFTest.test_vrf_link_local_ip_addresses) ... ok
2023-09-08T18:08:41.1039723Z DEBUG - test_vrf_loopbacks_ips (__main__.VRFTest.test_vrf_loopbacks_ips) ... ok
2023-09-08T18:08:41.1040144Z DEBUG - test_vrf_static_route (__main__.VRFTest.test_vrf_static_route) ... FAIL
2023-09-08T18:08:41.1040341Z DEBUG - 
2023-09-08T18:08:41.1040734Z DEBUG - ======================================================================
2023-09-08T18:08:41.1041153Z DEBUG - FAIL: test_vrf_static_route (__main__.VRFTest.test_vrf_static_route)
2023-09-08T18:08:41.1041529Z DEBUG - ----------------------------------------------------------------------
2023-09-08T18:08:41.1041841Z DEBUG - Traceback (most recent call last):
2023-09-08T18:08:41.1042345Z DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_vrf.py", line 230, in test_vrf_static_route
2023-09-08T18:08:41.1042748Z DEBUG -     self.assertIn(f' ip route {prefix} {next_hop}', frrconfig)
2023-09-08T18:08:41.1043277Z DEBUG - AssertionError: ' ip route 10.0.100.0/24 192.0.100.1' not found in 'vrf red\n vni 100\nexit-vrf\n!'

For the tests above, which configs are actually being used?

For example case 5 above (test_vrf_static_route) I would expect it to fail because 192.0.100.1 is not a valid next-hop according to:

https://github.com/vyos/vyos-1x/blob/current/smoketest/config-tests/basic-vyos

https://github.com/vyos/vyos-1x/blob/current/smoketest/configs/basic-vyos

https://github.com/vyos/vyos-1x/blob/current/smoketest/configs/vrf-basic

Also regarding case 5 I dont think "vni {table}" is actually being set by vyos-configd.

This is for example how a static route within vrf looks like in VyOS 1.5-rolling-202309080021:

set vrf name INTERNET protocols static route 0.0.0.0/0 next-hop 192.168.0.1 distance '1'
set vrf name INTERNET table '101'

and here is the output of /run/frr/config/frr.conf:

vrf INTERNET
 ip route 0.0.0.0/0 192.168.0.1
exit-vrf

PR https://github.com/vyos/vyos-1x/pull/2230

PR created (fixes https://github.com/vyos/vyos-build/commit/a863fe0): https://github.com/vyos/vyos-build/pull/395

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6131805039/job/16642103169

Still errors in:

27875 DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_interfaces_ethernet.py

28673 DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_protocols_pim6.py

28995 DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_system_conntrack.py

pasik added a subscriber: pasik.Sep 9 2023, 5:19 PM

Regarding the failing smoketest test_system_conntrack.py (test_conntrack_ignore):

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6133954453/job/16646294279#step:10:28999

https://github.com/vyos/vyos-1x/blob/current/smoketest/scripts/cli/test_system_conntrack.py#L276

I think its a combo of how verify_nftables is being called and how the arrays nftables_search and nftables6_search are constructed.

Proposed fixes:

Change how verify_nftables is being called (this way the extracted ruleset wont have packet and bytes counters in the output):

self.verify_nftables(nftables_search, 'raw', False, '-s')
self.verify_nftables(nftables6_search, 'ip6 raw', False, '-s')

Adjust arrays nftables_search and nftables6_search to match output of nft which verify_nftables is using:

nftables_search = [
    ['ip saddr 192.0.2.1 ip daddr 192.0.2.2 tcp dport 22 counter notrack'],
    ['ip saddr 192.0.2.1 ip daddr @A_conntracktest counter notrack']
]

nftables6_search = [
    ['ip6 saddr fe80::1 ip6 daddr fe80::2 tcp dport 22 counter notrack'],
    ['ip6 saddr fe80::1 ip6 daddr @A6_conntracktest6 counter notrack'],
    ['ip6 saddr fe80::1 ip6 daddr != fe80::3 counter notrack']
]

Unless the comments must be included in the verify aswell?

Because then it should be:

nftables_search = [
    ['ip saddr 192.0.2.1 ip daddr 192.0.2.2 tcp dport 22 counter notrack comment "ignore-1"'],
    ['ip saddr 192.0.2.1 ip daddr @A_conntracktest counter notrack comment "ignore-2"']
]

nftables6_search = [
    ['ip6 saddr fe80::1 ip6 daddr fe80::2 tcp dport 22 counter notrack comment "ignore-11"'],
    ['ip6 saddr fe80::1 ip6 daddr @A6_conntracktest6 counter notrack comment "ignore-12"'],
    ['ip6 saddr fe80::1 ip6 daddr != fe80::3 counter notrack comment "ignore-13"']
]

The failed smoketest test_protocols_pim6.py seems to have been taken care of by:

https://vyos.dev/T5518

https://github.com/vyos/vyos-1x/pull/2233

The failed smoketest test_interfaces_ethernet.py can be seen at:

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6133954453/job/16646294279#step:10:27879

Fix by @sever regarding those failing conntrack smoketest: https://github.com/vyos/vyos-1x/pull/2234

Latest run https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943

Fixed:

1: test_protocols_pim6.py
https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943#step:10:28678

2: test_system_conntrack.py
https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943#step:10:28974

Still fails:

3: test_interfaces_ethernet.py
https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943#step:10:27880

Looks like leftover from dhcp-tests (all the fails within this testsuite is regarding the leftovers aka tearDown fails):

DEBUG - ======================================================================
DEBUG - FAIL: test_dhcp_vrf (__main__.EthernetInterfaceTest.test_dhcp_vrf)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_interfaces_ethernet.py", line 145, in tearDown
DEBUG -     self.assertNotIn(AF_INET, ifaddresses(intf))
DEBUG - AssertionError: 2 unexpectedly found in {17: [{'addr': '52:54:00:00:00:00', 'broadcast': 'ff:ff:ff:ff:ff:ff'}], 2: [{'addr': '192.0.2.101', 'netmask': '255.255.255.0', 'broadcast': '192.0.2.255'}], 10: [{'addr': 'fe80::5054:ff:fe00:0%eth0', 'netmask': 'ffff:ffff:ffff:ffff::/64'}]}

Builds passing: https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6142937552

Smoketests fail for vyos:current (test_netns.py)Closed, ResolvedPublicBUGActions

Description

Details

Related Objects

Event Timeline

Smoketests fail for vyos:current (test_netns.py)
Closed, ResolvedPublicBUG
Actions