Page MenuHomeVyOS Platform

Builds sometime after 2020-12-17 have broken routing after reboot
Closed, ResolvedPublicBUG

Description

Confirmed on the 2020-12-31 build too.

Take this config:

 route 0.0.0.0/0 {
     next-hop x.x.x.x {
     }
 }

table 146 {
     interface-route 0.0.0.0/0 {
         next-hop-interface tun2 {
         }
     }
 }
 table 200 {
     interface-route 0.0.0.0/0 {
         next-hop-interface wg4 {
         }
     }
 }

Results in

S>* 0.0.0.0/0 [1/0] is directly connected, tun2, weight 1, 00:03:46
  *                 is directly connected, wg4, weight 1, 00:03:46
  *                 via x.x.x.x, eth0.7, weight 1, 00:03:46 (normal default gateway)

That's bad, it should look like:

S>* 0.0.0.0/0 [1/0] via x.x.x.x, eth0.7, 02:35:00

After a reboot, this is what vtysh looks like (missing the table options)

ip route 0.0.0.0/0 x.x.x.x
ip route 0.0.0.0/0 wg4
ip route 0.0.0.0/0 tun2

If I delete the two table X configs and readd them, the appropriate config happens in vtysh

ip route 0.0.0.0/0 tun2 table 146
ip route 0.0.0.0/0 wg4 table 200
ip route 0.0.0.0/0 x.x.x.x

So something happens during a reboot that prevents the table from getting added to FRR, but it works fine if configing from the CLI (until a reboot)

Details

Difficulty level
Unknown (require assessment)
Version
1.3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

kroy renamed this task from Builds sometime after 2020-12-17 have broken routing to Builds sometime after 2020-12-17 have broken routing after reboot.Dec 31 2020, 7:43 PM
kroy created this task.

It feels like a bug which we received from upgrading to FRR 7.5 series.

https://github.com/FRRouting/frr/issues/7806

Running the VyOS code from vyatta-cfg-quagga repo with set -x yields the following commands send to FRR:

vyos@vyos# vtysh -c 'configure terminal' -c 'ip route 0.0.0.0/0 eth1 table 146'
vyos@vyos# vtysh -c 'configure terminal' -c 'ip route 0.0.0.0/0 vtun10 table 200'

vyos@vyos# vtysh -c "show run" | grep 0.0.0.0/0
ip route 0.0.0.0/0 vtun10 table 200
ip route 0.0.0.0/0 eth1 table 200
ip route 0.0.0.0/0 172.18.201.254 10

Running the exact same commands on FRR 7.3 series shows:

vyos@vyos# vtysh -c "show run" | grep 0.0.0.0/0
ip route 0.0.0.0/0 vtun10 table 200
ip route 0.0.0.0/0 eth1 table 146
ip route 0.0.0.0/0 172.18.201.254 10

Just to add to the knowledge surrounding this issue...

The reason it worked when I deleted/readded the commands after boot, because it is order dependent. So really it was fixing itself by accident, and that's not really a temporary bandaid for anyone until the source FRR is fixed.

Reverted to 7.3 let's wait for the fix to pop up in 7.5 FRR branch and we can try again.

Please re-test with latest current rolling release. Reverted back for FRR 7.5 and that bug should have been fixed in there.

Here is some more testing. Currently on FRR 7.5:

vyos@vyos:~$ show version frr
FRRouting 7.5-20210212-00-gbc460d132 (vyos).

VyOS version is:

vyos@vyos:~$ show version

Version:          VyOS 1.4-rolling-202102141111
Release Train:    sagitta

Here's my config:

vyos@vyos:~$ show configuration commands | match static
set protocols static route 10.255.255.255/32 interface eth0
set protocols static table 100 route 10.255.255.255/32 interface eth0
set protocols static table 101 route 10.255.255.255/32 interface eth0
set protocols static table 102 route 10.255.255.255/32 interface eth0
set protocols static table 103 route 10.255.255.255/32 interface eth0
set protocols static table 104 route 10.255.255.255/32 interface eth0

Here's how the routing table looks:

vyos@vyos:~$ show ip route static
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:03

Here's how the other routing tables look:

vyos@vyos:~$ show ip route table 100
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 100:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:30
vyos@vyos:~$ show ip route table 101
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 101:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:31
vyos@vyos:~$ show ip route table 102
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 102:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:32
vyos@vyos:~$ show ip route table 103
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 103:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:33
vyos@vyos:~$ show ip route table 104
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 104:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:34

Reboot and login:

login as: vyos
Pre-authentication banner message from server:
| Welcome to VyOS
End of banner message from server
vyos@10.0.0.80's password:
Linux vyos 5.10.14-amd64-vyos #1 SMP Thu Feb 11 19:53:47 UTC 2021 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Feb 17 01:56:55 2021 from 10.0.0.2


vyos@vyos:~$
vyos@vyos:~$
vyos@vyos:~$
vyos@vyos:~$
vyos@vyos:~$ show run

  Invalid command: show [run]

vyos@vyos:~$ show ip route static
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:00:16

Checking the tables now:

vyos@vyos:~$ show ip route table 100
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 100:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:50
vyos@vyos:~$ show ip route table 101
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 101:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:52
vyos@vyos:~$ show ip route table 102
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 102:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:53
vyos@vyos:~$ show ip route table 103
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 103:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:54
vyos@vyos:~$ show ip route table 104
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 104:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:54

It *seems* to work now with FRR on the tables with static routes:

vyos@vyos:~$ vtysh -c "show run" | grep 10.255.255.255/32
ip route 10.255.255.255/32 eth0 table 104
ip route 10.255.255.255/32 eth0 table 103
ip route 10.255.255.255/32 eth0 table 102
ip route 10.255.255.255/32 eth0 table 101
ip route 10.255.255.255/32 eth0 table 100
ip route 10.255.255.255/32 eth0

Sorry. Yep. Can confirm, problem fixed.

erkin set Issue type to Bug (incorrect behavior).Sun, Aug 29, 11:39 AM