Page MenuHomeVyOS Platform

Static routes not installed into kernel nor frr
Confirmed, Requires assessmentPublicBUG

Description

Hi,

i have an issue with static routes, sometimes static routes are not installed into the kernel nor an static route config item in frr:

set protocols static route6 xxx:xxx:b046:1::/64 next-hop xxx:xxx:b004:3::2 
set protocols static route6 xxx:xxx:b046:c000::/56 next-hop xxx:xxx:b004:3::2
run show inter
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
tun0       xxx:xxx:b004:3::1/64            u/u  Cus: xxx [1G]

Routes are missing because they were not configured from vyos into frr:

vtysh -c 'show running-config' | grep 'ipv6 route'
ipv6 route ::/0 xxx:xxx:4b:62::2
ipv6 route ::/0 xxx:xxx:121::254 254
run show ipv6 route static 
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>  ::/0 [1/0] via xxx:xxx:4b:62::2 (recursive), weight 1, 1d12h14m
  *              via fe80::da84:66ff:feea:7015, eth1, weight 1, 1d12h14m
S   ::/0 [254/0] via xxx:xxxx:121::254, dum1, weight 1, 02w4d09h

If i add that routes manually via vtysh it just works fine. So, vyos does not configure frr probably in case of static routes.

Details

Difficulty level
Unknown (require assessment)
Version
1.4-rolling-202106190417
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Event Timeline

@ernstjo Can you share an example of your tunnel interface?
I don't understand yet how to reproduce it.
If you delete routes and add again, do you get the same result?

Try to touch frr debug to collect more information
https://docs.vyos.io/en/latest/debugging.html#frr

It's a basic tunnel setup and also reproducible with routes which do not point to a tunnel interface and happens also with physical interfaces.

Tunnel config

set interfaces tunnel tun0 address '2a0f:5707:b004:3::1/64'
set interfaces tunnel tun0 description 'Cus: xxx [1G]'
set interfaces tunnel tun0 encapsulation 'sit'
set interfaces tunnel tun0 parameters ip ttl '255'
set interfaces tunnel tun0 parameters ipv6 hoplimit '255'
set interfaces tunnel tun0 remote 'xxxx'
set interfaces tunnel tun0 source-address 'xxx'

I can add routes via frr manually and without any issue:

ipv6 route xxx:xxx:b046:c000::/56 xxx:xxx:b004:3::2

So, vyos having issues to add the static route entries into frr and not having anything to do with tunnels or interfaces.

How to get the debug logs? I already enabled debug mode.

Tested configuration:

set interfaces ethernet eth1 address '192.0.2.1/24'
set interfaces ethernet eth1 address 'dead:beef:b004:3::1/64'
set interfaces tunnel tun0 address '2a0f:5707:b004:3::1/64'
set interfaces tunnel tun0 encapsulation 'sit'
set interfaces tunnel tun0 parameters ip ttl '255'
set interfaces tunnel tun0 parameters ipv6 hoplimit '255'
set interfaces tunnel tun0 remote '192.0.2.2'
set interfaces tunnel tun0 source-address '192.0.2.1'
set protocols static route6 cafe:e1f:b046:1::/64 next-hop 2a0f:5707:b004:3::2
set protocols static route6 cafe:e1f:b046:c000::/56 next-hop 2a0f:5707:b004:3::2

Routes and pings:

vyos@r1-roll# run show ipv6 route static | match "tun"
S>* cafe:e1f:b046:1::/64 [1/0] via 2a0f:5707:b004:3::2, tun0, weight 1, 00:01:28
S>* cafe:e1f:b046:c000::/56 [1/0] via 2a0f:5707:b004:3::2, tun0, weight 1, 00:01:28
[edit]
vyos@r1-roll# 

vyos@r1-roll# run ping cafe:e1f:b046:1::1
PING cafe:e1f:b046:1::1(cafe:e1f:b046:1::1) 56 data bytes
64 bytes from cafe:e1f:b046:1::1: icmp_seq=1 ttl=64 time=0.265 ms
64 bytes from cafe:e1f:b046:1::1: icmp_seq=2 ttl=64 time=0.893 ms
^C
--- cafe:e1f:b046:1::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1049ms
rtt min/avg/max/mdev = 0.265/0.579/0.893/0.314 ms
[edit]
vyos@r1-roll#

Version:

vyos@r1-roll# run show ver

Version:          VyOS 1.4-rolling-202107201147
Release Train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Tue 20 Jul 2021 19:10 UTC

How to get the debug logs? I already enabled debug mode.

sudo systemctl stop vyos-configd
sudo touch /tmp/vyos.frr.debug

Just that shitty. When i enable debug mode route install works fine when i disable debugging i won't work.
I have that situation on multiple routers.

run show ipv6 route static 
[edit]
set protocols static route6 xx:xxx:b046::/48 next-hop xxx:xxx:b000::1
[edit]
vyos# commit
[edit]
vyos# run show ipv6 route static 
[edit]

Route not installed..

Enable debug mode:

touch /tmp/vyos.frr.debug
[edit]
# sudo systemctl stop vyos-configd
[edit]
# set protocols static route6 2a0f:5707:b046::/48 next-hop 2a0f:5707:b000::1
[edit]
# commit

Debug Log:

add_before:   add           16 !
add_before:   add           17 ipv6 route xxx:xxx:b040:1::/64 xxx:5707:b004:1::2    
add_before:   add           18 ipv6 route xxx:xxx:b040:2::/64 xxx:5707:b004:2::2    
add_before:   add           19 ipv6 route xxx:xxx:b046::/48 xxx:5707:b000::1    
add_before:   add           20 !
add_before:   add           21 !
add_before:   add           22 
commit_configuration:  Commiting configuration
commit_configuration: new_config   0 !
commit_configuration: new_config   1 frr version 7.5.1-20201222-185-gb3f4ff1d9
commit_configuration: new_config   2 frr defaults traditional
commit_configuration: new_config   3 hostname debian
commit_configuration: new_config   4 log syslog
commit_configuration: new_config   5 log facility local7
commit_configuration: new_config   6 hostname xxxx
commit_configuration: new_config   7 service integrated-vtysh-config
commit_configuration: new_config   8 !
commit_configuration: new_config   9 
commit_configuration: new_config  10 
commit_configuration: new_config  11 !
commit_configuration: new_config  12 vrf mgmt
commit_configuration: new_config  13  ip route 0.0.0.0/0 xxx.xxx.61.254
commit_configuration: new_config  14  exit-vrf
commit_configuration: new_config  15 !
commit_configuration: new_config  16 !
commit_configuration: new_config  17 ipv6 route xxx:xxx:b040:1::/64 xxx:xxx:b004:1::2    
commit_configuration: new_config  18 ipv6 route xxx:xxx:b040:2::/64 xxx:xxx:b004:2::2    
commit_configuration: new_config  19 ipv6 route xxx:xxx:b046::/48 xxx:xxx:b000::1    
commit_configuration: new_config  20 !
commit_configuration: new_config  21 !
commit_configuration: new_config  22 
commit_configuration: new_config  23 line vty
commit_configuration: new_config  24 !
commit_configuration: new_config  25 end
run show ipv6 route static 
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* xxx:xxx:b040:1::/64 [1/0] via xxx:xxx:b004:1::2, tun01523, weight 1, 00:00:51
S>* xxx:xxx:b040:2::/64 [1/0] via xxx:xxx:b004:2::2, tun3001523, weight 1, 00:00:51
S>  xxx:xxx:b046::/48 [1/0] via xxx:xxx:b000::1 (recursive), weight 1, 00:00:06
  *                             via fe80::9459:a8ff:fe1c:c073, l2tpeth1, weight 1, 00:00:06
[edit]

All static routes installed,..but only in debug mode.

Interesting. But I never saw it.
Can you check the latest rolling? I have no other ideas yet.

It happens on different versions and not fixed in the latest version.

I can't reproduce it in any version.
Let's check what is different, and why only in your case do you have a problem, but other users never have this issue.

Do you use standard user (vyos) or not?
Which hypervisor?
How do you deploy the instances?
Do you use radius authentication or something like this?
Anything else?

There must be something in common in the configuration that causes your instances to behave like this.

Do you use standard user (vyos) or not?
yes, standard vyos user

Which hypervisor?
VMware, Proxmox and Baremetal

How do you deploy the instances?
Classical via rolling isos

Do you use radius authentication or something like this?
No

Anything else?
I'm using vrf feature for mgmt access / separation. Mgmt VRF has static default route ipv4/ipv6.

show vrf

VRF name          state     mac address        flags                     interfaces
--------          -----     -----------        -----                     ----------

mgmt              up        66:a8:c2:0f:1d:7c  noarp,master,up,lower_up  eth0
# run show ipv6 route vrf mgmt
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF mgmt:
S>* ::/0 [1/0] via xx:xx:1:30::1, eth0, weight 1, 04w0d01h
K * ::/0 [255/8192] unreachable (ICMP unreachable), 04w0d01h
C>* xx:xx:1:30::/64 is directly connected, eth0, 04w0d01h
C>* fe80::/64 is directly connected, eth0, 04w0d01h
# run show ip route vrf mgmt
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF mgmt:
S>* 0.0.0.0/0 [1/0] via xx.xx.23.1, eth0, weight 1, 04w0d01h
K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04w0d01h
C>* xx.xx.23.0/25 is directly connected, eth0, 04w0d01h

I can reproduce this issue on different routers (Baremetal or VMs). Debug mode on, ipv6 routes install. Debug mode off, static route config is missing in frr, thus not installed into the kernel.

All systems using vrf support to separate out of band management. Systems without vrf enabled it works fine without debug mode on.

Viacheslav changed the task status from Open to Confirmed.EditedFri, Jul 23, 10:29 AM

I can confirm that happens only if you ssh via vrf and set static route6 in default vrf
And it works without debug, if you stop vyos-configd service

sudo systemctl stop vyos-configd