Page MenuHomeVyOS Platform

FRRouting/zebra 7.5.1 does not redistribute routes to other protocols
Confirmed, NormalPublicBUG

Description

Details

In some cases zebra does not redistribute connected routes to other protocols. For example, if there is more than one address added to an interface at the same time.

A configuration example to reproduce the problem:

set high-availability vrrp group vrrp1 interface 'eth0'
set high-availability vrrp group vrrp1 virtual-address 192.168.123.1/24
set high-availability vrrp group vrrp1 virtual-address 192.168.123.2/24
set high-availability vrrp group vrrp1 vrid '100'
set interfaces ethernet eth0 address '192.168.122.5/24'
set protocols bgp 65000 address-family ipv4-unicast redistribute connected

Routing tables will look like this:

[email protected]:~$ show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C>* 192.168.122.0/24 is directly connected, eth0, 00:02:12
C * 192.168.123.0/24 is directly connected, eth0, 00:00:45
C>* 192.168.123.0/24 is directly connected, eth0, 00:00:45

[email protected]:~$ show ip bgp 
BGP table version is 1, local router ID is 192.168.123.2, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.122.0/24 0.0.0.0                  0         32768 ?

Displayed  1 routes and 1 total paths

A faster way to reproduce

To reproduce this it is enough to add to config only:

set protocols bgp 65000 address-family ipv4-unicast redistribute connected

And then try these two commands:

sudo ip a add 192.168.123.1/24 dev eth0 && sudo ip a add 192.168.123.2/24 dev eth0
sudo ip a add 192.168.123.1/24 dev eth0 & sudo ip a add 192.168.123.2/24 dev eth0

The first command will work as expected because addresses are added one by one:

[email protected]:~$ sudo ip a add 192.168.123.1/24 dev eth0 && sudo ip a add 192.168.123.2/24 dev eth0
[email protected]:~$ show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C * 192.168.123.0/24 is directly connected, eth0, 00:00:02
C>* 192.168.123.0/24 is directly connected, eth0, 00:00:02
[email protected]:~$ show ip bgp 
BGP table version is 5, local router ID is 192.168.123.2, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.123.0/24 0.0.0.0                  0         32768 ?

Displayed  1 routes and 1 total paths
[email protected]:~$ sudo ip a flush dev eth0

But the second one:

[email protected]:~$ sudo ip a add 192.168.123.1/24 dev eth0 & sudo ip a add 192.168.123.2/24 dev eth0
[1] 3312
[1]+  Done                    sudo ip a add 192.168.123.1/24 dev eth0
[email protected]:~$ show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C * 192.168.123.0/24 is directly connected, eth0, 00:00:03
C>* 192.168.123.0/24 is directly connected, eth0, 00:00:03
[email protected]:~$ show ip bgp 
No BGP prefixes displayed, 0 exist
[email protected]:~$ sudo ip a flush dev eth0

FRR debug for the normal command is:

Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165771, pid=3362
Oct 07 18:02:50 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x80:
Oct 07 18:02:50 vyos zebra[940]:   IFA_LOCAL     192.168.123.1/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_ADDRESS   192.168.123.1/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:02:50 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.1/24 on eth0(0)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8dfc20 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de610, re 0x560dcb8dd6d0 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 unicast proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8dfd00, re 0x560dcb8dfc20 (connected)
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8de610, re 0x560dcb8dd6d0 (connected)
Oct 07 18:02:50 vyos zebra[940]: netlink_nexthop_msg_encode: ID (19): directly connected, eth0(2) vrf default(0)
Oct 07 18:02:50 vyos zebra[940]: netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=19
Oct 07 18:02:50 vyos zebra[940]: nl_batch_send: netlink-dp (NS 0), batch size=40, msg cnt=1
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): no fib nhg
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): rib nhg matched, changed 'false'
Oct 07 18:02:50 vyos zebra[940]: (0:254):192.168.123.0/24: Redist update re 0x560dcb8dfc20 (connected), old 0x0 (None)
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client bgp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client bgp: type connected, vrf_id 0, p 192.168.123.0/24
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client ldp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client ldp: type connected, vrf_id 0, p 192.168.123.0/24
Oct 07 18:02:50 vyos zebra[940]: Not Notifying Owner: connected about prefix 192.168.123.0/24(254) 2 vrf: 0
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165771, pid=3364
Oct 07 18:02:50 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x81:
Oct 07 18:02:50 vyos zebra[940]:   IFA_LOCAL     192.168.123.2/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_ADDRESS   192.168.123.2/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:02:50 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.2/24 on eth0(0)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8de6f0 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de610, re 0x560dcb8de100 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8de610, re 0x560dcb8dd6d0 (connected)
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): no fib nhg
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): rib nhg matched, changed 'false'
Oct 07 18:02:50 vyos zebra[940]: Not Notifying Owner: connected about prefix 192.168.123.0/24(254) 2 vrf: 0

And in case of problem:

Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165832, pid=3404
Oct 07 18:03:51 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x80:
Oct 07 18:03:51 vyos zebra[940]:   IFA_LOCAL     192.168.123.2/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_ADDRESS   192.168.123.2/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:03:51 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.2/24 on eth0(0)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8de180 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de820, re 0x560dcb8de700 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 unicast proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165832, pid=3405
Oct 07 18:03:51 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x81:
Oct 07 18:03:51 vyos zebra[940]:   IFA_LOCAL     192.168.123.1/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_ADDRESS   192.168.123.1/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:03:51 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.1/24 on eth0(0)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8dda60 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de820, re 0x560dcb8d8140 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8dfd00, re 0x560dcb8de180 (connected)
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8de820, re 0x560dcb8de700 (connected)
Oct 07 18:03:51 vyos zebra[940]: netlink_nexthop_msg_encode: ID (22): directly connected, eth0(2) vrf default(0)
Oct 07 18:03:51 vyos zebra[940]: netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=22
Oct 07 18:03:51 vyos zebra[940]: nl_batch_send: netlink-dp (NS 0), batch size=40, msg cnt=1
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): no fib nhg
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): rib nhg matched, changed 'false'
Oct 07 18:03:51 vyos zebra[940]: Not Notifying Owner: connected about prefix 192.168.123.0/24(254) 2 vrf: 0

As we can see, the redistribution event is missed in the second case:

Oct 07 18:02:50 vyos zebra[940]: (0:254):192.168.123.0/24: Redist update re 0x560dcb8dfc20 (connected), old 0x0 (None)
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client bgp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client bgp: type connected, vrf_id 0, p 192.168.123.0/24
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client ldp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client ldp: type connected, vrf_id 0, p 192.168.123.0/24

Additional info

The bug cannot be reproduced in FRRouting 8.0+. We should consider FRR update or find and backport the change that fixes this to FRR 7.5.1, if this is possible.

Details

Difficulty level
Hard (possibly days)
Version
1.3.2
Why the issue appeared?
Issues in third-party code
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

zsdc changed the task status from Open to Confirmed.Oct 7 2022, 6:39 PM
zsdc triaged this task as Normal priority.
zsdc changed Version from 1.3.1-S2 to 1.3.2.
zsdc changed Why the issue appeared? from Will be filled on close to Issues in third-party code.

Hi @zsdc! This seems to be related to T4028. The relevant commits are:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
commit 92980561382fc04380414a6e2f6ca6746c2fe5e9 ┃
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━
Author: Donald Sharp <[email protected]>
Date:   Mon Apr 19 19:23:45 2021 -0400

    zebra: Allow one connected route per network mask on a interface
    
    Currently FRR reads the kernel for interface state and FRR
    creates a connected route per address on an interface.  If
    you are in a situation where you have multiple addresses
    on an interface just create 1 connected route for them:
    ........
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
commit 7fb9825cf7e762add68f5108df4eddda1247f198 ┃
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━
Author: Volodymyr Huti <[email protected]>
Date:   Mon Apr 11 06:58:14 2022 +0300

    zebra: set ZEBRA_IFC_DOWN on connected routes for inactive interfaces
    
    If you are in a situation where you have multiple addresses on an
    interface, zebra creates one connected route for them.
    The issue is that the rib entry is not created if addresses were
    added before the interface was running.
    .....

From local testing, manually applying Donald`s patch on top of the base_7.6 tag seems to fix the issue -

[R1]
sudo ip l set dev ens6 up
sudo ip a flush dev ens6
sleep 3
sudo ip a add 192.168.122.3/24 dev ens6 & sudo ip a add 192.168.122.2/24 dev ens6

I can see the prefix being redistributed to another device.

[R2][email protected]:~$ vtr
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure
S   0.0.0.0/0 [1/0] is directly connected, ens5 inactive, weight 1, 00:01:06
...
B>  192.168.122.0/24 [200/0] via 10.0.0.1 (recursive), weight 1, 00:00:05
  *                            via 10.0.0.1, ens4 onlink, weight 1, 00:00:05

@v.huti Thanks for investigating and testing! How about you create a PR with the patch against 1.3 / equuleus ?