Page MenuHomeVyOS Platform

FRR: Committing large number of peers in configuration results in temporarily incomplete config resulting in route leaks
Closed, DuplicatePublicBUG

Description

Upon committing a large batch of BGP peers into the configuration (~40), it appears that FRR configures the peers, but doesn't apply the complete configuration for each peer in order, thus resulting in filters not being correctly applied and too many routes being sent, upsetting peers!

The problem resolves itself eventually once the config is completely applied, but not before causing problems upstream.

This smells a bit like the old issues at startup where FRR starts without complete config/filters in place.

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.2.4
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Event Timeline

bbabich created this task.Jan 5 2020, 5:22 PM
syncer changed the task status from Open to Needs testing.Jan 6 2020, 12:25 AM
syncer assigned this task to Viacheslav.
syncer triaged this task as Normal priority.
syncer edited projects, added VyOS 1.3 Equuleus; removed VyOS 1.2 Crux.
pasik added a subscriber: pasik.Jan 6 2020, 12:26 PM

Hi @bbabich
How we can reproduce this bug?
I tested with 55 bgp-sessions. Each with its own unique filter. All filters applied as needed.
Without filters I announced 111 routes.
With filters per session I export 1 route to each peer.

Commit for 55 peers with uniq filters ~ 23 sec.
The first changes for peers begin at 12 seconds.

I generate the following configuration.

#!/usr/bin/env bash

for I in `seq 0 54`
do
  echo "set interface ethernet eth0 address 10.0.$I.4/29"
  echo "set protocols bgp 65002 neighbor 10.0.$I.1 remote-as '65001'"
  echo "set interfaces dummy dum1 address 100.0.$I.1/24"

  echo "set policy prefix-list NETOUT-$I rule 10 action 'permit'
set policy prefix-list NETOUT-$I rule 10 prefix '100.0.$I.0/24'
set policy route-map NETOUT-$I rule 10 action 'permit'
set policy route-map NETOUT-$I rule 10 match ip address prefix-list 'NETOUT-$I'"

  echo "set protocols bgp 65002 neighbor 10.0.$I.1 address-family ipv4-unicast route-map export NETOUT-$I"

done

echo "set protocols bgp 65002 address-family ipv4-unicast redistribute connected"

Before filters

$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 192.168.122.216, local AS number 65001 vrf-id 0
BGP table version 2234
RIB entries 221, using 40 KiB of memory
Peers 55, using 1124 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
10.0.0.4        4      65002     117     169        0    0    0 00:35:47          111
10.0.1.4        4      65002     117     168        0    0    0 00:35:47          111
10.0.2.4        4      65002     117     158        0    0    0 00:35:46          111
10.0.3.4        4      65002     117     147        0    0    0 00:35:44          111
10.0.4.4        4      65002     117     136        0    0    0 00:35:43          111
10.0.5.4        4      65002     117     126        0    0    0 00:35:42          111
10.0.6.4        4      65002     116     118        0    0    0 00:35:41          111
10.0.7.4        4      65002     116     117        0    0    0 00:35:41          111
10.0.8.4        4      65002     116     116        0    0    0 00:35:41          111
10.0.9.4        4      65002     116     115        0    0    0 00:35:41          111
10.0.10.4       4      65002     117     167        0    0    0 00:35:47          111
10.0.11.4       4      65002     117     166        0    0    0 00:35:47          111
10.0.12.4       4      65002     117     166        0    0    0 00:35:47          111
10.0.13.4       4      65002     117     165        0    0    0 00:35:46          111
10.0.14.4       4      65002     117     164        0    0    0 00:35:46          111
10.0.15.4       4      65002     117     163        0    0    0 00:35:46          111
10.0.16.4       4      65002     117     162        0    0    0 00:35:46          111
10.0.17.4       4      65002     117     161        0    0    0 00:35:46          111
10.0.18.4       4      65002     117     160        0    0    0 00:35:46          111
10.0.19.4       4      65002     117     159        0    0    0 00:35:46          111
10.0.20.4       4      65002     117     157        0    0    0 00:35:45          111
10.0.21.4       4      65002     117     156        0    0    0 00:35:45          111
10.0.22.4       4      65002     117     155        0    0    0 00:35:45          111
10.0.23.4       4      65002     117     154        0    0    0 00:35:45          111
10.0.24.4       4      65002     117     153        0    0    0 00:35:45          111
10.0.25.4       4      65002     117     152        0    0    0 00:35:45          111
10.0.26.4       4      65002     117     151        0    0    0 00:35:45          111
10.0.27.4       4      65002     117     150        0    0    0 00:35:45          111
10.0.28.4       4      65002     117     149        0    0    0 00:35:45          111
10.0.29.4       4      65002     117     148        0    0    0 00:35:44          111
10.0.30.4       4      65002     117     146        0    0    0 00:35:44          111
10.0.31.4       4      65002     117     145        0    0    0 00:35:44          111
10.0.32.4       4      65002     117     144        0    0    0 00:35:44          111
10.0.33.4       4      65002     117     143        0    0    0 00:35:44          111
10.0.34.4       4      65002     117     142        0    0    0 00:35:44          111
10.0.35.4       4      65002     117     141        0    0    0 00:35:44          111
10.0.36.4       4      65002     117     140        0    0    0 00:35:44          111
10.0.37.4       4      65002     117     139        0    0    0 00:35:43          111
10.0.38.4       4      65002     117     138        0    0    0 00:35:43          111
10.0.39.4       4      65002     117     137        0    0    0 00:35:43          111
10.0.40.4       4      65002     117     135        0    0    0 00:35:43          111
10.0.41.4       4      65002     117     133        0    0    0 00:35:43          111
10.0.42.4       4      65002     117     133        0    0    0 00:35:43          111
10.0.43.4       4      65002     117     132        0    0    0 00:35:43          111
10.0.44.4       4      65002     117     132        0    0    0 00:35:42          111
10.0.45.4       4      65002     117     131        0    0    0 00:35:42          111
10.0.46.4       4      65002     117     130        0    0    0 00:35:42          111
10.0.47.4       4      65002     117     129        0    0    0 00:35:42          111
10.0.48.4       4      65002     117     128        0    0    0 00:35:42          111
10.0.49.4       4      65002     117     127        0    0    0 00:35:42          111
10.0.50.4       4      65002     117     125        0    0    0 00:35:42          111
10.0.51.4       4      65002     117     124        0    0    0 00:35:42          111
10.0.52.4       4      65002     116     122        0    0    0 00:35:41          111
10.0.53.4       4      65002     116     120        0    0    0 00:35:41          111
10.0.54.4       4      65002     116     119        0    0    0 00:35:41          111

Total number of neighbors 55

After commit

$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 192.168.122.216, local AS number 65001 vrf-id 0
BGP table version 2401
RIB entries 109, using 20 KiB of memory
Peers 55, using 1124 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
10.0.0.4        4      65002     123     176        0    0    0 00:39:49            1
10.0.1.4        4      65002     123     175        0    0    0 00:39:49            1
10.0.2.4        4      65002     123     165        0    0    0 00:39:48            1
10.0.3.4        4      65002     123     154        0    0    0 00:39:46            1
10.0.4.4        4      65002     123     143        0    0    0 00:39:45            1
10.0.5.4        4      65002     123     133        0    0    0 00:39:44            1
10.0.6.4        4      65002     122     125        0    0    0 00:39:43            1
10.0.7.4        4      65002     122     124        0    0    0 00:39:43            1
10.0.8.4        4      65002     122     123        0    0    0 00:39:43            1
10.0.9.4        4      65002     122     122        0    0    0 00:39:43            1
10.0.10.4       4      65002     123     174        0    0    0 00:39:49            1
10.0.11.4       4      65002     123     173        0    0    0 00:39:49            1
10.0.12.4       4      65002     123     173        0    0    0 00:39:49            1
10.0.13.4       4      65002     123     172        0    0    0 00:39:48            1
10.0.14.4       4      65002     123     171        0    0    0 00:39:48            1
10.0.15.4       4      65002     123     170        0    0    0 00:39:48            1
10.0.16.4       4      65002     123     169        0    0    0 00:39:48            1
10.0.17.4       4      65002     123     168        0    0    0 00:39:48            1
10.0.18.4       4      65002     123     167        0    0    0 00:39:48            1
10.0.19.4       4      65002     123     166        0    0    0 00:39:48            1
10.0.20.4       4      65002     123     164        0    0    0 00:39:47            1
10.0.21.4       4      65002     123     163        0    0    0 00:39:47            1
10.0.22.4       4      65002     123     162        0    0    0 00:39:47            1
10.0.23.4       4      65002     123     161        0    0    0 00:39:47            1
10.0.24.4       4      65002     123     160        0    0    0 00:39:47            1
10.0.25.4       4      65002     123     159        0    0    0 00:39:47            1
10.0.26.4       4      65002     123     158        0    0    0 00:39:47            1
10.0.27.4       4      65002     123     157        0    0    0 00:39:47            1
10.0.28.4       4      65002     123     156        0    0    0 00:39:47            1
10.0.29.4       4      65002     123     155        0    0    0 00:39:46            1
10.0.30.4       4      65002     123     153        0    0    0 00:39:46            1
10.0.31.4       4      65002     123     152        0    0    0 00:39:46            1
10.0.32.4       4      65002     123     151        0    0    0 00:39:46            1
10.0.33.4       4      65002     123     150        0    0    0 00:39:46            1
10.0.34.4       4      65002     123     149        0    0    0 00:39:46            1
10.0.35.4       4      65002     123     148        0    0    0 00:39:46            1
10.0.36.4       4      65002     123     147        0    0    0 00:39:46            1
10.0.37.4       4      65002     123     146        0    0    0 00:39:45            1
10.0.38.4       4      65002     123     145        0    0    0 00:39:45            1
10.0.39.4       4      65002     123     144        0    0    0 00:39:45            1
10.0.40.4       4      65002     123     142        0    0    0 00:39:45            1
10.0.41.4       4      65002     123     140        0    0    0 00:39:45            1
10.0.42.4       4      65002     123     140        0    0    0 00:39:45            1
10.0.43.4       4      65002     123     139        0    0    0 00:39:45            1
10.0.44.4       4      65002     123     139        0    0    0 00:39:44            1
10.0.45.4       4      65002     123     138        0    0    0 00:39:44            1
10.0.46.4       4      65002     123     137        0    0    0 00:39:44            1
10.0.47.4       4      65002     123     136        0    0    0 00:39:44            1
10.0.48.4       4      65002     123     135        0    0    0 00:39:44            1
10.0.49.4       4      65002     123     134        0    0    0 00:39:44            1
10.0.50.4       4      65002     123     132        0    0    0 00:39:44            1
10.0.51.4       4      65002     123     131        0    0    0 00:39:44            1
10.0.52.4       4      65002     122     129        0    0    0 00:39:43            1
10.0.53.4       4      65002     122     127        0    0    0 00:39:43            1
10.0.54.4       4      65002     122     126        0    0    0 00:39:43            1

Total number of neighbors 55

Table bgp, one route per one peer.

$ show ip bgp
BGP table version is 2401, local router ID is 192.168.122.216, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 100.0.0.0/24     10.0.0.4                 0             0 65002 ?
*> 100.0.1.0/24     10.0.1.4                 0             0 65002 ?
*> 100.0.2.0/24     10.0.2.4                 0             0 65002 ?
*> 100.0.3.0/24     10.0.3.4                 0             0 65002 ?
*> 100.0.4.0/24     10.0.4.4                 0             0 65002 ?
*> 100.0.5.0/24     10.0.5.4                 0             0 65002 ?
*> 100.0.6.0/24     10.0.6.4                 0             0 65002 ?
*> 100.0.7.0/24     10.0.7.4                 0             0 65002 ?
*> 100.0.8.0/24     10.0.8.4                 0             0 65002 ?
*> 100.0.9.0/24     10.0.9.4                 0             0 65002 ?
*> 100.0.10.0/24    10.0.10.4                0             0 65002 ?
*> 100.0.11.0/24    10.0.11.4                0             0 65002 ?
*> 100.0.12.0/24    10.0.12.4                0             0 65002 ?
*> 100.0.13.0/24    10.0.13.4                0             0 65002 ?
*> 100.0.14.0/24    10.0.14.4                0             0 65002 ?
*> 100.0.15.0/24    10.0.15.4                0             0 65002 ?
*> 100.0.16.0/24    10.0.16.4                0             0 65002 ?
*> 100.0.17.0/24    10.0.17.4                0             0 65002 ?
*> 100.0.18.0/24    10.0.18.4                0             0 65002 ?
*> 100.0.19.0/24    10.0.19.4                0             0 65002 ?
*> 100.0.20.0/24    10.0.20.4                0             0 65002 ?
*> 100.0.21.0/24    10.0.21.4                0             0 65002 ?
*> 100.0.22.0/24    10.0.22.4                0             0 65002 ?
*> 100.0.23.0/24    10.0.23.4                0             0 65002 ?
*> 100.0.24.0/24    10.0.24.4                0             0 65002 ?
*> 100.0.25.0/24    10.0.25.4                0             0 65002 ?
*> 100.0.26.0/24    10.0.26.4                0             0 65002 ?
*> 100.0.27.0/24    10.0.27.4                0             0 65002 ?
*> 100.0.28.0/24    10.0.28.4                0             0 65002 ?
*> 100.0.29.0/24    10.0.29.4                0             0 65002 ?
*> 100.0.30.0/24    10.0.30.4                0             0 65002 ?
*> 100.0.31.0/24    10.0.31.4                0             0 65002 ?
*> 100.0.32.0/24    10.0.32.4                0             0 65002 ?
*> 100.0.33.0/24    10.0.33.4                0             0 65002 ?
*> 100.0.34.0/24    10.0.34.4                0             0 65002 ?
*> 100.0.35.0/24    10.0.35.4                0             0 65002 ?
*> 100.0.36.0/24    10.0.36.4                0             0 65002 ?
*> 100.0.37.0/24    10.0.37.4                0             0 65002 ?
*> 100.0.38.0/24    10.0.38.4                0             0 65002 ?
*> 100.0.39.0/24    10.0.39.4                0             0 65002 ?
*> 100.0.40.0/24    10.0.40.4                0             0 65002 ?
*> 100.0.41.0/24    10.0.41.4                0             0 65002 ?
*> 100.0.42.0/24    10.0.42.4                0             0 65002 ?
*> 100.0.43.0/24    10.0.43.4                0             0 65002 ?
*> 100.0.44.0/24    10.0.44.4                0             0 65002 ?
*> 100.0.45.0/24    10.0.45.4                0             0 65002 ?
*> 100.0.46.0/24    10.0.46.4                0             0 65002 ?
*> 100.0.47.0/24    10.0.47.4                0             0 65002 ?
*> 100.0.48.0/24    10.0.48.4                0             0 65002 ?
*> 100.0.49.0/24    10.0.49.4                0             0 65002 ?
*> 100.0.50.0/24    10.0.50.4                0             0 65002 ?
*> 100.0.51.0/24    10.0.51.4                0             0 65002 ?
*> 100.0.52.0/24    10.0.52.4                0             0 65002 ?
*> 100.0.53.0/24    10.0.53.4                0             0 65002 ?
*> 100.0.54.0/24    10.0.54.4                0             0 65002 ?

Displayed  55 routes and 55 total paths

This can be mitigated by:

  1. add to ALL ipv4 peers "address-family ipv4-unicast" set
set protocols bgp xxxxxx neighbor x.x.x.x address-family ipv4-unicast
  1. disable default ipv4 peering
set protocols bgp xxxxxx parameters default no-ipv4-unicast

Duplicated with T2215

c-po moved this task from Need Triage to Backlog on the VyOS 1.3 Equuleus board.Jun 2 2020, 5:48 PM
c-po moved this task from Backlog to Finished on the VyOS 1.3 Equuleus board.Jun 2 2020, 7:37 PM