Page MenuHomeVyOS Platform

BGP Route Reflects to all neighbors when one neighbor has route-reflect-client
In progress, NormalPublicBUG

Description

On 1.3.0-rc6 and 1.3.0-epa1

When any BGP neighbor has route-reflector-client set, all neighbors received reflected routes.

This can cause border BGP routers to leak routes to wider network

Details

Difficulty level
Easy (less than an hour)
Version
1.3.0-epa1
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change
Issue type
Bug (incorrect behavior)

Event Timeline

Also tested 1.4-rolling-202110020217 and it exhibits the same issue

@francis It is not clear. Can you provide an example of configuration? What do you get and what do you expect?

fernando removed a subscriber: fernando.
fernando added a subscriber: fernando.

Devices:

RR1 - BGP Peer / Route Reflector / 10.0.0.1
RR2 - BGP Peer / Route Reflector / 10.0.0.2
RR3 - BGP Peer / Route Reflector / 10.0.0.3

P1 - BGP Peer / 10.0.0.21

Config:

P1 Config:

protocols {
    bgp 11111 {
        address-family {
            ipv4-unicast {
                network 10.0.0.100/32 {
                }
                network 10.0.0.101/32 {
                }
                network 192.168.0/24 {
                }
            }
        }
        neighbor 10.0.0.1 {
            address-family {
                ipv4-unicast {
                    weight 100
                }
            }
            remote-as 11111
        }
        neighbor 10.0.0.2 {
            address-family {
                ipv4-unicast {
                    weight 100
                }
            }
            remote-as 11111
        }
        parameters {
            bestpath {
                as-path {
                    multipath-relax
                }
            }
            router-id 10.0.0.21
        }
    }

RR1 Config:

protocols {
    bgp 11111 {
        address-family {
            ipv4-unicast {
                network 10.0.0.1/32 {
                }
                network 10.0.0.9/32 {
                }
            }
        }
        neighbor 10.0.0.6 {
            address-family {
                ipv4-unicast {
                    nexthop-self {
                        force {
                        }
                    }
                    route-reflector-client {
                    }
                    weight 50
                }
            }
            remote-as 11111
        }
        neighbor 10.0.0.21 {
            address-family {
                ipv4-unicast {
                    weight 100
                }
            }
            remote-as 11111
        }
        parameters {
            bestpath {
                as-path {
                    multipath-relax
                }
            }
            router-id 10.0.0.1
        }
    }

RR2 Config:

protocols {
    bgp 11111 {
        address-family {
            ipv4-unicast {
                network 10.0.0.2/32 {
                }
                network 10.0.0.9/32 {
                }
            }
        }
        neighbor 10.0.0.6 {
            address-family {
                ipv4-unicast {
                    nexthop-self {
                        force {
                        }
                    }
                    route-reflector-client {
                    }
                    weight 50
                }
            }
            remote-as 11111
        }
        neighbor 10.0.0.21 {
            address-family {
                ipv4-unicast {
                    weight 100
                }
            }
            remote-as 11111
        }
        parameters {
            bestpath {
                as-path {
                    multipath-relax
                }
            }
            router-id 10.0.0.2
        }
    }

RR3 Config:

protocols {
    bgp 11111 {
        address-family {
            ipv4-unicast {
                network 10.0.0.6/32 {
                }
                network 10.0.0.9/32 {
                }
            }
        }
        neighbor 10.0.0.1 {
            address-family {
                ipv4-unicast {
                    nexthop-self {
                        force {
                        }
                    }
                    route-reflector-client {
                    }
                    weight 50
                }
            }
            remote-as 11111
        }
        neighbor 10.0.0.2 {
            address-family {
                ipv4-unicast {
                    nexthop-self {
                        force {
                        }
                    }
                    route-reflector-client {
                    }
                    weight 50
                }
            }
            remote-as 11111
        }
        parameters {
            bestpath {
                as-path {
                    multipath-relax
                }
            }
            router-id 10.0.0.6
        }
    }

The issue is that on 10.0.0.21, routes are learned from RR3 even though there is no connection to that BGP device, and learned routes should NOT be propagated to this BGP peer from 10.0.0.1 or 10.0.0.2 since it's not a reflector client.

Here is the expected BGP Output of show ip bgp on 10.0.0.21 (non route reflector client)

BGP table version is 405, local router ID is 10.0.0.21, vrf id 0
Default local pref 100, local AS 11111
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*=10.0.0.1/32   10.0.0.1              0    100    100 i
*>10.0.0.2/32   10.0.0.2              0    100    100 i

vs Actual:

BGP table version is 405, local router ID is 10.0.0.21, vrf id 0
Default local pref 100, local AS 11111
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*=10.0.0.1/32   10.0.0.1              0    100    100 i
*>i                 10.0.0.2              0    100    100 i
*= 10.0.0.2/32   10.0.0.1              0    100    100 i
*>i                 10.0.0.2              0    100    100 i
*> 10.0.0.6/32   10.0.0.1              0    100    100 i
*=i                 10.0.0.2              0    100    100 i

Neighbors that are not route-reflector-client should not receive learned routes, only routes that are explicitly set in the config. So in the example above, I would not expect to have multipath routes for 10.0.0.1 and 10.0.0.2, and I would not expect to see any route for 10.0.0.6

This creates routing loops on all BGP neighbors as they are all advertising the same routes

there is a recommendation that if you use RR in the same hierarchy and avoid loop , we need to set 'cluster-id'

A cluster is a collection of route reflectors and their clients, and is used by route reflectors to avoid looping.

bgp cluster-id A.B.C.D

and It's the way to set on Vyos:

 set protocols bgp <asn> parameters cluster-id <id>

This command specifies cluster ID which identifies a collection of route reflectors and their clients, and is used by route reflectors to avoid looping. By default cluster ID is set to the BGP router id value, but can be set to an arbitrary 32-bit value.

Although a cluster ID might be helpful the real problem is that the routes are reflected to all peers – not just ones that are route reflector clients:

When you configure a route reflector you have to tell the router whether the other IBGP router is a client or non-client. A client is an IBGP router that the route reflector will “reflect” routes to, the non-client is just a regular IBGP neighbor.

Viacheslav changed the task status from Open to Needs testing.Oct 8 2021, 7:08 AM
Unknown Object (User) added a subscriber: Unknown Object (User).Oct 15 2021, 1:42 PM

If Cluster ID is not used, full IBGP mesh must be used. Exception is RR client, they should only have peering with RR.
Router 10.0.0.21 has no peering with 10.0.0.3.
This is incorrect IBGP design.

This statement confirms what I said above:

When you configure a route reflector you have to tell the router whether the other IBGP router is a client or non-client. A client is an IBGP router that the route reflector will “reflect” routes to, the non-client is just a regular IBGP neighbor.

Thus 10.0.0.21 and 10.0.0.3 must have IBGP peering.

Therefore, this is not a VyOS bug, but the wrong network design.

@NikolayP The concern here is solely this: IBGP neighbors that do not have route-reflector-client set should not received learned routes. Currently, they do. If this in intentional, then the docs should be updated to clarify this.

but the wrong network design.

This config provided is a simplified example. Real world use case is this: geo separated backbone routers that connect to local peering routers. The design intentionally doesn't have a full mesh because that isn't the goal.

Unknown Object (User) added a comment.Oct 15 2021, 2:58 PM

@francis Sorry, I don't understand the problem.
Agree that route received from one IBGP peer should not be forwarded to another IBGP peer. Except for the RR client.

I still think fullmesh IBGP is required. But I also agree with the statement above.
I will try to check the behavior of VyOS

Francis

In the real-world to avoid it they used cluster-id / a session BGP between them , it's the idea of RR :

An RR reflects routes between these groups, and may reflect routes
  among client peers.  An RR along with its client peers form a
  cluster.  The Non-Client peer must be fully meshed but the Client
  peers need not be fully meshed.

if you use cluster-id :

RR receives the routes from an IBGP neighbor which has the same cluster ID, routes are discarded.

https://datatracker.ietf.org/doc/html/rfc4456#page-6

Unknown Object (User) added a comment.Oct 16 2021, 3:02 AM

Tested on VyOS 1.3.0-epa1.
Confirm IBGP reflection to non-RR-Client
Lab Topology:

image.png (441×771 px, 89 KB)

RR1 & RR2 -route reflectors
P 3 - RR-Client for RR1 & RR2
P1 - IBGP peering with RR1 only
OSPF-core router - only for core network
Result: P1 gets P 3 routes fron RR1:
vyos@VyOS-P1:~$ sh ip bgp neighbors 10.0.0.1 received-routes
*> 10.0.0.201/32 10.0.0.3 0 100 100 i
*> 10.0.0.202/32 10.0.0.3 0 100 100 i
*> 192.168.3.0/24 10.0.0.3 0 100 100 i

Interestingly, the routes from P1 does not come to RR2 (works as it should)
Also P1 does not get routes from RR2 (works as it should)
Only routes from RR-client P 3 comes via RR1 to P1

Configs here:

set system host-name 'VyOS-P1'
set interfaces dummy dum100 address '10.0.0.100/32'
set interfaces dummy dum101 address '10.0.0.101/32'
set interfaces dummy dum192 address '192.168.0.21/24'
set interfaces ethernet eth0 address '4.4.4.4/24'
set interfaces loopback lo address '10.0.0.21/32'
set protocols bgp 11111 address-family ipv4-unicast network 10.0.0.100/32
set protocols bgp 11111 address-family ipv4-unicast network 10.0.0.101/32
set protocols bgp 11111 address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp 11111 neighbor 10.0.0.1 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp 11111 neighbor 10.0.0.1 address-family ipv4-unicast weight '100'
set protocols bgp 11111 neighbor 10.0.0.1 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.1 update-source 'lo'
set protocols bgp 11111 parameters bestpath as-path multipath-relax
set protocols bgp 11111 parameters router-id '10.0.0.21'
set protocols ospf area 0 network '4.4.4.0/24'
set protocols ospf area 0 network '10.0.0.21/32'


set system host-name 'VyOS-RR1'
set interfaces ethernet eth1 address '1.1.1.1/24'
set interfaces loopback lo address '10.0.0.1/32'
set protocols bgp 11111 neighbor 10.0.0.2 address-family ipv4-unicast
set protocols bgp 11111 neighbor 10.0.0.2 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.2 update-source 'lo'
set protocols bgp 11111 neighbor 10.0.0.3 address-family ipv4-unicast route-reflector-client
set protocols bgp 11111 neighbor 10.0.0.3 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.3 update-source 'lo'
set protocols bgp 11111 neighbor 10.0.0.21 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.21 update-source 'lo'
set protocols bgp 11111 parameters bestpath as-path multipath-relax
set protocols bgp 11111 parameters router-id '10.0.0.1'
set protocols ospf area 0 network '10.0.0.1/32'
set protocols ospf area 0 network '1.1.1.0/24'


set system host-name 'VyOS-RR2'
set interfaces ethernet eth2 address '2.2.2.2/24'
set interfaces loopback lo address '10.0.0.2/32'
set protocols bgp 11111 neighbor 10.0.0.1 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.1 update-source 'lo'
set protocols bgp 11111 neighbor 10.0.0.3 address-family ipv4-unicast route-reflector-client
set protocols bgp 11111 neighbor 10.0.0.3 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.3 update-source 'lo'
set protocols bgp 11111 neighbor 10.0.0.21 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.21 update-source 'lo'
set protocols bgp 11111 parameters bestpath as-path multipath-relax
set protocols bgp 11111 parameters router-id '10.0.0.2'
set protocols ospf area 0 network '10.0.0.2/32'
set protocols ospf area 0 network '2.2.2.0/24'


set system host-name 'VyOS-P3'
set interfaces dummy dum192 address '192.168.3.3/24'
set interfaces dummy dum201 address '10.0.0.201/32'
set interfaces dummy dum202 address '10.0.0.202/32'
set interfaces ethernet eth3 address '3.3.3.3/24'
set interfaces loopback lo address '10.0.0.3/32'
set protocols bgp 11111 address-family ipv4-unicast network 10.0.0.201/32
set protocols bgp 11111 address-family ipv4-unicast network 10.0.0.202/32
set protocols bgp 11111 address-family ipv4-unicast network 192.168.3.0/24
set protocols bgp 11111 neighbor 10.0.0.1 address-family ipv4-unicast nexthop-self force
set protocols bgp 11111 neighbor 10.0.0.1 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.1 update-source 'lo'
set protocols bgp 11111 neighbor 10.0.0.2 address-family ipv4-unicast nexthop-self force
set protocols bgp 11111 neighbor 10.0.0.2 remote-as '11111'
set protocols bgp 11111 neighbor 10.0.0.2 update-source 'lo'
set protocols bgp 11111 parameters bestpath as-path multipath-relax
set protocols bgp 11111 parameters router-id '10.0.0.3'
set protocols ospf area 0 network '10.0.0.3/32'
set protocols ospf area 0 network '3.3.3.0/24'
syncer changed the task status from Needs testing to In progress.Oct 17 2021, 1:13 PM
syncer triaged this task as Normal priority.