Page MenuHomePhabricator

Current multi-table usage with VRF-netns tables in FRR is partially broken for PBR.
Open, HighPublicBUG


VRF-netns table routes are seemingly not capable of using interfaces routes defined in them to determine route reach-ability.

control@edg-rt01a# run sh ip route table 10
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route

S> [1/0] via (recursive), 00:01:05
  *                   unreachable, 00:01:05
S>* [1/0] is directly connected, eth0.252, 00:00:09

Route reach-ability can be re-established using a combined interface and next-hop route type.

control@edg-rt01a:~$ sh ip route table 10
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route

S>* [1/0] via, eth0.252, 00:01:46

Combined interface and next-hop route types are not producible with the current routing configuration syntax.
FRR will attempt to auto-determine the appropriate interface the first time the route is created, but this will not hold on a reboot because the VyOS side routing configuration does not store this information.


Difficulty level
Normal (likely a few hours)
Why the issue appeared?
Will be filled on close

Further examination of this issue indicates that some of the behavior may be caused by FRR starting prior to VRRP on boot, and backup VRRP routers with VIP only interfaces.

syncer triaged this task as High priority.Oct 11 2018, 5:57 AM
syncer moved this task from Needs Triage to Backlog on the VyOS 1.2 Crux (VyOS 1.2.0-rc2) board.

@Watcher7 I do not really understand what you mean, can you share your configs or a way to reproduce and elaborate a bit more?

Watcher7 added a comment.EditedOct 11 2018, 4:56 PM

Yes. It's a double issue actually. Hopefully this explains it better:

  1. The next-hop based routes in the alternate routing tables seem to be unaware of interface routes in the same table.
  2. VyOS command syntax cannot currently specify both a next-hop and interface for the same static route, despite FRR being able to do so.
    • FRR will attempt to add an interface to a next-hop route (based on which interface has a subnet that includes the next hop) automatically, but this information is not preserved in the VyOS config file.
    • Since FRR starts prior to VRRP (keepalived); interfaces with ONLY 'virtual' addresses will not receive FRR's automatic interface detection because they do not have a subnet when the route is created. This renders the routes unreachable and FRR does not refresh their status.

The last two parts of issue number 2 would be probably hard to "fix". The automatic interface detection is probably hard to track because I imagine it's internal FRR state, and FRR starting prior to VRRP is probably the correct behavior that should be kept.

Modifying the static route syntax would temporarily fix both issues by allowing a user to manually specify both an interface and a next-hop (thus preserving the state in the vyos config file).

syncer added a subscriber: syncer.Oct 11 2018, 4:58 PM

Don't think we need to modify anything
vrrp must start before frr starts

You're probably right. I thought I had a potential conflict if it started first, but now I can't think of what it was. Maybe there wasn't a conflict to begin with.
Being able to apply both an interface and next hop to the same route would still be extremely useful though, but could be slated for later I guess.
There is still the question of why FRR can't figure out what to do with separate interface routes in the same table.

UnicronNL added a comment.EditedOct 12 2018, 5:21 PM


do you mean like:

set protocols static table 10 route next-hop interface ethx

If so could you please create a new task for creating new syntax?

Also would be nice if you could provide an example configuration to start off with.

starting vrrp before frr is hard, maybe not possible due to the vtysh commands we use in the configuration scripts

I had already created a task for a new syntax and linked it as a related task, would you like me to create a new separate one?

@Watcher7 Ah i see, was confused... tnx

pasik added a subscriber: pasik.Oct 15 2018, 9:51 PM
syncer assigned this task to dmbaturin.Oct 18 2018, 5:12 AM
adestis added a subscriber: adestis.Feb 9 2019, 6:11 AM