Page MenuHomeVyOS Platform

Current multi-table usage with VRF-netns tables in FRR is partially broken for PBR.
Resolved (N/A)PublicBUG

Description

VRF-netns table routes are seemingly not capable of using interfaces routes defined in them to determine route reach-ability.

control@edg-rt01a# run sh ip route table 10
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route

S>  0.0.0.0/0 [1/0] via 10.7.252.1 (recursive), 00:01:05
  *                   unreachable, 00:01:05
S>* 10.7.252.0/24 [1/0] is directly connected, eth0.252, 00:00:09

Route reach-ability can be re-established using a combined interface and next-hop route type.

control@edg-rt01a:~$ sh ip route table 10
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route

S>* 0.0.0.0/0 [1/0] via 10.7.252.1, eth0.252, 00:01:46

Combined interface and next-hop route types are not producible with the current routing configuration syntax.
FRR will attempt to auto-determine the appropriate interface the first time the route is created, but this will not hold on a reboot because the VyOS side routing configuration does not store this information.

Details

Difficulty level
Normal (likely a few hours)
Version
1.2.0-rc2
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Event Timeline

Further examination of this issue indicates that some of the behavior may be caused by FRR starting prior to VRRP on boot, and backup VRRP routers with VIP only interfaces.

syncer triaged this task as High priority.Oct 11 2018, 5:57 AM
syncer moved this task from Needs Triage to Backlog on the VyOS 1.2 Crux (VyOS 1.2.0-rc2) board.

@Watcher7 I do not really understand what you mean, can you share your configs or a way to reproduce and elaborate a bit more?

@UnicronNL
Yes. It's a double issue actually. Hopefully this explains it better:

  1. The next-hop based routes in the alternate routing tables seem to be unaware of interface routes in the same table.
  2. VyOS command syntax cannot currently specify both a next-hop and interface for the same static route, despite FRR being able to do so.
    • FRR will attempt to add an interface to a next-hop route (based on which interface has a subnet that includes the next hop) automatically, but this information is not preserved in the VyOS config file.
    • Since FRR starts prior to VRRP (keepalived); interfaces with ONLY 'virtual' addresses will not receive FRR's automatic interface detection because they do not have a subnet when the route is created. This renders the routes unreachable and FRR does not refresh their status.

The last two parts of issue number 2 would be probably hard to "fix". The automatic interface detection is probably hard to track because I imagine it's internal FRR state, and FRR starting prior to VRRP is probably the correct behavior that should be kept.

Modifying the static route syntax would temporarily fix both issues by allowing a user to manually specify both an interface and a next-hop (thus preserving the state in the vyos config file).

Don't think we need to modify anything
vrrp must start before frr starts

You're probably right. I thought I had a potential conflict if it started first, but now I can't think of what it was. Maybe there wasn't a conflict to begin with.
Being able to apply both an interface and next hop to the same route would still be extremely useful though, but could be slated for later I guess.
There is still the question of why FRR can't figure out what to do with separate interface routes in the same table.

@Watcher7

do you mean like:

set protocols static table 10 route 192.168.0.0/24 next-hop 192.168.0.1 interface ethx

If so could you please create a new task for creating new syntax?

Also would be nice if you could provide an example configuration to start off with.

starting vrrp before frr is hard, maybe not possible due to the vtysh commands we use in the configuration scripts

I had already created a task for a new syntax and linked it as a related task, would you like me to create a new separate one?

syncer added a subscriber: dmbaturin.
syncer added a subscriber: zsdc.
dmbaturin set Is it a breaking change? to Unspecified (possibly destroys the router).
  1. VyOS command syntax cannot currently specify both a next-hop and interface for the same static route, despite FRR being able to do so.

This option was added.

vyos@r4# set protocols static route 203.0.113.1/32 next-hop 192.0.2.1 
Possible completions:
   disable      Disable IPv4 next-hop static route
   distance     Distance value for this route
   next-hop-interface
                IPv4 gateway interface name
   next-hop-vrf VRF to leak route

@Watcher7 Could you re-check? If you still have these issues, provide please a configuration on how to reproduce it.
Thanks.

Viacheslav lowered the priority of this task from High to Normal.Dec 24 2021, 9:21 AM

Close the task
@Watcher7 Re-test it or describe steps hot to reproduce, as since 1.2-rc2 was implemented a lot of changes regarding vrf + frr.
You can set both vrf + next-hop address

Re-open it if necessary.