Page MenuHomeVyOS Platform

Policy Based Route Issue with Rules for Multiple Tables
Needs reporter action, LowPublicBUG

Description

The Cable Modem provided by my ISP has a default IP that can be used to get status information about the modem. In my setup I have two WAN connections my device flow is <End User Devices> --> <Firewall pfSense> --> <Router VyOS> --> <Cable Modem>. To access the Cable Modem on that preassigned IP, the flow is <End User Devices> --> <Firewall pfSense> --> <Router VyOS> --> <Cable Modem>.

I have the relevant config below. I have removed IP specifics from the outputs.

If the ISP connected to that Cable modem is up traffic comes into the VyOS on 'bond1.221' and leaves on bond1.211'. One issue is when that ISP is down my firewall sends traffic through the other WAN interface which comes into the VyOS on bond1.121. 'set policy route FIOS_01_INET rule 1001' controls traffic coming into that interface which would forward the traffic to the other ISP instead of the Cable Modem. To address this I created 'set policy route FIOS_01_INET rule 1000' which then should route traffic to the IP of the Cable Modem to bond1.121.

However, when I added that rule, I could access the Cable Modem, but traffic wouldn't flow to the ISP. I rebooted the VyOS and traffic would then flow to the ISP, but no longer to the Cable Modem. I am running vyos-1.3.0-rc6-amd64. I was going to test this on vyos-1.3.0-amd64, but per T4206 that I posted earlier, I can use that version.

falkor@nenyas-edge-01:~$ sh ip route 192.168.x.1
Routing entry for 192.168.x.0/24
  Known via "connected", distance 0, metric 0, best
  Last update 01:02:03 ago
  * directly connected, bond1.121

falkor@nenyas-edge-01:~$


falkor@nenyas-edge-01:~$ show ip route table 111
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 111:
S>* 0.0.0.0/0 [1/0] via x.x.x.1, bond1.111, weight 1, 00:28:40
falkor@nenyas-edge-01:~$


falkor@nenyas-edge-01:~$ show conf commands | match 'policy route|source rule 1001|table'
set interfaces bonding bond1 vif 211 policy route 'FIOS_01_INET'
set interfaces bonding bond1 vif 221 policy route 'WOW_01_INET'
set nat source rule 1001 description 'NAT Traffic to 192.168.100.0/24'
set nat source rule 1001 destination address '192.168.x.0/24'
set nat source rule 1001 outbound-interface 'bond1.121'
set nat source rule 1001 translation address '192.168.x.x'
set policy route FIOS_01_INET rule 1000 description 'Route traffic to ISP Modem (192.168.x.x) to ISP Interface'
set policy route FIOS_01_INET rule 1000 destination address '192.168.x.0/24'
set policy route FIOS_01_INET rule 1000 disable
set policy route FIOS_01_INET rule 1000 set table 'main'
set policy route FIOS_01_INET rule 1001 description 'Route traffic from the specified subnets through FIOS_01_INET'
set policy route FIOS_01_INET rule 1001 set table '111'
set policy route FIOS_01_INET rule 1001 source group network-group 'FIOS_01_INET_NETS'
set policy route WOW_01_INET rule 1001 description 'Route traffic from the specified subnets through WOW_01_INET'
set policy route WOW_01_INET rule 1001 set table '121'
set policy route WOW_01_INET rule 1001 source group network-group 'WOW_01_INET_NETS'
set protocols static table 111 route 0.0.0.0/0 dhcp-interface 'bond1.111'
set protocols static table 121 route 0.0.0.0/0 next-hop x.x.x.x
falkor@nenyas-edge-01:~$

Details

Difficulty level
Unknown (require assessment)
Version
1.3.0 rc6
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

For first do these changes as in commit
Try policy local route, for example:

set policy local-route rule 10 set table 111
set policy local-route rule 10 source 192.0.2.0/24

It more looks like you need failover with custom hook-scripts

Failover is handled by my firewall which is upstream of VyOS which I am using more as a router than anything. The commit you listed I believe is actually the fix for T4206, not for this, but I can certainly try that to see if I'm up and running and to see if the issue I'm reporting here is resolved, since I have only tried this setup in 1.3.0 RC6. I'm not sure why you'd think I'd need " failover with custom hook-scripts" for this issue. All I'm trying to do is have a PBR for traffic with the destination IP of local VyOS interfaces to use the main table rather than the vrf table. I also have an issue where if I ping the IP on the FIOS WAN interface from upstream, the reply traffic from the VyOS is sent downstream to the FiOS gateway, so this fails. However, the VyOS isn't doing that for the WOW! WAN interface, and I get the replies as expected. So it seems there are strange things happening. Either things not being cleaned up and/or not being set up right.

Viacheslav changed the task status from Open to Needs reporter action.Jan 20 2024, 10:47 AM
Viacheslav triaged this task as Low priority.

@Rhongomiant Is it still relevant?