WAN load-balancing exclude rules break PBR
Open, HighPublicBUG
Actions

Assigned To

None

Authored By

	zsdc
	May 31 2022, 11:01 AM

Description

For exclude type of rules in load-balancing the firewall verdict statement is accept. In nftables, this is terminal statement and means that no other action with a packet will be performed in the same hook, if only we do not have other chain with higher priority.
The simple config that can be used to show the problem:

set interfaces ethernet eth0 address '192.168.0.2/24'
set interfaces ethernet eth1 address '192.168.1.1/24'
set interfaces tunnel tun0 address '192.168.72.20/31'
set interfaces tunnel tun0 description 'magic-wan'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 mtu '1476'
set interfaces tunnel tun0 remote '192.168.0.1'
set interfaces tunnel tun0 source-address '192.168.0.2'
set load-balancing wan disable-source-nat
set load-balancing wan flush-connections
set load-balancing wan interface-health eth0 failure-count '2'
set load-balancing wan interface-health eth0 nexthop '192.168.0.1'
set load-balancing wan interface-health eth0 success-count '1'
set load-balancing wan interface-health eth0 test 10 resp-time '5'
set load-balancing wan interface-health eth0 test 10 target '8.8.8.8'
set load-balancing wan interface-health eth0 test 10 ttl-limit '1'
set load-balancing wan interface-health eth0 test 10 type 'ping'
set load-balancing wan rule 20 destination address '0.0.0.0/0'
set load-balancing wan rule 20 exclude
set load-balancing wan rule 20 inbound-interface 'eth1'
set load-balancing wan rule 20 protocol 'all'
set load-balancing wan rule 40 failover
set load-balancing wan rule 40 inbound-interface 'eth1'
set load-balancing wan rule 40 interface eth0 weight '10'
set load-balancing wan rule 40 protocol 'all'
set load-balancing wan sticky-connections inbound
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 translation address 'masquerade'
set nat source rule 200 outbound-interface 'tun0'
set nat source rule 200 translation address 'masquerade'
set policy route magic-wan rule 100 set table '100'
set protocols static route 0.0.0.0/0 next-hop 192.168.0.1
set protocols static table 100 route 0.0.0.0/0 next-hop 192.168.72.21

Here we exclude all the traffic from load-balancing for clear view. This config will generate the next nftables table:

table ip mangle {
	chain PREROUTING {
		type filter hook prerouting priority mangle; policy accept;
		iifname "eth0" ct state new counter packets 0 bytes 0 jump ISP_eth0_IN
		counter packets 68 bytes 5339 jump WANLOADBALANCE_PRE
		counter packets 52 bytes 4336 jump VYATTA_FW_IN_HOOK
	}

	chain INPUT {
		type filter hook input priority mangle; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority mangle; policy accept;
	}

	chain OUTPUT {
		type route hook output priority mangle; policy accept;
		counter packets 35 bytes 2484 jump VYATTA_FW_LOCALOUT_HOOK
	}

	chain POSTROUTING {
		type filter hook postrouting priority mangle; policy accept;
		counter packets 74 bytes 5699 jump VYATTA_FW_OUT_HOOK
	}

	chain VYATTA_FW_OUT_HOOK {
	}

	chain VYATTA_FW_IN_HOOK {
		iifname "eth1" counter packets 0 bytes 0 jump magic-wan
	}

	chain VYATTA_FW_LOCALOUT_HOOK {
	}

	chain magic-wan {
		counter packets 0 bytes 0 jump VYATTA_PBR_100 comment "magic-wan-100"
		counter packets 0 bytes 0 return comment "magic-wan-1000000 default-action accept"
	}

	chain VYATTA_PBR_100 {
		counter packets 0 bytes 0 meta mark set 0x80000063 
		counter packets 0 bytes 0 accept
	}

	chain WANLOADBALANCE_PRE {
		iifname "eth1" counter packets 21 bytes 1435 accept
		iifname "eth1" ct state new counter packets 0 bytes 0 jump ISP_eth0
		iifname "eth1" counter packets 0 bytes 0 meta mark set ct mark
	}

	chain ISP_eth0 {
		counter packets 0 bytes 0 ct mark set 0xc9 
		counter packets 0 bytes 0 meta mark set 0xc9 
		counter packets 0 bytes 0 accept
	}

	chain ISP_eth0_IN {
		counter packets 0 bytes 0 ct mark set 0xc9 
	}
}

In the table we have only one base chain in a hook prerouting that contains both load-balancing rules and PBR. Since load-balancing is executed first, packets accepted by exclude rules will never reach PBR.