when using load balancing with a pppoe connection and another static or dhcp connection load balancing will never use the pppoe connection due to the way pppoe uses the iptables vs the other connection.
static routes with a different metric does not alleviate this as it would if one connection was dhcp and another static.
issue has persisted since original pull from vyatta codebase
i believe that in edgeos this issue is the primary reason load balancing is handled completely differently.
in edgeos each loadbalanced interface has it's ip table set directly at the interface-health section of the loadbalancing config. ex:
load-balance {
group <name> {
interface <name> {
route {
default
table <name>
}
weight <number>
failover-only
route-test {
type {
default
ping {
target <host>
}
script <name>
}
initial-delay <number>
interval <number>
count {
success <number>
failure <number>
}
}
}
}
}What version have you been using?
This can be achieved in VyOS 1.2 albeit it's harder to resolve. I unfortunately lost a config example with pppoe, static and dhcp over ethernet working correctly. It requires using static interface routes for dhcp to fix weights; pppoe force-gateway option; and an adjustment of the static routes weight to match the dhcp ones (this was four years ago).
priochange: I'd like to first see where the PR stems from, before proceeding.
been using the VyOS 1.2.0-beta1
i've never had this work successfully even on vyatta. back then i know they were working on a patch for this issue but it was never released.
static routes for the pppoe interface can't be done if the gateway changes (interface-routes don't work) not sure if static route x.x.x.x/x dhcp-interface would work either, i could not get either working with a pppoe tunnel.
the weights are only one issue, the other being even when the static routes were setup with the weight of 2 for both interfaces, the nat for both, and the loadbalancing configuration; only the dhcp interface was routable. it seems that the pppoe interface was not able to share the same iptable. (rather there was no way to set the weight of the pppoe interface to be the same as the dhcp one(a route of 0.0.0.0 with a weight of 2 did not work))
if there was a way to configure which iptable each interface used for outbound traffic i believe this would resolve the issue.
I added a force-gateway option some time ago. Regardless it's somewhat expected on 1.2, it needs testing and review. I meant 1.1.7 in my previous post (yes, confusion).
Tried editing our config. Could not find a force gateway option in the load balance sections. Can you confirm where the force gateway option would be?
https://community.ubnt.com/t5/EdgeRouter/Issue-NAT-Exclude-with-load-balance/m-p/628739#U628739 this example shows how edgeos has dealt with this issue.
effectively binding a route table to each wan interface. probably an iptable modify in the background.
Test #1:
vyos@vyos# run show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 - u/u
eth1 203.0.113.2/24 u/u
eth2 10.0.0.1/24 u/u
eth3 - u/u
lo 127.0.0.1/8 u/u
::1/128
pppoe0 198.51.100.101/32 u/u
[edit]
vyos@vyos# run show config comm | grep "ppp\|load"
set interfaces pppoe pppoe0 access-concentrator 'ACN'
set interfaces pppoe pppoe0 authentication password 'foo'
set interfaces pppoe pppoe0 authentication username 'foo'
set interfaces pppoe pppoe0 source-interface 'eth0'
set load-balancing wan interface-health eth1 nexthop '203.0.113.1'
set load-balancing wan interface-health pppoe0 nexthop '198.51.100.1'
set load-balancing wan rule 10 inbound-interface 'eth2'
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 interface pppoe0
[edit]
vyos@vyos# sudo ip rule
0: from all lookup local
32764: from all fwmark 0xca lookup 202
32765: from all fwmark 0xc9 lookup 201
32766: from all lookup main
32767: from all lookup default
[edit]
vyos@vyos# sudo nft list table ip mangle
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
chain WANLOADBALANCE_PRE {
iifname "eth2" ct state new meta random & 2147483647 < 1073741824 counter packets 277 bytes 17631 jump ISP_eth1
iifname "eth2" ct state new counter packets 260 bytes 16623 jump ISP_pppoe0
iifname "eth2" counter packets 1 bytes 84 meta mark set ct mark
}
chain PREROUTING {
type filter hook prerouting priority mangle; policy accept;
counter packets 1161 bytes 85454 jump WANLOADBALANCE_PRE
}
chain ISP_eth1 {
counter packets 277 bytes 17631 ct mark set 0xc9
counter packets 277 bytes 17631 meta mark set 0xc9
counter packets 277 bytes 17631 accept
}
chain ISP_pppoe0 {
counter packets 260 bytes 16623 ct mark set 0xca
counter packets 260 bytes 16623 meta mark set 0xca
counter packets 260 bytes 16623 accept
}
}
[edit]
vyos@vyos# sudo nft list table ip nat
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain VYOS_PRE_SNAT_HOOK {
type nat hook postrouting priority srcnat - 1; policy accept;
counter packets 513 bytes 32846 jump WANLOADBALANCE
}
chain WANLOADBALANCE {
ct mark 0xc9 counter packets 266 bytes 17003 snat to 203.0.113.2
ct mark 0xca counter packets 245 bytes 15723 snat to 198.51.100.101
}
}And traceroute from host, with IP 10.0.0.X attached to eth2:
vyos@HOST# run traceroute www.google.com traceroute to www.google.com (216.58.202.68), 30 hops max, 60 byte packets 1 10.0.0.1 (10.0.0.1) 0.353 ms 0.320 ms 0.314 ms 2 203.0.113.1 (203.0.113.1) 1.034 ms 198.51.100.1 (198.51.100.1) 1.013 ms 1.006 ms 3 elazul-p0r1nro01.elazul.local (192.168.0.1) 1.409 ms 1.476 ms 1.902 ms 4 192.168.100.1 (192.168.100.1) 2.255 ms 2.349 ms 2.399 ms 5 host2.186-13-10.telmex.net.ar (186.13.10.2) 4.828 ms 5.295 ms 5.227 ms 6 be5-2.c1900-br-05.claro.net.ar (170.51.254.172) 27.922 ms be5-2.cf223-br-05.claro.net.ar (170.51.254.176) 22.535 ms be5-2.c1900-br-05.claro.net.ar (170.51.254.172) 33.019 ms 7 142.250.47.162 (142.250.47.162) 26.209 ms 23.478 ms 142.250.165.154 (142.250.165.154) 25.694 ms 8 * * * 9 142.251.239.190 (142.251.239.190) 22.661 ms 142.251.239.154 (142.251.239.154) 25.749 ms 74.125.242.209 (74.125.242.209) 23.170 ms 10 74.125.242.200 (74.125.242.200) 24.448 ms 108.170.237.241 (108.170.237.241) 28.069 ms 74.125.242.217 (74.125.242.217) 28.035 ms 11 gru10s11-in-f4.1e100.net (216.58.202.68) 23.321 ms 172.253.53.17 (172.253.53.17) 24.143 ms 172.253.53.33 (172.253.53.33) 22.769 ms [edit] vyos@HOST# run traceroute www.google.com traceroute to www.google.com (216.58.202.68), 30 hops max, 60 byte packets 1 10.0.0.1 (10.0.0.1) 0.697 ms 0.665 ms * 2 198.51.100.1 (198.51.100.1) 1.334 ms 203.0.113.1 (203.0.113.1) 0.848 ms 0.845 ms 3 elazul-p0r1nro01.elazul.local (192.168.0.1) 1.588 ms 1.582 ms 1.718 ms 4 192.168.100.1 (192.168.100.1) 2.520 ms * 2.503 ms 5 host2.186-13-10.telmex.net.ar (186.13.10.2) 5.224 ms 5.660 ms 5.699 ms 6 be5-2.c1900-br-05.claro.net.ar (170.51.254.172) 31.393 ms be5-2.cf223-br-05.claro.net.ar (170.51.254.176) 31.993 ms be5-2.c1900-br-05.claro.net.ar (170.51.254.172) 26.143 ms 7 142.250.47.162 (142.250.47.162) 23.695 ms 23.489 ms 27.413 ms 8 * * * 9 74.125.242.209 (74.125.242.209) 25.237 ms 142.251.61.46 (142.251.61.46) 24.253 ms 142.251.239.146 (142.251.239.146) 27.112 ms 10 142.250.46.111 (142.250.46.111) 27.108 ms 108.170.237.241 (108.170.237.241) 28.657 ms 27.356 ms 11 eze06s09-in-f4.1e100.net (216.58.202.68) 24.586 ms 21.382 ms 172.253.53.17 (172.253.53.17) 25.747 ms [edit] vyos@HOST#
Also, correct behavior when 1 interface is pppoe, and other dhcp-client:
vyos@vyos# run show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 - u/u
eth1 203.0.113.151/24 u/u
eth2 10.0.0.1/24 u/u
eth3 - u/u
lo 127.0.0.1/8 u/u
::1/128
pppoe0 198.51.100.101/32 u/u
[edit]
vyos@vyos# run show config comm | grep "addr\|ppp\|wan"
set interfaces ethernet eth1 address 'dhcp'
set interfaces ethernet eth2 address '10.0.0.1/24'
set interfaces pppoe pppoe0 access-concentrator 'ACN'
set interfaces pppoe pppoe0 authentication password 'foo'
set interfaces pppoe pppoe0 authentication username 'foo'
set interfaces pppoe pppoe0 source-interface 'eth0'
set load-balancing wan interface-health eth1 nexthop 'dhcp'
set load-balancing wan interface-health pppoe0 nexthop '198.51.100.1'
set load-balancing wan rule 10 inbound-interface 'eth2'
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 interface pppoe0
[edit]And test from client 10.0.0.X:
vyos@HOST# run traceroute vyos.dev traceroute to vyos.dev (104.18.20.61), 30 hops max, 60 byte packets 1 10.0.0.1 (10.0.0.1) 0.420 ms 0.471 ms 0.410 ms 2 203.0.113.1 (203.0.113.1) 1.080 ms 198.51.100.1 (198.51.100.1) 1.687 ms 203.0.113.1 (203.0.113.1) 1.110 ms 3 elazul-p0r1nro01.elazul.local (192.168.0.1) 1.972 ms 1.779 ms 1.813 ms 4 192.168.100.1 (192.168.100.1) 2.626 ms 2.620 ms 2.524 ms 5 host2.186-13-10.telmex.net.ar (186.13.10.2) 6.073 ms 6.068 ms 6.123 ms 6 10.2.187.165 (10.2.187.165) 25.672 ms 10.2.187.161 (10.2.187.161) 25.036 ms 25.476 ms 7 host59.170-51-254.telmex.net.ar (170.51.254.59) 24.340 ms 22.982 ms 23.588 ms 8 104.18.20.61 (104.18.20.61) 24.399 ms 24.760 ms 23.823 ms [edit] vyos@HOST# run traceroute vyos.dev traceroute to vyos.dev (104.18.20.61), 30 hops max, 60 byte packets 1 10.0.0.1 (10.0.0.1) 0.239 ms 0.214 ms 0.226 ms 2 198.51.100.1 (198.51.100.1) 1.681 ms 1.606 ms 1.536 ms 3 elazul-p0r1nro01.elazul.local (192.168.0.1) 1.833 ms 1.851 ms 2.022 ms 4 192.168.100.1 (192.168.100.1) 2.645 ms 2.589 ms 2.443 ms 5 host2.186-13-10.telmex.net.ar (186.13.10.2) 4.778 ms 5.145 ms 4.683 ms 6 10.2.187.165 (10.2.187.165) 25.074 ms 10.2.187.161 (10.2.187.161) 25.864 ms 26.215 ms 7 host59.170-51-254.telmex.net.ar (170.51.254.59) 25.331 ms 25.097 ms 25.010 ms 8 104.18.20.61 (104.18.20.61) 25.975 ms 25.776 ms 30.063 ms [edit] vyos@HOST#