This may be more of a linux bug or require some tuning to resolve. I have multiple ISPs so when troubleshooting issues on specific paths, I often use curl, mtr, ping, telnet, and traceroute and specify a source IP. I find that they only work for the first two WAN interfaces. The 3rd interface fails. It seems to be tied to the metric of the default route associated with the interface. I know it's not an issue with that specific interface/default route because if I change the metric on the 3rd default route so that it is one of the first two, it and the other one in the first two work, but now the interface that has the 3rd default route fails.
Below are examples.
In example set 1, you can see that the default route for bond1.111 is 1st and the speedtest for that interface succeeds, the default route for bond1.121 is 2nd and the speedtest for that interface succeeds, and the default route eth0.10 is 3rd and the speedtest for that interface fails with errors.
In example set 2, you can see that the default route for bond1.111 is 2nd and the speedtest for that interface succeeds, the default route for bond1.121 is 3rd and the speedtest for that interface fails with errors, and the default route eth0.10 is 1st and the speedtest for that interface succeeds.
I am running vyos-1.3.0-rc6-amd64. I was going to test this on vyos-1.3.0-amd64, but per T4206 that I posted earlier, I can use that version.
Example Set 1:
root@nenyas-edge-01:~/ookla-speedtest-1.1.1# show ip route | head -9 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup S 0.0.0.0/0 [210/0] via x.x.x1, bond1.111, weight 1, 01:15:17 S>* 0.0.0.0/0 [10/0] via x.x.x.105, bond1.121, weight 1, 01:15:39 S 0.0.0.0/0 [20/0] via 10.x.x.1, eth0.10, weight 1, 01:15:39 root@nenyas-edge-01:~/ookla-speedtest-1.1.1# root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.111 Speedtest by Ookla Server: Frontier - Tampa, FL (id = 27513) ISP: Frontier Communications Latency: 1.68 ms (0.09 ms jitter) Download: 77.20 Mbps (data used: 90.9 MB ) Upload: 91.80 Mbps (data used: 114.2 MB ) Packet Loss: 0.0% Result URL: https://www.speedtest.net/result/c/d7df8668-a146-4cb0-9d98-48456e322328 root@nenyas-edge-01:~/ookla-speedtest-1.1.1# root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.121 Speedtest by Ookla Server: Velocity Online - Tallahassee, FL (id = 884) ISP: WideOpenWest Latency: 28.09 ms (1.90 ms jitter) Download: 537.66 Mbps (data used: 943.2 MB ) Upload: 13.56 Mbps (data used: 22.6 MB ) Packet Loss: 0.0% Result URL: https://www.speedtest.net/result/c/cd3be733-3f02-40e7-bb92-007ab26e0ad4 root@nenyas-edge-01:~/ookla-speedtest-1.1.1#
Example Set 2:
root@nenyas-edge-01:~# show ip route | head -9 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup S>* 0.0.0.0/0 [5/0] via 10.x.x.1, eth0.10, weight 1, 00:00:26 S 0.0.0.0/0 [210/0] via x.x.x.1, bond1.111, weight 1, 01:21:07 S 0.0.0.0/0 [10/0] via x.x.x.105, bond1.121, weight 1, 01:21:29 root@nenyas-edge-01:~# root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.111 Speedtest by Ookla Server: Frontier - Tampa, FL (id = 27513) ISP: Frontier Communications Latency: 1.77 ms (0.06 ms jitter) Download: 74.93 Mbps (data used: 98.8 MB ) Upload: 91.88 Mbps (data used: 114.2 MB ) Packet Loss: 0.0% Result URL: https://www.speedtest.net/result/c/e9c10e20-4add-4cbd-947f-e318660916fd root@nenyas-edge-01:~/ookla-speedtest-1.1.1# root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.121 [2022-01-24 23:48:34.392] [error] Configuration - Couldn't connect to server (Network is unreachable) [2022-01-24 23:48:34.392] [error] Configuration - Cannot retrieve configuration document (0) [2022-01-24 23:48:34.392] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) [2022-01-24 23:48:34.393] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) [error] Configuration - Could not retrieve or read configuration (ConfigurationError) root@nenyas-edge-01:~/ookla-speedtest-1.1.1# root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I eth0.10 Speedtest by Ookla Server: Frontier - Tampa, FL (id = 27513) ISP: WideOpenWest Latency: 41.44 ms (1.36 ms jitter) Download: 74.30 Mbps (data used: 87.8 MB ) Upload: 99.89 Mbps (data used: 123.1 MB ) Packet Loss: 0.0% Result URL: https://www.speedtest.net/result/c/fc63e464-c5a0-4efb-b5a8-aa4d2926aea8 root@nenyas-edge-01:~/ookla-speedtest-1.1.1#