This may be more of a linux bug or require some tuning to resolve. I have multiple ISPs so when troubleshooting issues on specific paths, I often use curl, mtr, ping, telnet, and traceroute and specify a source IP. I find that they only work for the first two WAN interfaces. The 3rd interface fails. It seems to be tied to the metric of the default route associated with the interface. I know it's not an issue with that specific interface/default route because if I change the metric on the 3rd default route so that it is one of the first two, it and the other one in the first two work, but now the interface that has the 3rd default route fails.
Below are examples.
In example set 1, you can see that the default route for bond1.111 is 1st and the speedtest for that interface succeeds, the default route for bond1.121 is 2nd and the speedtest for that interface succeeds, and the default route eth0.10 is 3rd and the speedtest for that interface fails with errors.
In example set 2, you can see that the default route for bond1.111 is 2nd and the speedtest for that interface succeeds, the default route for bond1.121 is 3rd and the speedtest for that interface fails with errors, and the default route eth0.10 is 1st and the speedtest for that interface succeeds.
I am running vyos-1.3.0-rc6-amd64. I was going to test this on vyos-1.3.0-amd64, but per T4206 that I posted earlier, I can use that version.
Example Set 1:
root@nenyas-edge-01:~/ookla-speedtest-1.1.1# show ip route | head -9
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
S 0.0.0.0/0 [210/0] via x.x.x1, bond1.111, weight 1, 01:15:17
S>* 0.0.0.0/0 [10/0] via x.x.x.105, bond1.121, weight 1, 01:15:39
S 0.0.0.0/0 [20/0] via 10.x.x.1, eth0.10, weight 1, 01:15:39
root@nenyas-edge-01:~/ookla-speedtest-1.1.1#
root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.111
Speedtest by Ookla
Server: Frontier - Tampa, FL (id = 27513)
ISP: Frontier Communications
Latency: 1.68 ms (0.09 ms jitter)
Download: 77.20 Mbps (data used: 90.9 MB )
Upload: 91.80 Mbps (data used: 114.2 MB )
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/d7df8668-a146-4cb0-9d98-48456e322328
root@nenyas-edge-01:~/ookla-speedtest-1.1.1#
root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.121
Speedtest by Ookla
Server: Velocity Online - Tallahassee, FL (id = 884)
ISP: WideOpenWest
Latency: 28.09 ms (1.90 ms jitter)
Download: 537.66 Mbps (data used: 943.2 MB )
Upload: 13.56 Mbps (data used: 22.6 MB )
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/cd3be733-3f02-40e7-bb92-007ab26e0ad4
root@nenyas-edge-01:~/ookla-speedtest-1.1.1#
Example Set 2:
root@nenyas-edge-01:~# show ip route | head -9
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
S>* 0.0.0.0/0 [5/0] via 10.x.x.1, eth0.10, weight 1, 00:00:26
S 0.0.0.0/0 [210/0] via x.x.x.1, bond1.111, weight 1, 01:21:07
S 0.0.0.0/0 [10/0] via x.x.x.105, bond1.121, weight 1, 01:21:29
root@nenyas-edge-01:~#
root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.111
Speedtest by Ookla
Server: Frontier - Tampa, FL (id = 27513)
ISP: Frontier Communications
Latency: 1.77 ms (0.06 ms jitter)
Download: 74.93 Mbps (data used: 98.8 MB )
Upload: 91.88 Mbps (data used: 114.2 MB )
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/e9c10e20-4add-4cbd-947f-e318660916fd
root@nenyas-edge-01:~/ookla-speedtest-1.1.1#
root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I bond1.121
[2022-01-24 23:48:34.392] [error] Configuration - Couldn't connect to server (Network is unreachable)
[2022-01-24 23:48:34.392] [error] Configuration - Cannot retrieve configuration document (0)
[2022-01-24 23:48:34.392] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[2022-01-24 23:48:34.393] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[error] Configuration - Could not retrieve or read configuration (ConfigurationError)
root@nenyas-edge-01:~/ookla-speedtest-1.1.1#
root@nenyas-edge-01:~/ookla-speedtest-1.1.1# ./speedtest -I eth0.10
Speedtest by Ookla
Server: Frontier - Tampa, FL (id = 27513)
ISP: WideOpenWest
Latency: 41.44 ms (1.36 ms jitter)
Download: 74.30 Mbps (data used: 87.8 MB )
Upload: 99.89 Mbps (data used: 123.1 MB )
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/fc63e464-c5a0-4efb-b5a8-aa4d2926aea8
root@nenyas-edge-01:~/ookla-speedtest-1.1.1#