Page MenuHomePhabricator

Cannot Clamp MSS on Transient Bridge Interfaces - Turn On br_netfilter
Closed, ResolvedPublic

Description

I've been following task T314, but i believe this is a separate issue.

When using a virtual VyOS firewall, i cannot clamp MSS.
My particular setup is VyOS 1.2.0, using libvirtd 4.0, but i've tested on ESX 6.5 with the same results.

When trying to apply the following fix from T314 (though slightly modified), i get the following -

iptables -t mangle -I POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o tun0 -j TCPMSS --set-mss 1400
iptables -t mangle -I POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400
root@vyos:/home/vyos# iptables-save -t mangle -c
# Generated by iptables-save v1.4.21 on Tue Feb 12 20:33:01 2019
*mangle
:PREROUTING ACCEPT [119690:106898908]
:INPUT ACCEPT [119261:106717952]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [105490:57099434]
:POSTROUTING ACCEPT [105490:57099434]
[0:0] -A POSTROUTING -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400
[0:0] -A POSTROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400
COMMIT

However, when i tried the same command on a physical VyOS box, the counters started incrementing.

iptables-save -t mangle -c
# Generated by iptables-save v1.4.20 on Tue Feb 12 20:25:38 2019
*mangle
:PREROUTING ACCEPT [843:107149]
:INPUT ACCEPT [589:59787]
:FORWARD ACCEPT [294:52053]
:OUTPUT ACCEPT [495:273495]
:POSTROUTING ACCEPT [789:325548]
[17:1028] -A POSTROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400

I've searched around and the best i could come up with thus far is to disable tcp offloading with Ethtool, but that didn't make a difference -

ethtool -K tun0 tx off sg off tso off
ethtool -K eth0 tx off sg off tso off
ethtool -K eth1 tx off sg off tso off

I also tried using virtio, e1000, and rtl8139, plus the ethtool changes, none of which changed the results.

I'm clearly missing something, has anyone else run across this issue?

Edit:
I've modified the title to reflect the problem better.

Details

Difficulty level
Unknown (require assessment)
Version
1.2.0 EPA
Why the issue appeared?
Will be filled on close

Event Timeline

xrpixer created this task.Feb 12 2019, 8:50 PM
xrpixer created this object in space S1 VyOS Public.
c-po added a subscriber: c-po.Feb 13 2019, 4:31 AM

Your second command does kot specify any output interface whereas the first command speciefies tun0. Especially on ESXi you see almost no difference compared ro a vietual Box.

I myself run 1.2.0 in both a Physical and ESXi instance on PPPoE and use the clamping commands successfully on both nodes

In T1245#32694, @c-po wrote:

Your second command does kot specify any output interface whereas the first command speciefies tun0. Especially on ESXi you see almost no difference compared ro a vietual Box.
I myself run 1.2.0 in both a Physical and ESXi instance on PPPoE and use the clamping commands successfully on both nodes

Yes, i purposely left out an output interface on the first command to apply to all interfaces. But also ran one specifying tun0 to make sure it wasn't required.

One thing i've noticed in testing today is that traffic sourced from the VyOS box itself is getting clamped.
But passthrough traffic is not.

My use case is a bit different than a PPPoE connection. I'm using a layer2 bridge that gets piped over a VPN.

My setup is this -

LAN SEGMENT > VyOS eth1 > Br0 > tun0 ---- gre-bridge ---- tun0 > Br0 with layer3 address.

Come to think of it, there isn't any routing involved. Which may be why iptables isn't clamping my MSS.
Thing is this for sure works on a UBNT EdgeRouter. They do the same thing to clamp MSS across the board.

@c-po thanks for mentioning the PPPoE connection, really got me thinking about the word POSTROUTING!
The solution is this -

As mentioned here on StackExchange,
https://unix.stackexchange.com/questions/410112/iptables-not-filtering-bridged-traffic

The problem is that br_netfilter is a separate module in 4.x kernels. UBNT Routers are still 3.x kernels which is why they worked.
I've added the following to /config/scripts/vyos-postconfig-bootup.script

modprobe br_netfilter
iptables -t mangle -I POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400

This applies a system wide clamp on MSS, even if the traffic is passing over a bridge to a tun interface without a layer3 address being involved.
This has thus far fixed the problem i was seeing. Now my hit count is incrementing continually with the traffic, and TLS handshakes are no longer failing.

[425:25316] -A POSTROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400

Would be nice if we merged this into VyOS, or a way in "set firewall" to turn it on.
Seems to be the only way to do a layer2 bridge over an MTU limited medium, such as an IPSec VPN.

Does anyone have any thoughts on this going forward?

c-po added a comment.Feb 13 2019, 7:01 AM

This will be part of a bigger workpackage when the whole firewalling is rewritten. There is yet no ETA.

I also use the same workaround in /config/scripts/vyos-postconfig-bootup.script.

I do not see this as firewall configuration as EsgeOS does it. It should reside under policy routing or as a per interface command like ip mss-clamp in Cisco devices, as we already have interface ethernet mtu command

Sorry i'm not sure we're on the same page.

I'm good with the workaround in /config/scripts/vyos-postconfig-bootup.script.
Adding -
iptables -t mangle -I POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400

And EdgeOS configures a line that looks very similar when you do -
set firewall options mss-clamp mss 1400

I see there's a greater conversation there and am fine with using the workaround.

The question i'm proposing now is should we turn on br_netfilter.
In the older kernels, br_netfilter wasn't a thing and iptables would affect transient traffic on bridge interfaces.
In newer 4.x kernels, br_netfilter was created and must be loaded to affect transient traffic.

The use case really being layer2 bridges where you wouldn't route any traffic, you're just bridging between two areas.

xrpixer renamed this task from Cannot Clamp MSS on Virtual Machine to Cannot Clamp MSS on Transient Bridge Interfaces - Turn On br_netfilter.Feb 13 2019, 3:28 PM
xrpixer updated the task description. (Show Details)
c-po added a comment.Feb 13 2019, 4:39 PM

br_netfilter should already be compuled as a module. Can you sudo modprobe br_netfilter? To see if it fits your purpose? If so we can autoload it on system bootup

Yes! That's what i need.
In my script above i had to put modprobe br_netfilter so it loads on system boot.

modprobe br_netfilter
iptables -t mangle -I POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400

If we could have br_netfilter loaded on boot in the image that would be great and would fix this problem.

c-po added a comment.Feb 13 2019, 10:16 PM

Latest rolling should autoload the module

Thank you!
I'll test the next rolling asap and report back.

pasik added a subscriber: pasik.Feb 14 2019, 6:49 PM

The last rolling worked great. Saw the module was loaded on boot and MSS was clamped correctly.
Thanks!

xrpixer closed this task as Resolved.Feb 14 2019, 7:48 PM
xrpixer claimed this task.
syncer reopened this task as In progress.Feb 15 2019, 10:06 AM
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.
syncer added a subscriber: syncer.Apr 17 2019, 9:05 PM

@c-po can you backport this to crux, please

syncer triaged this task as Normal priority.Apr 17 2019, 9:05 PM
c-po claimed this task.Apr 20 2019, 2:52 PM
c-po added a comment.Apr 20 2019, 2:54 PM

Already in 1.2.1

c-po closed this task as Resolved.Apr 20 2019, 2:54 PM
c-po moved this task from Needs Triage to Finished on the VyOS 1.2 Crux (VyOS 1.2.1) board.