Page MenuHomeVyOS Platform

Tcp-mss option in policy calls kernel-panic
Closed, ResolvedPublicBUG

Description

To reproduce this bug.

set policy route MSS-CLAMP rule 10 protocol tcp
set policy route MSS-CLAMP rule 10 set tcp-mss pmtu
set policy route MSS-CLAMP rule 10 tcp flags SYN
set interfaces ethernet eth0 policy route MSS-CLAMP
commit

Initiate any TCP session.

vyos@r1-roll:~$ sudo curl https://yahoo.com

Original post https://forum.vyos.io/t/kernel-panic-with-set-mss-pmtu/5850

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.3-rolling-202009090118
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Viacheslav renamed this task from Tcp-mss option in policy call kernel-panic to Tcp-mss option in policy calls kernel-panic.Sep 9 2020, 5:21 PM
Viacheslav triaged this task as High priority.

How about migrating this iptables rule from mangle to filter?

More logs

vyos@r2-roll# sudo curl https://yahoo.com
[ 2045.620295] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 2045.628755] #PF: supervisor read access in kernel mode
[ 2045.630777] #PF: error_code(0x0000) - not-present page
[ 2045.632483] PGD 0 P4D 0 
[ 2045.633374] Oops: 0000 [#1] SMP PTI
[ 2045.634465] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.5-amd64-vyos #1
[ 2045.635948] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1 04/01/2014
[ 2045.637941] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[ 2045.639334] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 89 f7 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 89 4c 24 08 48 83 e7 fe <48> 8b 47 08 48 8b 40 20 e8 13 d7 d5 f7 8b 74 24 10 4c 8b 4c 24 08
[ 2045.644694] RSP: 0018:ffffb762000038c8 EFLAGS: 00010246
[ 2045.645850] RAX: 00000000000005dc RBX: ffff8fd3c3756c62 RCX: 0000000000000002
[ 2045.647383] RDX: 0000000000000000 RSI: 00000000198f064a RDI: 0000000000000000
[ 2045.649438] RBP: ffffb76200003978 R08: ffffb762000038f0 R09: 0000000000000014
[ 2045.650995] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[ 2045.652575] R13: ffff8fd3d11aa500 R14: 0000000000000028 R15: ffffb76200003ba8
[ 2045.654101] FS:  0000000000000000(0000) GS:ffff8fd3d8c00000(0000) knlGS:0000000000000000
[ 2045.655683] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2045.657013] CR2: 0000000000000008 CR3: 000000000602e006 CR4: 0000000000160ef0
[ 2045.658538] Call Trace:
[ 2045.659148]  <IRQ>
[ 2045.659622]  tcpmss_tg4+0x2c/0xa0 [xt_TCPMSS]
[ 2045.660541]  nft_target_eval_xt+0x30/0x50 [nft_compat]
[ 2045.661592]  nft_do_chain+0x149/0x4c0 [nf_tables]
[ 2045.662561]  ? pollwake+0x6f/0x90
[ 2045.663377]  ? wake_up_q+0xa0/0xa0
[ 2045.664257]  ? sock_def_readable+0x32/0x60
[ 2045.665276]  ? __udp_enqueue_schedule_skb+0x133/0x260
[ 2045.666320]  ? udp_queue_rcv_one_skb+0x2be/0x460
[ 2045.667207]  ? udp_unicast_rcv_skb.isra.66+0x6f/0x80
[ 2045.668455]  ? __udp4_lib_rcv+0x553/0xb70
[ 2045.669392]  nft_do_chain_ipv4+0x61/0x80 [nf_tables]
[ 2045.670528]  nf_hook_slow+0x3f/0xc0
[ 2045.671364]  nf_hook_slow_list+0x89/0x130
[ 2045.673401]  ip_sublist_rcv+0x1fb/0x210
[ 2045.674261]  ? ip_rcv_finish_core.isra.22+0x400/0x400
[ 2045.675423]  ip_list_rcv+0x132/0x156
[ 2045.676398]  __netif_receive_skb_list_core+0x296/0x2c0
[ 2045.677722]  netif_receive_skb_list_internal+0x1a1/0x2c0
[ 2045.679238]  ? check_preempt_curr+0x75/0x90
[ 2045.680142]  gro_normal_list.part.162+0x14/0x30
[ 2045.680957]  napi_complete_done+0x62/0x170
[ 2045.681702]  virtqueue_napi_complete+0x25/0x60 [virtio_net]
[ 2045.682697]  virtnet_poll+0x2e0/0x330 [virtio_net]
[ 2045.683513]  net_rx_action+0xf6/0x2e0
[ 2045.684220]  __do_softirq+0xd2/0x227
[ 2045.685054]  asm_call_on_stack+0x12/0x20
[ 2045.685985]  </IRQ>
[ 2045.686596]  do_softirq_own_stack+0x34/0x40
[ 2045.687725]  irq_exit_rcu+0x98/0xa0
[ 2045.688530]  common_interrupt+0x73/0x140
[ 2045.689440]  asm_common_interrupt+0x1e/0x40
[ 2045.690348] RIP: 0010:native_safe_halt+0xe/0x10
[ 2045.691283] Code: 48 8b 04 25 c0 7b 01 00 3e 80 48 02 20 48 8b 00 a8 08 75 c4 eb 80 cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d 96 b5 55 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 86 b5 55 00 f4 c3 cc cc e8 bb 6a a2
[ 2045.694894] RSP: 0018:ffffffffb8c03eb8 EFLAGS: 00000246
[ 2045.696154] RAX: ffffffffb82acce0 RBX: 0000000000000000 RCX: ffff8fd3d8c232c0
[ 2045.697897] RDX: 00000000000674ca RSI: 0000000000000087 RDI: 0000000000000000
[ 2045.699533] RBP: ffffffffb8c90dc0 R08: 0000032a500d3a2d R09: 0000000000000000
[ 2045.701165] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 2045.703893] R13: 0000000000000000 R14: ffffffffffffffff R15: ffffffffb8c134c0
[ 2045.705614]  ? __sched_text_end+0x6/0x6
[ 2045.706625]  default_idle+0x5/0x10
[ 2045.707539]  do_idle+0x212/0x2d0
[ 2045.708374]  cpu_startup_entry+0x14/0x20
[ 2045.709342]  start_kernel+0x515/0x534
[ 2045.710369]  secondary_startup_64+0xa4/0xb0
[ 2045.711445] Modules linked in: ip_set xt_TCPMSS xt_comment fuse nft_chain_nat xt_CT xt_tcpudp nft_compat nfnetlink_cthelper nft_counter nf_tables nfnetlink nf_nat_pptp nf_conntrack_pptp nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper iTCO_wdt pcspkr evdev iTCO_vendor_support button virtio_balloon virtio_console mpls_iptunnel mpls_router ip_tunnel mpls_gso br_netfilter bridge stp llc virtio_rng rng_core ip_tables x_tables autofs4 usb_storage ohci_hcd uhci_hcd ehci_hcd sd_mod t10_pi squashfs zstd_decompress loop overlay ext4 crc32c_generic crc16 mbcache jbd2 nls_ascii hid_generic usbhid hid sr_mod cdrom ahci libahci virtio_net net_failover failover virtio_blk libata xhci_pci crc32c_intel i2c_i801 i2c_smbus lpc_ich scsi_mod xhci_hcd virtio_pci virtio_ring virtio
[ 2045.729452] CR2: 0000000000000008
[ 2045.730318] ---[ end trace 54d1da27fbab7803 ]---
[ 2045.731486] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[ 2045.733004] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 89 f7 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 89 4c 24 08 48 83 e7 fe <48> 8b 47 08 48 8b 40 20 e8 13 d7 d5 f7 8b 74 24 10 4c 8b 4c 24 08
[ 2045.738357] RSP: 0018:ffffb762000038c8 EFLAGS: 00010246
[ 2045.739661] RAX: 00000000000005dc RBX: ffff8fd3c3756c62 RCX: 0000000000000002
[ 2045.741324] RDX: 0000000000000000 RSI: 00000000198f064a RDI: 0000000000000000
[ 2045.743053] RBP: ffffb76200003978 R08: ffffb762000038f0 R09: 0000000000000014
[ 2045.744727] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[ 2045.746401] R13: ffff8fd3d11aa500 R14: 0000000000000028 R15: ffffb76200003ba8
[ 2045.748186] FS:  0000000000000000(0000) GS:ffff8fd3d8c00000(0000) knlGS:0000000000000000
[ 2045.750434] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2045.751923] CR2: 0000000000000008 CR3: 000000000602e006 CR4: 0000000000160ef0
[ 2045.753728] Kernel panic - not syncing: Fatal exception in interrupt
[ 2045.755228] Kernel Offset: 0x36c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 2045.757667] Rebooting in 60 seconds..

LTS version

vyos@r-1.2.5# commit
[ interfaces ethernet eth0 policy route MSS-CLAMP ]
iptables: Invalid argument. Run `dmesg' for more information.

[[interfaces ethernet eth0]] failed
Commit failed
[edit]
vyos@r-1.2.5# 


vyos@r-1.2.5# sudo dmesg | tail -n 5
[    9.311242] Bridge firewalling registered
[    9.328112] fuse init (API version 7.27)
[   10.245771] Process accounting resumed
[   49.562499] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks
[  154.168320] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks
[edit]
vyos@r-1.2.5#

So it's some wrong logic after rewriting firewall?

Firewall is yet not migrated, but MSS clamping was moved (T314), please retest on rolling after new iso with T2870 implemented, if kernel still crashes.

@c-po , the same behavior even with kernel 5.8.8

vyos@R1:~$ uname -a
Linux R1 5.8.8-amd64-vyos #1 SMP Thu Sep 10 08:58:42 UTC 2020 x86_64 GNU/Linux

Any reason why we need to use MSS Clamp in mangle chain?

Still present in the latest rolling

VyOS 1.3-rolling-202009140541
Linux r1-roll 4.19.145-amd64-vyos #1 SMP Mon Sep 14 05:08:01 UTC 2020 x86_64 GNU/Linux
This comment was removed by Viacheslav.
This comment was removed by Viacheslav.
Viacheslav claimed this task.
Viacheslav removed Viacheslav as the assignee of this task.

It seems a wrong logic.
We want that option to have an effect on "local" and "forward" directions, so we use table mangle and "PREROUTING" and VYATTA_FW_IN_HOOK hook
Generated rules

*mangle
:VYATTA_FW_IN_HOOK
-A PREROUTING -j VYATTA_FW_IN_HOOK

-A VYATTA_FW_IN_HOOK -i eth0 -j MSS-CLAMP
-A MSS-CLAMP -p tcp -m tcp --tcp-flags SYN SYN -m comment --comment MSS-CLAMP-10 -j TCPMSS --clamp-mss-to-pmtu
-A MSS-CLAMP -m comment --comment "MSS-CLAMP-10000 default-action accept" -j RETURN

For "prerouting" VYATTA_FW_IN_HOOK we can't use the option -j TCPMSS --clamp-mss-to-pmtu, we can only handle -j TCPMSS --set-mss xxxx

And -j TCPMSS --clamp-mss-to-pmtu should be used only in FORWARD, OUTPUT and POSTROUTING hooks
ref. https://github.com/torvalds/linux/blob/master/net/netfilter/xt_TCPMSS.c#L263-L284

What we see, when we are trying to use that rule without any hook in mangle => prerouting

root@r5:/home/vyos# sudo iptables -t mangle -A PREROUTING  -i eth0 -p tcp -m tcp --tcp-flags SYN SYN -j TCPMSS --clamp-mss-to-pmtu
iptables v1.8.2 (nf_tables):  RULE_APPEND failed (Invalid argument): rule in chain PREROUTING

vyos@r5# sudo dmesg | tail -n 1
[48323.007403] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks

It makes sense to use this option only for FORWARD or OUTPUT traffic. So I think we should get rid of that option, and define it in some else CLI with different logic.

set policy route MSS-CLAMP rule 10 set tcp-mss pmtu
Viacheslav changed the task status from Open to Needs testing.Nov 27 2020, 1:56 PM
Viacheslav claimed this task.
erkin set Issue type to Bug (incorrect behavior).Aug 29 2021, 1:09 PM
erkin removed a subscriber: Active contributors.