Page MenuHomeVyOS Platform

Tcp-mss option in policy calls kernel-panic
Open, HighPublicBUG

Description

To reproduce this bug.

set policy route MSS-CLAMP rule 10 protocol tcp
set policy route MSS-CLAMP rule 10 set tcp-mss pmtu
set policy route MSS-CLAMP rule 10 tcp flags SYN
set interfaces ethernet eth0 policy route MSS-CLAMP
commit

Initiate any TCP session.

vyos@r1-roll:~$ sudo curl https://yahoo.com

Original post https://forum.vyos.io/t/kernel-panic-with-set-mss-pmtu/5850

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.3-rolling-202009090118
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Event Timeline

Viacheslav renamed this task from Tcp-mss option in policy call kernel-panic to Tcp-mss option in policy calls kernel-panic.Wed, Sep 9, 5:21 PM
Viacheslav triaged this task as High priority.
Dmitry added a subscriber: Dmitry.Wed, Sep 9, 6:59 PM

How about migrating this iptables rule from mangle to filter?

More logs

vyos@r2-roll# sudo curl https://yahoo.com
[ 2045.620295] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 2045.628755] #PF: supervisor read access in kernel mode
[ 2045.630777] #PF: error_code(0x0000) - not-present page
[ 2045.632483] PGD 0 P4D 0 
[ 2045.633374] Oops: 0000 [#1] SMP PTI
[ 2045.634465] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.5-amd64-vyos #1
[ 2045.635948] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1 04/01/2014
[ 2045.637941] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[ 2045.639334] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 89 f7 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 89 4c 24 08 48 83 e7 fe <48> 8b 47 08 48 8b 40 20 e8 13 d7 d5 f7 8b 74 24 10 4c 8b 4c 24 08
[ 2045.644694] RSP: 0018:ffffb762000038c8 EFLAGS: 00010246
[ 2045.645850] RAX: 00000000000005dc RBX: ffff8fd3c3756c62 RCX: 0000000000000002
[ 2045.647383] RDX: 0000000000000000 RSI: 00000000198f064a RDI: 0000000000000000
[ 2045.649438] RBP: ffffb76200003978 R08: ffffb762000038f0 R09: 0000000000000014
[ 2045.650995] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[ 2045.652575] R13: ffff8fd3d11aa500 R14: 0000000000000028 R15: ffffb76200003ba8
[ 2045.654101] FS:  0000000000000000(0000) GS:ffff8fd3d8c00000(0000) knlGS:0000000000000000
[ 2045.655683] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2045.657013] CR2: 0000000000000008 CR3: 000000000602e006 CR4: 0000000000160ef0
[ 2045.658538] Call Trace:
[ 2045.659148]  <IRQ>
[ 2045.659622]  tcpmss_tg4+0x2c/0xa0 [xt_TCPMSS]
[ 2045.660541]  nft_target_eval_xt+0x30/0x50 [nft_compat]
[ 2045.661592]  nft_do_chain+0x149/0x4c0 [nf_tables]
[ 2045.662561]  ? pollwake+0x6f/0x90
[ 2045.663377]  ? wake_up_q+0xa0/0xa0
[ 2045.664257]  ? sock_def_readable+0x32/0x60
[ 2045.665276]  ? __udp_enqueue_schedule_skb+0x133/0x260
[ 2045.666320]  ? udp_queue_rcv_one_skb+0x2be/0x460
[ 2045.667207]  ? udp_unicast_rcv_skb.isra.66+0x6f/0x80
[ 2045.668455]  ? __udp4_lib_rcv+0x553/0xb70
[ 2045.669392]  nft_do_chain_ipv4+0x61/0x80 [nf_tables]
[ 2045.670528]  nf_hook_slow+0x3f/0xc0
[ 2045.671364]  nf_hook_slow_list+0x89/0x130
[ 2045.673401]  ip_sublist_rcv+0x1fb/0x210
[ 2045.674261]  ? ip_rcv_finish_core.isra.22+0x400/0x400
[ 2045.675423]  ip_list_rcv+0x132/0x156
[ 2045.676398]  __netif_receive_skb_list_core+0x296/0x2c0
[ 2045.677722]  netif_receive_skb_list_internal+0x1a1/0x2c0
[ 2045.679238]  ? check_preempt_curr+0x75/0x90
[ 2045.680142]  gro_normal_list.part.162+0x14/0x30
[ 2045.680957]  napi_complete_done+0x62/0x170
[ 2045.681702]  virtqueue_napi_complete+0x25/0x60 [virtio_net]
[ 2045.682697]  virtnet_poll+0x2e0/0x330 [virtio_net]
[ 2045.683513]  net_rx_action+0xf6/0x2e0
[ 2045.684220]  __do_softirq+0xd2/0x227
[ 2045.685054]  asm_call_on_stack+0x12/0x20
[ 2045.685985]  </IRQ>
[ 2045.686596]  do_softirq_own_stack+0x34/0x40
[ 2045.687725]  irq_exit_rcu+0x98/0xa0
[ 2045.688530]  common_interrupt+0x73/0x140
[ 2045.689440]  asm_common_interrupt+0x1e/0x40
[ 2045.690348] RIP: 0010:native_safe_halt+0xe/0x10
[ 2045.691283] Code: 48 8b 04 25 c0 7b 01 00 3e 80 48 02 20 48 8b 00 a8 08 75 c4 eb 80 cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d 96 b5 55 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 86 b5 55 00 f4 c3 cc cc e8 bb 6a a2
[ 2045.694894] RSP: 0018:ffffffffb8c03eb8 EFLAGS: 00000246
[ 2045.696154] RAX: ffffffffb82acce0 RBX: 0000000000000000 RCX: ffff8fd3d8c232c0
[ 2045.697897] RDX: 00000000000674ca RSI: 0000000000000087 RDI: 0000000000000000
[ 2045.699533] RBP: ffffffffb8c90dc0 R08: 0000032a500d3a2d R09: 0000000000000000
[ 2045.701165] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 2045.703893] R13: 0000000000000000 R14: ffffffffffffffff R15: ffffffffb8c134c0
[ 2045.705614]  ? __sched_text_end+0x6/0x6
[ 2045.706625]  default_idle+0x5/0x10
[ 2045.707539]  do_idle+0x212/0x2d0
[ 2045.708374]  cpu_startup_entry+0x14/0x20
[ 2045.709342]  start_kernel+0x515/0x534
[ 2045.710369]  secondary_startup_64+0xa4/0xb0
[ 2045.711445] Modules linked in: ip_set xt_TCPMSS xt_comment fuse nft_chain_nat xt_CT xt_tcpudp nft_compat nfnetlink_cthelper nft_counter nf_tables nfnetlink nf_nat_pptp nf_conntrack_pptp nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper iTCO_wdt pcspkr evdev iTCO_vendor_support button virtio_balloon virtio_console mpls_iptunnel mpls_router ip_tunnel mpls_gso br_netfilter bridge stp llc virtio_rng rng_core ip_tables x_tables autofs4 usb_storage ohci_hcd uhci_hcd ehci_hcd sd_mod t10_pi squashfs zstd_decompress loop overlay ext4 crc32c_generic crc16 mbcache jbd2 nls_ascii hid_generic usbhid hid sr_mod cdrom ahci libahci virtio_net net_failover failover virtio_blk libata xhci_pci crc32c_intel i2c_i801 i2c_smbus lpc_ich scsi_mod xhci_hcd virtio_pci virtio_ring virtio
[ 2045.729452] CR2: 0000000000000008
[ 2045.730318] ---[ end trace 54d1da27fbab7803 ]---
[ 2045.731486] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[ 2045.733004] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 89 f7 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 89 4c 24 08 48 83 e7 fe <48> 8b 47 08 48 8b 40 20 e8 13 d7 d5 f7 8b 74 24 10 4c 8b 4c 24 08
[ 2045.738357] RSP: 0018:ffffb762000038c8 EFLAGS: 00010246
[ 2045.739661] RAX: 00000000000005dc RBX: ffff8fd3c3756c62 RCX: 0000000000000002
[ 2045.741324] RDX: 0000000000000000 RSI: 00000000198f064a RDI: 0000000000000000
[ 2045.743053] RBP: ffffb76200003978 R08: ffffb762000038f0 R09: 0000000000000014
[ 2045.744727] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[ 2045.746401] R13: ffff8fd3d11aa500 R14: 0000000000000028 R15: ffffb76200003ba8
[ 2045.748186] FS:  0000000000000000(0000) GS:ffff8fd3d8c00000(0000) knlGS:0000000000000000
[ 2045.750434] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2045.751923] CR2: 0000000000000008 CR3: 000000000602e006 CR4: 0000000000160ef0
[ 2045.753728] Kernel panic - not syncing: Fatal exception in interrupt
[ 2045.755228] Kernel Offset: 0x36c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 2045.757667] Rebooting in 60 seconds..

LTS version

vyos@r-1.2.5# commit
[ interfaces ethernet eth0 policy route MSS-CLAMP ]
iptables: Invalid argument. Run `dmesg' for more information.

[[interfaces ethernet eth0]] failed
Commit failed
[edit]
vyos@r-1.2.5# 


vyos@r-1.2.5# sudo dmesg | tail -n 5
[    9.311242] Bridge firewalling registered
[    9.328112] fuse init (API version 7.27)
[   10.245771] Process accounting resumed
[   49.562499] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks
[  154.168320] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks
[edit]
vyos@r-1.2.5#

So it's some wrong logic after rewriting firewall?

c-po added a subscriber: c-po.Thu, Sep 10, 8:56 AM

Firewall is yet not migrated, but MSS clamping was moved (T314), please retest on rolling after new iso with T2870 implemented, if kernel still crashes.

pasik added a subscriber: pasik.Thu, Sep 10, 11:07 AM

@c-po , the same behavior even with kernel 5.8.8

vyos@R1:~$ uname -a
Linux R1 5.8.8-amd64-vyos #1 SMP Thu Sep 10 08:58:42 UTC 2020 x86_64 GNU/Linux

Any reason why we need to use MSS Clamp in mangle chain?

Still present in the latest rolling

VyOS 1.3-rolling-202009140541
Linux r1-roll 4.19.145-amd64-vyos #1 SMP Mon Sep 14 05:08:01 UTC 2020 x86_64 GNU/Linux
This comment was removed by Viacheslav.