Page MenuHomeVyOS Platform
Create Task
Maniphest T4323

ospf6d crashes on latest vyos nightly
Closed, ResolvedPublic
Actions

Description

I upgraded one of my routers to vyos 1.4-rolling-202203261656

ipv6 routes don't get learned via ospfv3 and checking the ospf6d journal reveals a segfault crash:

Mar 27 01:51:59 305-1700-gw ospf6d[9012]: [YWKRY-MSCC3] ospf6_spf_calculation: No router LSA for area 0.0.0.0
Mar 27 01:51:59 305-1700-gw ospf6d[9012]: [V8P0C-HB5Z2] ASBR[default:Status:1]: Update
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: Received signal 11 at 1648360319 (si_addr 0x0, PC 0x55e46c36e7e4); aborting...
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6d) [0x7f8c6f29be>
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f8c6f29c0d5]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xc8601) [0x7f8c6f2c8601]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f8c6f1cc140]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/frr/ospf6d(+0x4a7e4) [0x55e46c36e7e4]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xbb287) [0x7f8c6f2bb287]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_map_apply_ext+0x4e0) [0x7f8c6f2bd030]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/frr/ospf6d(ospf6_asbr_redistribute_add+0x13a) [0x55e46c37425a]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/frr/ospf6d(+0x80376) [0x55e46c3a4376]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xec145) [0x7f8c6f2ec145]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f8c6f2da23d]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f8c6f294458]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/frr/ospf6d(main+0xc6) [0x55e46c36d5b6]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f8c6f016d0a]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: /usr/lib/frr/ospf6d(_start+0x2a) [0x55e46c36dafa]
Mar 27 01:51:59 305-1700-gw OSPF6[9012]: in thread zclient_read scheduled from lib/zclient.c:4074 zclient_event()

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

starcraft66 triaged this task as High priority.Mar 27 2022, 5:56 AM
starcraft66 created this task.
starcraft66 created this object in space S1 VyOS Public.
c-po claimed this task.Mar 27 2022, 6:55 AM
c-po changed the task status from Open to Confirmed.Mar 27 2022, 6:59 AM
c-po added a comment.Mar 27 2022, 7:12 AM

Feels like the bug was introduced in https://github.com/FRRouting/frr/commit/36295a387c241d3dba7aaa305a1860a6a07ab0cf

Pinning FRR version 8.2.2 explicitly and not the branch with some bugfixes. Please test again with any rolling ISO after vyos-1.4-rolling-202203270217-amd64.iso

c-po added a comment.Mar 27 2022, 8:57 AM

Re-run with the latest FRR code-base and it also seems to be fixed - please try the latest VyOS version ;)

pasik added a subscriber: pasik.Mar 27 2022, 10:50 AM
starcraft66 added a comment.Mar 27 2022, 5:12 PM

Still seems to be broken on 1.4-rolling-202203270930

vyos@305-1700-gw:~$ journalctl -b /usr/lib/frr/ospf6d                                                                         
-- Journal begins at Sun 2022-03-27 13:06:56 EDT, ends at Sun 2022-03-27 13:10:14 EDT. --                                     
Mar 27 13:07:08 debian ospf6d[1125]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00                                       
Mar 27 13:07:31 305-1700-gw ospf6d[1125]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00                                  
Mar 27 13:07:31 305-1700-gw ospf6d[1125]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Mar 27 13:07:52 305-1700-gw ospf6d[1125]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Mar 27 13:07:52 305-1700-gw ospf6d[1125]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Mar 27 13:07:57 305-1700-gw ospf6d[1125]: [YWKRY-MSCC3] ospf6_spf_calculation: No router LSA for area 0.0.0.0
Mar 27 13:07:57 305-1700-gw ospf6d[1125]: [V8P0C-HB5Z2] ASBR[default:Status:1]: Update
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: Received signal 11 at 1648400877 (si_addr 0x0, PC 0x5625620d67e4); aborting...
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6d) [0x7f49b3693e>
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f49b36940d5]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xc8601) [0x7f49b36c0601]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f49b35c4140]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/frr/ospf6d(+0x4a7e4) [0x5625620d67e4]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xbb287) [0x7f49b36b3287]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_map_apply_ext+0x4e0) [0x7f49b36b5030]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/frr/ospf6d(ospf6_asbr_redistribute_add+0x13a) [0x5625620dc25a]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/frr/ospf6d(+0x80376) [0x56256210c376]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xec145) [0x7f49b36e4145]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f49b36d223d]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f49b368c458]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/frr/ospf6d(main+0xc6) [0x5625620d55b6]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f49b340ed0a]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: /usr/lib/frr/ospf6d(_start+0x2a) [0x5625620d5afa]
Mar 27 13:07:57 305-1700-gw OSPF6[1125]: in thread zclient_read scheduled from lib/zclient.c:4074 zclient_event()
Mar 27 13:08:02 305-1700-gw ospf6d[4433]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
vyos@305-1700-gw:~$ show system image
The system currently has the following image(s) installed:

   1: 1.4-rolling-202203270930 (default boot) (running image)
   2: 1.4-rolling-202203261656
c-po added a comment.Mar 27 2022, 5:41 PM

Can not get it crashed here. Please share your full config.

show configuration commands

so I can reproduce it

starcraft66 added a comment.Mar 27 2022, 6:44 PM

I'm not comfortable posting my config here for ever but I've uploaded it to ubuntu pastebin and set it to expire in a week: https://paste.ubuntu.com/p/BW5KxyjPQr/

c-po added a comment.Mar 28 2022, 6:52 PM

FRR bugreport filed https://github.com/FRRouting/frr/issues/10911

c-po added a comment.Mar 30 2022, 4:54 PM

Bugfix just got merged into FRR codebase - next VyOS rolling ISO image will have the fix.

c-po changed the task status from Confirmed to Needs testing.Mar 31 2022, 7:28 PM
starcraft66 added a comment.Mar 31 2022, 8:17 PM

I can confirm that the issue is fixed on today's rolling release!

starcraft66 closed this task as Resolved.Mar 31 2022, 8:17 PM
Viacheslav added a project: VyOS 1.4 Sagitta.Apr 1 2022, 11:14 AM
Viacheslav moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.