IP routes loss after AgentX master agent failed to respond to ping
Resolved (N/A)PublicBUG
Actions

Assigned To

None

Authored By

	jalazaro
	Mar 24 2022, 12:32 PM

Description

During a network switches upgrade, three VyOS connected to them lost some IP routes. Both of them, BGP process stopped working, an the other lost its IP static routes. All of them are running version VyOS 1.2.6-S1. The normal situation recovered after rebooting those three VyOS.

Can anyone help me? thanks.

I got this log from VyOS stopped BGP process....

Mar 22 23:16:02 de-rhr-ba4ngwfi01-01 Keepalived_vrrp[5366]: AgentX master agent failed to respond to ping. Attempting to re-register.
Mar 22 23:16:08 de-rhr-ba4ngwfi01-01 lldpd[5562]: AgentX master agent failed to respond to ping. Attempting to re-register.
Mar 22 23:18:02 de-rhr-ba4ngwfi01-01 staticd[1201]: Terminating on signal
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.068687] INFO: task jbd2/sda1-8:245 blocked for more than 120 seconds.
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.068966] Not tainted 4.19.142-amd64-vyos #1
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069078] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069245] Call Trace:
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069250] 0xffffffff95c098c0
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069257] ? 0xffffffff95c0a530
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069257] 0xffffffff95c09d1d
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069258] 0xffffffff95c0a06d
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069259] 0xffffffff95c0a538
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069260] 0xffffffff95c0a0ee
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069260] 0xffffffff95c0a1a6
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069261] ? 0xffffffff956a9630
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069264] 0xffffffffc0376c8a
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069265] ? 0xffffffff95e001a5
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069267] ? 0xffffffffc037a085
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069268] 0xffffffffc037a085
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069269] ? 0xffffffff956a8de0
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069269] ? 0xffffffffc0379fb0
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069270] 0xffffffff95687005
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069273] ? 0xffffffff95686f00
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069273] 0xffffffff95e00215
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069284] INFO: task snmpd:5553 blocked for more than 120 seconds.
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069419] Not tainted 4.19.142-amd64-vyos #1
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069528] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069693] Call Trace:
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069694] 0xffffffff95c098c0
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069695] ? 0xffffffff95c0a530
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069695] 0xffffffff95c09d1d
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069696] 0xffffffff95c0a06d
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069696] 0xffffffff95c0a538
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069697] 0xffffffff95c0a0ee
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069697] 0xffffffff95c0a1a6
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069698] ? 0xffffffff956a9630
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069698] 0xffffffffc03749d6
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069700] 0xffffffffc0374b97
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069700] 0xffffffffc03880f7
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069703] 0xffffffffc03b773d
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069704] 0xffffffffc03b8f72
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069705] 0xffffffff957dbeeb
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069710] 0xffffffffc03f7343
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069711] 0xffffffff957dbeeb
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069712] 0xffffffff957de531
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069714] 0xffffffff95603518
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069717] 0xffffffff95e00088
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069718] RIP: 0033:0x00007f331e6f02f7
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069724] Code: Bad RIP value.
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069725] RSP: 002b:00007ffc5fd15938 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069726] RAX: ffffffffffffffda RBX: 00007ffc5fd15950 RCX: 00007f331e6f02f7
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069727] RDX: 0000000000000001 RSI: 00007ffc5fcf837c RDI: 00007ffc5fd15950
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069727] RBP: 000000000186c640 R08: 00007ffc5fcf83a0 R09: 0000000000000000
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069728] R10: 0000000000000000 R11: 0000000000000202 R12: 000000000186ca40
Mar 22 23:18:18 de-rhr-ba4ngwfi01-01 kernel: [14000307.069729] R13: 00007ffc5fd15d50 R14: 000000000186ca40 R15: 00007ffc5fd34728
Mar 22 23:19:22 de-rhr-ba4ngwfi01-01 watchfrr.sh: Cannot stop bfdd: pid file not found
Mar 22 23:19:22 de-rhr-ba4ngwfi01-01 watchfrr.sh: Cannot stop staticd: pid file not found
Mar 22 23:19:22 de-rhr-ba4ngwfi01-01 watchfrr.sh: Cannot stop ripngd: pid file not found

And I got this message from VyOS lost IP static routes...

Mar 22 22:07:37 de-rhr-ba4ngwft01-01 Keepalived_vrrp[30692]: AgentX master agent failed to respond to ping. Attempting to re-register.
Mar 22 22:07:44 de-rhr-ba4ngwft01-01 lldpd[30792]: AgentX master agent failed to respond to ping. Attempting to re-register.
Mar 22 22:07:54 de-rhr-ba4ngwft01-01 kernel: [14000186.691601] [VLAN2305-INBOUND-100-A] IN=eth3.2305 OUT=eth1.200 MAC=00:50:56:bd:f9:1f:00:50:56:bd:9a:31:08:00 SRC=10.4.233.68 DST=208.91.0.89 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=20758 DF PROTO=TCP SPT=41822 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Mar 22 22:08:34 de-rhr-ba4ngwft01-01 kernel: [14000227.135220] [VLAN2301-INBOUND-100-A] IN=eth3.2301 OUT=eth1.200 MAC=00:50:56:bd:f9:1f:00:50:56:bd:14:6f:08:00 SRC=10.4.232.68 DST=208.91.0.89 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=37325 DF PROTO=TCP SPT=38064 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984298] IPv4: martian source 10.4.6.19 from 10.4.6.19, on dev eth1.180
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984313] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd 0b 67 08 06 .......PV..g..
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984343] IPv4: martian source 10.4.6.19 from 10.4.6.19, on dev eth1.180
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984345] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd 0b 67 08 06 .......PV..g..
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984370] IPv4: martian source 10.4.6.19 from 10.4.6.19, on dev eth1.180
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984372] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd 0b 67 08 06 .......PV..g..
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984376] IPv4: martian source 10.4.6.19 from 10.4.6.19, on dev eth1.180
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984378] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd 0b 67 08 06 .......PV..g..
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984381] IPv4: martian source 10.4.6.19 from 10.4.6.19, on dev eth1.180
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984383] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd 0b 67 08 06 .......PV..g..
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984448] IPv4: martian source 10.4.232.1 from 10.4.232.1, on dev eth3.2300
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984452] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd bd db 08 06 .......PV.....
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984468] IPv4: martian source 10.4.232.1 from 10.4.232.1, on dev eth3.2300
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984470] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd bd db 08 06 .......PV.....
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984757] IPv4: martian source 10.4.232.65 from 10.4.232.65, on dev eth3.2301
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984762] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd bd db 08 06 .......PV.....
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984888] IPv4: martian source 10.4.232.1 from 10.4.232.1, on dev eth3.2300
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984896] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd bd db 08 06 .......PV.....
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984938] IPv4: martian source 10.4.232.1 from 10.4.232.1, on dev eth3.2300
Mar 22 22:09:07 de-rhr-ba4ngwft01-01 kernel: [14000259.984945] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bd bd db 08 06 .......PV.....
Mar 22 22:09:57 de-rhr-ba4ngwft01-01 staticd[1204]: Terminating on signal
Mar 22 22:10:02 de-rhr-ba4ngwft01-01 kernel: [14000315.399864] [VLAN2300-INBOUND-100-A] IN=eth3.2300 OUT=eth1.200 MAC=00:50:56:bd:f9:1f:00:50:56:bd:35:68:08:00 SRC=10.4.232.4 DST=208.91.0.89 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=23385 DF PROTO=TCP SPT=38098 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Mar 22 22:10:03 de-rhr-ba4ngwft01-01 kernel: [14000316.397271] [VLAN2300-INBOUND-100-A] IN=eth3.2300 OUT=eth1.200 MAC=00:50:56:bd:f9:1f:00:50:56:bd:35:68:08:00 SRC=10.4.232.4 DST=208.91.0.89 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=23386 DF PROTO=TCP SPT=38098 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.817745] INFO: task jbd2/sda1-8:246 blocked for more than 120 seconds.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.817970] Not tainted 4.19.142-amd64-vyos #1
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818246] Call Trace:
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818253] 0xffffffffb10098c0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818254] ? 0xffffffffb100a530
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818254] 0xffffffffb1009d1d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818255] 0xffffffffb100a06d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818256] 0xffffffffb100a538
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818256] 0xffffffffb100a0ee
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818257] 0xffffffffb100a1a6
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818257] ? 0xffffffffb0aa9630
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818260] 0xffffffffc0286c8a
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818262] ? 0xffffffffb12001a5
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818263] ? 0xffffffffc028a085
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818264] 0xffffffffc028a085
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818265] ? 0xffffffffb0aa8de0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818265] ? 0xffffffffc0289fb0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818266] 0xffffffffb0a87005
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818267] ? 0xffffffffb0a86f00
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818267] 0xffffffffb1200215
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818275] INFO: task rs:main Q:Reg:26770 blocked for more than 120 seconds.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818425] Not tainted 4.19.142-amd64-vyos #1
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818534] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818707] Call Trace:
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818708] 0xffffffffb10098c0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818709] ? 0xffffffffb0f0e336
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818710] ? 0xffffffffb100a530
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818710] 0xffffffffb1009d1d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818710] 0xffffffffb100a06d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818711] 0xffffffffb100a538
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818711] 0xffffffffb100a0ee
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818712] 0xffffffffb100a1a6
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818712] ? 0xffffffffb0aa9630
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818713] 0xffffffffc02849d6
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818714] 0xffffffffc0284b97
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818715] 0xffffffffc02ee0f7
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818717] 0xffffffffc03077b5
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818718] 0xffffffffc0307995
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818719] ? 0xffffffffc0283a33
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818719] 0xffffffffc030b93e
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818720] 0xffffffffb0bf907d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818721] 0xffffffffb0be91bf
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818722] 0xffffffffb0be8bc9
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818722] 0xffffffffb0b4a944
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818723] 0xffffffffc02f7b86
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818724] 0xffffffffb0bc993b
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818725] 0xffffffffb0bcaafe
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818726] 0xffffffffc035b75d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818727] 0xffffffffb0bcbc8c
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818728] 0xffffffffb0bcbebe
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818728] 0xffffffffb0bcc155
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818729] 0xffffffffb0a03518
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818730] 0xffffffffb1200088
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818731] RIP: 0033:0x00007f6c2b4a7a3d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818740] Code: Bad RIP value.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818741] RSP: 002b:00007f6c23ffe600 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818742] RAX: ffffffffffffffda RBX: 00007f6c180203b0 RCX: 00007f6c2b4a7a3d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818742] RDX: 000000000000012a RSI: 00007f6c180203b0 RDI: 0000000000000006
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818743] RBP: 0000000000000000 R08: fffffffffffeac20 R09: fffffffffffeac10
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818744] R10: fffffffffffeac00 R11: 0000000000000293 R12: 00007f6c1800c3b0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818744] R13: 00007f6c23ffe620 R14: 0000000000000000 R15: 000000000000012a
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818746] INFO: task snmpd:30783 blocked for more than 120 seconds.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818882] Not tainted 4.19.142-amd64-vyos #1
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.818991] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819154] Call Trace:
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819155] 0xffffffffb10098c0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819156] ? 0xffffffffb100a530
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819156] 0xffffffffb1009d1d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819157] 0xffffffffb100a06d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819157] 0xffffffffb100a538
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819158] 0xffffffffb100a0ee
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819158] 0xffffffffb100a1a6
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819159] ? 0xffffffffb0aa9630
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819160] 0xffffffffc02849d6
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819160] 0xffffffffc0284b97
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819161] 0xffffffffc02ee0f7
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819162] 0xffffffffc031d73d
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819163] 0xffffffffc031ef72
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819163] 0xffffffffb0bdbeeb
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819164] 0xffffffffc035d343
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819165] 0xffffffffb0bdbeeb
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819166] 0xffffffffb0bde531
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819167] 0xffffffffb0a03518
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819167] 0xffffffffb1200088
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819168] RIP: 0033:0x00007f483ff012f7
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819169] Code: Bad RIP value.
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819170] RSP: 002b:00007ffc774140c8 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819171] RAX: ffffffffffffffda RBX: 00007ffc774140e0 RCX: 00007f483ff012f7
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819171] RDX: 0000000000000001 RSI: 00007ffc773f6b0c RDI: 00007ffc774140e0
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819172] RBP: 00000000013c2020 R08: 00007ffc773f6b30 R09: 0000000000000000
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819172] R10: 0000000000000000 R11: 0000000000000202 R12: 00000000013c2420
Mar 22 22:10:15 de-rhr-ba4ngwft01-01 kernel: [14000327.819173] R13: 00007ffc774144e0 R14: 00000000013c2420 R15: 00007ffc77432eb8
Mar 22 22:11:16 de-rhr-ba4ngwft01-01 ntpd_intres[26697]: host name not found: 0.pool.ntp.org
Mar 22 22:11:16 de-rhr-ba4ngwft01-01 ntpd_intres[26697]: host name not found: 1.pool.ntp.org
Mar 22 22:11:16 de-rhr-ba4ngwft01-01 ntpd_intres[26697]: host name not found: 2.pool.ntp.org
Mar 22 22:11:17 de-rhr-ba4ngwft01-01 watchfrr.sh: Cannot stop bfdd: pid file not found
Mar 22 22:11:17 de-rhr-ba4ngwft01-01 watchfrr.sh: Cannot stop staticd: pid file not found
Mar 22 22:11:17 de-rhr-ba4ngwft01-01 watchfrr.sh: Cannot stop ripngd: pid file not found
Mar 22 22:11:53 de-rhr-ba4ngwft01-01 kernel: [14000426.271769] [VLAN2316-INBOUND-100-A] IN=eth3.2316 OUT=eth1.200 MAC=00:50:56:bd:f9:1f:00:50:56:bd:d3:27:08:00 SRC=10.4.236.4 DST=208.91.0.89 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=19925 DF PROTO=TCP SPT=52930 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Mar 22 22:11:54 de-rhr-ba4ngwft01-01 kernel: [14000427.270625] [VLAN2316-INBOUND-100-A] IN=eth3.2316 OUT=eth1.200 MAC=00:50:56:bd:f9:1f:00:50:56:bd:d3:27:08:00 SRC=10.4.236.4 DST=208.91.0.89 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=19926 DF PROTO=TCP SPT=52930 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Mar 22 22:12:15 de-rhr-ba4ngwft01-01 kernel: [14000447.968880] [VLAN2311-INBOUND-100-A] IN=eth3.2311 OUT=eth1.200 MAC=00:50:56:bd:f9:1f:00:50:56:bd:c3:2f:08:00 SRC=10.4.234.196 DST=208.91.0.89 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=58438 DF PROTO=TCP SPT=58792 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.645810] INFO: task jbd2/sda1-8:246 blocked for more than 120 seconds.
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.645970] Not tainted 4.19.142-amd64-vyos #1
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646079] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646248] Call Trace:
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646250] 0xffffffffb10098c0
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646252] ? 0xffffffffb100a530
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646252] 0xffffffffb1009d1d
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646253] 0xffffffffb100a06d
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646254] 0xffffffffb100a538
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646254] 0xffffffffb100a0ee
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646255] 0xffffffffb100a1a6
Mar 22 22:12:16 de-rhr-ba4ngwft01-01 kernel: [14000448.646255] ? 0xffffffffb0aa9630
:

Details

Difficulty level: Unknown (require assessment)
Version: 1.2.6-S1
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Behavior change
Issue type: Bug (incorrect behavior)

Event Timeline

jalazaro created this task.Mar 24 2022, 12:32 PM

jalazaro updated the task description. (Show Details)

I happened today in 17 vyos clusters after a switch failure. Same logs and no bgp process running:

Apr 1 10:18:17 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop bfdd: pid file not found
Apr 1 10:18:17 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop staticd: pid file not found
Apr 1 10:18:17 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop ripngd: pid file not found
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: snmpd: send_trap: Failure in sendto (Network is unreachable)

More detail :

pr  1 10:14:32 gb-glo-sg4ng1fw34-01 staticd[1189]: Terminating on signal
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.413934] INFO: task jbd2/sda1-8:246 blocked for more than 120 seconds.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414331]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414735] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414985] Call Trace:
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414992]  0xffffffffa88098c0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414995]  ? 0xffffffffa880a530
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414996]  0xffffffffa8809d1d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414997]  0xffffffffa880a06d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414999]  0xffffffffa880a538
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.414999]  0xffffffffa880a0ee
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415000]  0xffffffffa880a1a6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415001]  ? 0xffffffffa82a9630
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415009]  0xffffffffc03f7c8a
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415015]  ? 0xffffffffa8a001a5
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415016]  ? 0xffffffffc03fb085
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415016]  0xffffffffc03fb085
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415019]  ? 0xffffffffa82a8de0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415020]  ? 0xffffffffc03fafb0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415020]  0xffffffffa8287005
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415021]  ? 0xffffffffa8286f00
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415021]  0xffffffffa8a00215
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415032] INFO: task rs:main Q:Reg:93129 blocked for more than 120 seconds.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415177]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415459] Call Trace:
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415460]  0xffffffffa88098c0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415461]  ? 0xffffffffa880a530
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415461]  0xffffffffa8809d1d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415462]  0xffffffffa880a06d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415462]  0xffffffffa880a538
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415463]  0xffffffffa880a0ee
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415463]  0xffffffffa880a1a6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415464]  ? 0xffffffffa82a9630
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415464]  0xffffffffc03f59d6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415465]  0xffffffffc03f5b97
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415471]  0xffffffffc04510f7
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415478]  0xffffffffc046a7b5
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415482]  0xffffffffc046a995
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415483]  ? 0xffffffffc03f4a33
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415483]  0xffffffffc046e93e
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415484]  0xffffffffa83f907d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415485]  0xffffffffa83e91bf
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415486]  0xffffffffa83e8bc9
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415486]  0xffffffffa834a944
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415487]  0xffffffffc045ab86
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415492]  0xffffffffa83c993b
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415493]  0xffffffffa83caafe
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415494]  0xffffffffc04be75d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415499]  0xffffffffa83cbc8c
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415500]  0xffffffffa83cbebe
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415501]  0xffffffffa83cc155
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415501]  0xffffffffa8203518
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415502]  0xffffffffa8a00088
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415503] RIP: 0033:0x00007ff9f8dd0a3d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415510] Code: Bad RIP value.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415511] RSP: 002b:00007ff9f5f3d600 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415512] RAX: ffffffffffffffda RBX: 00007ff9ec007190 RCX: 00007ff9f8dd0a3d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415513] RDX: 000000000000009c RSI: 00007ff9ec007190 RDI: 0000000000000007
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415513] RBP: 0000000000000000 R08: 0000000000004290 R09: 0000000000004280
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415514] R10: 0000000000004270 R11: 0000000000000293 R12: 00007ff9ec006f20
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415514] R13: 00007ff9f5f3d620 R14: 0000000000000000 R15: 000000000000009c
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415516] INFO: task snmpd:95204 blocked for more than 120 seconds.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415644]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415770] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415925] Call Trace:
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415935]  0xffffffffa88098c0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415935]  ? 0xffffffffa8402b5f
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415936]  ? 0xffffffffa880a530
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415936]  0xffffffffa8809d1d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415936]  0xffffffffa880a06d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415937]  0xffffffffa880a538
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415937]  0xffffffffa880a2f6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415938]  ? 0xffffffffc03f5736
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415938]  0xffffffffa880a3a6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415939]  ? 0xffffffffa82a9630
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415939]  0xffffffffc03f5996
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415940]  0xffffffffc03f5b97
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415940]  0xffffffffc04510f7
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415941]  0xffffffffc0481069
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415947]  0xffffffffc048209f
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415948]  0xffffffffa83dbeeb
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415948]  0xffffffffc04c0343
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415951]  0xffffffffa83dbeeb
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415951]  0xffffffffa83de531
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415952]  0xffffffffa8203518
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415953]  0xffffffffa8a00088
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415953] RIP: 0033:0x00007fbd781de2f7
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415955] Code: Bad RIP value.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415955] RSP: 002b:00007ffd8b499968 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415956] RAX: ffffffffffffffda RBX: 00007ffd8b499980 RCX: 00007fbd781de2f7
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415957] RDX: 0000000000000001 RSI: 00007ffd8b47c3ac RDI: 00007ffd8b499980
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415957] RBP: 0000000001470fd0 R08: 00007ffd8b47c3d0 R09: 0000000000000000
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415958] R10: 0000000000000000 R11: 0000000000000202 R12: 00000000014713d0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415958] R13: 00007ffd8b499d80 R14: 00000000014713d0 R15: 00007ffd8b4b8758
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.415969] INFO: task bash:58488 blocked for more than 120 seconds.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416098]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416225] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416381] Call Trace:
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416383]  0xffffffffa88098c0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416384]  ? 0xffffffffa880a530
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416384]  0xffffffffa8809d1d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416384]  0xffffffffa880a06d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416385]  0xffffffffa880a538
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416385]  0xffffffffa880a2f6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416386]  0xffffffffa880a3a6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416386]  ? 0xffffffffa82a9630
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416387]  0xffffffffc03f5996
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416388]  0xffffffffc03f5b97
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416388]  0xffffffffc04510f7
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416389]  0xffffffffc0481069
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416389]  0xffffffffc046e842
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416390]  ? 0xffffffffa83e8ac8
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416391]  0xffffffffa83eb70d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416391]  0xffffffffc04bcaa9
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416392]  0xffffffffa83eb70d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416393]  0xffffffffa83c7d7d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416394]  0xffffffffa83da859
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416394]  ? 0xffffffffa83e668b
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416395]  0xffffffffa83ddad4
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416396]  ? 0xffffffffa837d06f
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416396]  ? 0xffffffffa83c66fc
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416397]  ? 0xffffffffa83ec6ae
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416397]  ? 0xffffffffa83c8ff2
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416398]  0xffffffffa83c8ff2
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416398]  0xffffffffa8203518
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416399]  0xffffffffa8a00088
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416399] RIP: 0033:0x00007ff9b56c8960
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416401] Code: Bad RIP value.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416401] RSP: 002b:00007fff0546b9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416404] RAX: ffffffffffffffda RBX: 0000000001bdff08 RCX: 00007ff9b56c8960
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416405] RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 0000000001be1548
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416405] RBP: 00007fff0546ba50 R08: 0000000000000020 R09: 0000000001bdfe88
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416406] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416406] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000001be1548
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416411] INFO: task bash:58497 blocked for more than 120 seconds.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416537]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416663] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416821] Call Trace:
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416822]  0xffffffffa88098c0
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416822]  ? 0xffffffffa84624b8
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416823]  0xffffffffa8809d1d
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416823]  0xffffffffa880db70
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416824]  ? 0xffffffffa8801383
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416824]  0xffffffffa8801383
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416825]  0xffffffffa880c95b
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416826]  0xffffffffa83c7d70
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416827]  ? 0xffffffffc04be88b
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416827]  ? 0xffffffffa83c79b6
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416828]  0xffffffffa83da859
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416828]  ? 0xffffffffa83e668b
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416829]  0xffffffffa83ddad4
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416829]  ? 0xffffffffa837d06f
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416830]  ? 0xffffffffa83c66fc
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416830]  ? 0xffffffffa83ec6ae
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416831]  ? 0xffffffffa83c8ff2
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416831]  0xffffffffa83c8ff2
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416832]  0xffffffffa8203518
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416832]  0xffffffffa8a00088
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416833] RIP: 0033:0x00007fc9417f5960
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416834] Code: Bad RIP value.
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416835] RSP: 002b:00007ffc1c714ed8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416835] RAX: ffffffffffffffda RBX: 0000000000836f08 RCX: 00007fc9417f5960
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416836] RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 0000000000838548
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416836] RBP: 00007ffc1c714f70 R08: 0000000000000020 R09: 0000000000836e88
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416837] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Apr  1 10:15:16 gb-glo-sg4ng1fw34-01 kernel: [29189055.416837] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000838548
Apr  1 10:15:57 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop bfdd: pid file not found
Apr  1 10:15:57 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop staticd: pid file not found
Apr  1 10:15:57 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop ripngd: pid file not found
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246290] INFO: task jbd2/sda1-8:246 blocked for more than 120 seconds.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246451]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246737] Call Trace:
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246741]  0xffffffffa88098c0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246742]  ? 0xffffffffa880a530
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246743]  0xffffffffa8809d1d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246743]  0xffffffffa880a06d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246744]  0xffffffffa880a538
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246744]  0xffffffffa880a0ee
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246745]  0xffffffffa880a1a6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246746]  ? 0xffffffffa82a9630
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246753]  0xffffffffc03f7c8a
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246758]  ? 0xffffffffa8a001a5
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246760]  ? 0xffffffffc03fb085
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246760]  0xffffffffc03fb085
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246763]  ? 0xffffffffa82a8de0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246763]  ? 0xffffffffc03fafb0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246764]  0xffffffffa8287005
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246765]  ? 0xffffffffa8286f00
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246765]  0xffffffffa8a00215
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246785] INFO: task rs:main Q:Reg:93129 blocked for more than 120 seconds.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.246933]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247060] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247217] Call Trace:
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247221]  0xffffffffa88098c0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247222]  ? 0xffffffffa880a530
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247222]  0xffffffffa8809d1d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247222]  0xffffffffa880a06d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247223]  0xffffffffa880a538
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247223]  0xffffffffa880a0ee
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247224]  0xffffffffa880a1a6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247224]  ? 0xffffffffa82a9630
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247225]  0xffffffffc03f59d6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247226]  0xffffffffc03f5b97
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247229]  0xffffffffc04510f7
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247237]  0xffffffffc046a7b5
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247241]  0xffffffffc046a995
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247242]  ? 0xffffffffc03f4a33
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247242]  0xffffffffc046e93e
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247243]  0xffffffffa83f907d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247244]  0xffffffffa83e91bf
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247244]  0xffffffffa83e8bc9
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247245]  0xffffffffa834a944
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247248]  0xffffffffc045ab86
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247251]  0xffffffffa83c993b
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247252]  0xffffffffa83caafe
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247253]  0xffffffffc04be75d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247256]  0xffffffffa83cbc8c
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247256]  0xffffffffa83cbebe
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247257]  0xffffffffa83cc155
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247258]  0xffffffffa8203518
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247258]  0xffffffffa8a00088
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247259] RIP: 0033:0x00007ff9f8dd0a3d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247269] Code: Bad RIP value.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247270] RSP: 002b:00007ff9f5f3d600 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247271] RAX: ffffffffffffffda RBX: 00007ff9ec007190 RCX: 00007ff9f8dd0a3d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247271] RDX: 000000000000009c RSI: 00007ff9ec007190 RDI: 0000000000000007
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247272] RBP: 0000000000000000 R08: 0000000000004290 R09: 0000000000004280
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247272] R10: 0000000000004270 R11: 0000000000000293 R12: 00007ff9ec006f20
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247273] R13: 00007ff9f5f3d620 R14: 0000000000000000 R15: 000000000000009c
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247274] INFO: task snmpd:95204 blocked for more than 120 seconds.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247403]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247530] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247690] Call Trace:
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247693]  0xffffffffa88098c0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247694]  ? 0xffffffffa8402b5f
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247694]  ? 0xffffffffa880a530
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247694]  0xffffffffa8809d1d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247695]  0xffffffffa880a06d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247695]  0xffffffffa880a538
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247696]  0xffffffffa880a2f6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247696]  ? 0xffffffffc03f5736
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247697]  0xffffffffa880a3a6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247697]  ? 0xffffffffa82a9630
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247697]  0xffffffffc03f5996
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247698]  0xffffffffc03f5b97
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247699]  0xffffffffc04510f7
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247699]  0xffffffffc0481069
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247705]  0xffffffffc048209f
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247706]  0xffffffffa83dbeeb
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247707]  0xffffffffc04c0343
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247709]  0xffffffffa83dbeeb
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247710]  0xffffffffa83de531
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247710]  0xffffffffa8203518
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247711]  0xffffffffa8a00088
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247711] RIP: 0033:0x00007fbd781de2f7
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247713] Code: Bad RIP value.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247714] RSP: 002b:00007ffd8b499968 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247714] RAX: ffffffffffffffda RBX: 00007ffd8b499980 RCX: 00007fbd781de2f7
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247715] RDX: 0000000000000001 RSI: 00007ffd8b47c3ac RDI: 00007ffd8b499980
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247715] RBP: 0000000001470fd0 R08: 00007ffd8b47c3d0 R09: 0000000000000000
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247716] R10: 0000000000000000 R11: 0000000000000202 R12: 00000000014713d0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247716] R13: 00007ffd8b499d80 R14: 00000000014713d0 R15: 00007ffd8b4b8758
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247740] INFO: task bash:58488 blocked for more than 120 seconds.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247868]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.247995] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248151] Call Trace:
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248152]  0xffffffffa88098c0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248153]  ? 0xffffffffa880a530
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248153]  0xffffffffa8809d1d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248154]  0xffffffffa880a06d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248154]  0xffffffffa880a538
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248155]  0xffffffffa880a2f6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248155]  0xffffffffa880a3a6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248156]  ? 0xffffffffa82a9630
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248156]  0xffffffffc03f5996
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248157]  0xffffffffc03f5b97
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248157]  0xffffffffc04510f7
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248158]  0xffffffffc0481069
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248159]  0xffffffffc046e842
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248159]  ? 0xffffffffa83e8ac8
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248160]  0xffffffffa83eb70d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248161]  0xffffffffc04bcaa9
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248161]  0xffffffffa83eb70d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248162]  0xffffffffa83c7d7d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248163]  0xffffffffa83da859
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248163]  ? 0xffffffffa83e668b
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248164]  0xffffffffa83ddad4
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248165]  ? 0xffffffffa837d06f
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248165]  ? 0xffffffffa83c66fc
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248166]  ? 0xffffffffa83ec6ae
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248166]  ? 0xffffffffa83c8ff2
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248167]  0xffffffffa83c8ff2
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248167]  0xffffffffa8203518
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248168]  0xffffffffa8a00088
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248168] RIP: 0033:0x00007ff9b56c8960
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248170] Code: Bad RIP value.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248170] RSP: 002b:00007fff0546b9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248173] RAX: ffffffffffffffda RBX: 0000000001bdff08 RCX: 00007ff9b56c8960
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248174] RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 0000000001be1548
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248175] RBP: 00007fff0546ba50 R08: 0000000000000020 R09: 0000000001bdfe88
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248175] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248175] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000001be1548
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248180] INFO: task bash:58497 blocked for more than 120 seconds.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248311]       Tainted: G             L    4.19.142-amd64-vyos #1
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248438] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248596] Call Trace:
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248597]  0xffffffffa88098c0
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248597]  ? 0xffffffffa84624b8
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248598]  0xffffffffa8809d1d
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248598]  0xffffffffa880db70
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248599]  ? 0xffffffffa8801383
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248599]  0xffffffffa8801383
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248600]  0xffffffffa880c95b
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248601]  0xffffffffa83c7d70
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248601]  ? 0xffffffffc04be88b
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248602]  ? 0xffffffffa83c79b6
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248602]  0xffffffffa83da859
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248603]  ? 0xffffffffa83e668b
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248604]  0xffffffffa83ddad4
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248604]  ? 0xffffffffa837d06f
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248605]  ? 0xffffffffa83c66fc
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248605]  ? 0xffffffffa83ec6ae
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248606]  ? 0xffffffffa83c8ff2
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248606]  0xffffffffa83c8ff2
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248607]  0xffffffffa8203518
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248607]  0xffffffffa8a00088
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248608] RIP: 0033:0x00007fc9417f5960
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248609] Code: Bad RIP value.
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248610] RSP: 002b:00007ffc1c714ed8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248610] RAX: ffffffffffffffda RBX: 0000000000836f08 RCX: 00007fc9417f5960
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248611] RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 0000000000838548
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248611] RBP: 00007ffc1c714f70 R08: 0000000000000020 R09: 0000000000836e88
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248612] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Apr  1 10:17:17 gb-glo-sg4ng1fw34-01 kernel: [29189176.248612] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000838548
Apr  1 10:18:17 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop bfdd: pid file not found
Apr  1 10:18:17 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop staticd: pid file not found
Apr  1 10:18:17 gb-glo-sg4ng1fw34-01 watchfrr.sh: Cannot stop ripngd: pid file not found
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: snmpd: send_trap: Failure in sendto (Network is unreachable)
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: message repeated 3 times: [ snmpd: send_trap: Failure in sendto (Network is unreachable)]
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 rsyslogd-2007: action 'action 3' suspended, next retry is Fri Apr  1 10:22:00 2022 [try http://www.rsyslog.com/e/2007 ]
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: snmpd: send_trap: Failure in sendto (Network is unreachable)
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: message repeated 8 times: [ snmpd: send_trap: Failure in sendto (Network is unreachable)]
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: snmpd: send_trap: Failure in sendto (Network is unreachable)
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: message repeated 9 times: [ snmpd: send_trap: Failure in sendto (Network is unreachable)]
Apr  1 10:21:30 gb-glo-sg4ng1fw34-01 snmpd[95204]: snmpd: send_trap: Failure in sendto (Network is unreachable)
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193505] IPv4: martian source 10.255.255.1 from 10.255.255.1, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193510] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193591] IPv4: martian source 169.254.169.254 from 169.254.169.254, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193592] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193595] IPv4: martian source 10.255.255.1 from 10.255.255.1, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193596] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193598] IPv4: martian source 169.254.169.254 from 169.254.169.254, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193599] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193601] IPv4: martian source 10.255.255.1 from 10.255.255.1, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193602] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193604] IPv4: martian source 169.254.169.254 from 169.254.169.254, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193605] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193608] IPv4: martian source 10.255.255.1 from 10.255.255.1, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193608] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193611] IPv4: martian source 169.254.169.254 from 169.254.169.254, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193612] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193614] IPv4: martian source 10.255.255.1 from 10.255.255.1, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193615] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193617] IPv4: martian source 169.254.169.254 from 169.254.169.254, on dev eth3
Apr  1 10:21:41 gb-glo-sg4ng1fw34-01 kernel: [29189440.193618] ll header: 00000000: ff ff ff ff ff ff 00 50 56 bb 5f 42 08 06        .......PV._B..

pasik added a subscriber: pasik.Apr 1 2022, 4:33 PM

I think all Chaos stater after this message:

vyos staticd[1189]: Terminating on signal

What could be the reason for the static routing daemon to crash? This is frr-log showing zebra, ospf,bgp etc down (the recover of the process was after reboot the vyos)

Apr 1 10:11:26 gb-glo-sg4ng1fw34-01 bgpd[1159]: %NOTIFICATION: sent to neighbor 109.228.63.134 4/0 (Hold Timer Expired) 0 bytes
Apr 1 10:11:26 gb-glo-sg4ng1fw34-01 bgpd[1159]: %ADJCHANGE: neighbor 109.228.63.134(Unknown) in vrf default Down BGP Notification send
Apr 1 10:12:10 gb-glo-sg4ng1fw34-01 ospf6d[1179]: [EC 100663310] snmp[warning]: AgentX master agent failed to respond to ping. Attempting to re-register.
Apr 1 10:12:11 gb-glo-sg4ng1fw34-01 zebra[1153]: [EC 100663310] snmp[warning]: AgentX master agent failed to respond to ping. Attempting to re-register.
Apr 1 10:12:17 gb-glo-sg4ng1fw34-01 ripd[1167]: [EC 100663310] snmp[warning]: AgentX master agent failed to respond to ping. Attempting to re-register.
Apr 1 10:12:19 gb-glo-sg4ng1fw34-01 ospfd[1175]: [EC 100663310] snmp[warning]: AgentX master agent failed to respond to ping. Attempting to re-register.
Apr 1 10:12:22 gb-glo-sg4ng1fw34-01 ospf6d[1179]: [EC 100663313] SLOW THREAD: task agentx_timeout (7fe79db1ab80) ran for 18017ms (cpu time 0ms)
Apr 1 10:12:23 gb-glo-sg4ng1fw34-01 zebra[1153]: [EC 100663313] SLOW THREAD: task agentx_timeout (7f224d34db80) ran for 18018ms (cpu time 0ms)
Apr 1 10:12:24 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 100663310] snmp[warning]: AgentX master agent failed to respond to ping. Attempting to re-register.
Apr 1 10:12:29 gb-glo-sg4ng1fw34-01 ripd[1167]: [EC 100663313] SLOW THREAD: task agentx_timeout (7f0f40f21b80) ran for 18018ms (cpu time 0ms)
Apr 1 10:12:31 gb-glo-sg4ng1fw34-01 ospfd[1175]: [EC 100663313] SLOW THREAD: task agentx_timeout (7f92d9504b80) ran for 18019ms (cpu time 0ms)
Apr 1 10:13:52 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] bgpd state -> unresponsive : no response yet to ping sent 90 seconds ago
Apr 1 10:13:52 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 100663303] Forked background command [pid 58498]: /usr/lib/frr/watchfrr.sh restart bgpd
Apr 1 10:14:12 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Warning: restart bgpd child process 58498 still running after 20 seconds, sending signal 15
Apr 1 10:14:12 gb-glo-sg4ng1fw34-01 watchfrr[1120]: restart bgpd process 58498 terminated due to signal 15
Apr 1 10:14:12 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] ospf6d state -> unresponsive : no response yet to ping sent 90 seconds ago
Apr 1 10:14:12 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 100663303] Forked background command [pid 58709]: /usr/lib/frr/watchfrr.sh restart ospf6d
Apr 1 10:14:13 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] zebra state -> unresponsive : no response yet to ping sent 90 seconds ago
Apr 1 10:14:14 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] ripd state -> unresponsive : no response yet to ping sent 90 seconds ago
Apr 1 10:14:14 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] pimd state -> unresponsive : no response yet to ping sent 90 seconds ago
Apr 1 10:14:16 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] ospfd state -> unresponsive : no response yet to ping sent 90 seconds ago
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Warning: restart ospf6d child process 58709 still running after 20 seconds, sending signal 15
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 watchfrr[1120]: restart ospf6d process 58709 terminated due to signal 15
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 100663303] Forked background command [pid 58913]: /usr/lib/frr/watchfrr.sh restart all
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 ripngd[1171]: Terminating on signal
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 zebra[1153]: [EC 4043309121] Client 'bfd' encountered an error and is shutting down.
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] staticd state -> down : read returned EOF
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] bfdd state -> down : read returned EOF
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 zebra[1153]: [EC 4043309121] Client 'static' encountered an error and is shutting down.
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 zebra[1153]: [EC 4043309121] Client 'ripng' encountered an error and is shutting down.
Apr 1 10:14:32 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] ripngd state -> down : read returned EOF
Apr 1 10:14:52 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Warning: restart all child process 58913 still running after 20 seconds, sending signal 15
Apr 1 10:14:52 gb-glo-sg4ng1fw34-01 watchfrr[1120]: restart all process 58913 terminated due to signal 15
Apr 1 10:15:57 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 100663303] Forked background command [pid 60153]: /usr/lib/frr/watchfrr.sh restart all
Apr 1 10:16:17 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Warning: restart all child process 60153 still running after 20 seconds, sending signal 15
Apr 1 10:16:17 gb-glo-sg4ng1fw34-01 watchfrr[1120]: restart all process 60153 terminated due to signal 15
Apr 1 10:18:17 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 100663303] Forked background command [pid 61399]: /usr/lib/frr/watchfrr.sh restart all
Apr 1 10:18:37 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Warning: restart all child process 61399 still running after 20 seconds, sending signal 15
Apr 1 10:18:37 gb-glo-sg4ng1fw34-01 watchfrr[1120]: restart all process 61399 terminated due to signal 15
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 ospf6d[1179]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 ospfd[1175]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 ripd[1167]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 ripd[1167]: [EC 100663313] SLOW THREAD: task agentx_timeout (7f0f40f21b80) ran for 525574ms (cpu time 0ms)
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 ripd[1167]: Terminating on signal
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: [EC 4043309121] Client 'rip' encountered an error and is shutting down.
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] ripd state -> down : unexpected read error: Connection reset by peer
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: [EC 100663313] SLOW THREAD: task agentx_timeout (7f224d34db80) ran for 532134ms (cpu time 1ms)
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: Terminating on signal
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 20 disconnected. 14 bgp routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 32 disconnected. 0 vnc routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 37 disconnected. 0 rip routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 42 disconnected. 0 ripng routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 47 disconnected. 0 ospf routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 52 disconnected. 0 ospf6 routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 57 disconnected. 0 pim routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 60 disconnected. 0 system routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 pimd[1184]: [EC 100663300] zclient_read_nexthop: zclient_read_header() failed
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 pimd[1184]: zclient_lookup_sched_now: zclient lookup immediate connection scheduled
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 pimd[1184]: [EC 100663313] SLOW THREAD: task zclient_lookup_read_pipe (55d39533f6a0) ran for 527772ms (cpu time 0ms)
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 pimd[1184]: Terminating on signal SIGINT
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 67 disconnected. 639 static routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: client 72 disconnected. 0 bfd routes removed from the rib
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] pimd state -> down : unexpected read error: Connection reset by peer
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 100663313] SLOW THREAD: task agentx_timeout (7f29891deb80) ran for 551403ms (cpu time 2ms)
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: Terminating on signal
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: %NOTIFICATION: sent to neighbor 109.228.63.135 6/3 (Cease/Peer Unconfigured) 0 bytes
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: %NOTIFICATION: sent to neighbor 109.228.63.136 6/3 (Cease/Peer Unconfigured) 0 bytes
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 100663299] buffer_write: write error on fd 20: Broken pipe
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 100663302] zclient_send_message: buffer_write failed to zclient fd 20, closing
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: %NOTIFICATION: sent to neighbor 109.228.63.135 6/2 (Cease/Administratively Shutdown) 0 bytes
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: %NOTIFICATION: sent to neighbor 109.228.63.136 6/2 (Cease/Administratively Shutdown) 0 bytes
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 33554499] sendmsg_nexthop: zclient_send_message() failed
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: %ADJCHANGE: neighbor 109.228.63.135(Unknown) in vrf default Down Neighbor deleted
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 33554499] sendmsg_nexthop: zclient_send_message() failed
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: %ADJCHANGE: neighbor 109.228.63.136(Unknown) in vrf default Down Neighbor deleted
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 33554499] sendmsg_nexthop: zclient_send_message() failed
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 bgpd[1159]: [EC 33554499] sendmsg_nexthop: zclient_send_message() failed
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 ospfd[1175]: [EC 100663313] SLOW THREAD: task agentx_timeout (7f92d9504b80) ran for 523629ms (cpu time 3ms)
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 ospfd[1175]: Terminating on signal
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] ospfd state -> down : unexpected read error: Connection reset by peer
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 watchfrr[1120]: zebra: slow echo response finally received after 527.134047 seconds
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 zebra[1153]: Zebra final shutdown
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] bgpd state -> down : unexpected read error: Connection reset by peer
Apr 1 10:21:30 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] zebra state -> down : read returned EOF
Apr 1 10:21:51 gb-glo-sg4ng1fw34-01 ospf6d[1179]: [EC 100663313] SLOW THREAD: task agentx_timeout (7fe79db1ab80) ran for 553649ms (cpu time 3ms)
Apr 1 10:21:51 gb-glo-sg4ng1fw34-01 ospf6d[1179]: Terminating on signal SIGINT
Apr 1 10:21:51 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 268435457] ospf6d state -> down : unexpected read error: Connection reset by peer
Apr 1 10:22:39 gb-glo-sg4ng1fw34-01 watchfrr[1120]: [EC 100663303] Forked background command [pid 63836]: /usr/lib/frr/watchfrr.sh restart all
Apr 1 10:22:39 gb-glo-sg4ng1fw34-01 zebra[63874]: client 20 says hello and bids fair to announce only bgp routes vrf=0
Apr 1 10:22:39 gb-glo-sg4ng1fw34-01 zebra[63874]: client 32 says hello and bids fair to announce only vnc routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 zebra[63874]: client 37 says hello and bids fair to announce only rip routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: ripd state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 zebra[63874]: client 42 says hello and bids fair to announce only ripng routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 zebra[63874]: client 47 says hello and bids fair to announce only ospf routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 zebra[63874]: client 52 says hello and bids fair to announce only ospf6 routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: bgpd state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 zebra[63874]: client 57 says hello and bids fair to announce only pim routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: pimd state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 zebra[63874]: client 67 says hello and bids fair to announce only static routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 zebra[63874]: client 72 says hello and bids fair to announce only bfd routes vrf=0
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: zebra state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Daemon: bgpd: is in Up state but expected it to be in DAEMON_DOWN state
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Daemon: ripd: is in Up state but expected it to be in DAEMON_DOWN state
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: ripngd state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: ospfd state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: ospf6d state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: Daemon: pimd: is in Up state but expected it to be in DAEMON_DOWN state
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: staticd state -> up : connect succeeded
Apr 1 10:22:40 gb-glo-sg4ng1fw34-01 watchfrr[1120]: bfdd state -> up : connect succeeded
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: watchfrr 7.3.1 starting: vty@0
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] zebra state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] bgpd state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] ripd state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] ripngd state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] ospfd state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] ospf6d state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] pimd state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] staticd state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 268435457] bfdd state -> down : initial connection attempt failed
Apr 1 10:45:49 gb-glo-sg4ng1fw34-01 watchfrr[1108]: [EC 100663303] Forked background command [pid 1109]: /usr/lib/frr/watchfrr.sh restart all
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 20 says hello and bids fair to announce only bgp routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 30 says hello and bids fair to announce only vnc routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 37 says hello and bids fair to announce only rip routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 42 says hello and bids fair to announce only ripng routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 47 says hello and bids fair to announce only ospf routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 52 says hello and bids fair to announce only ospf6 routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 57 says hello and bids fair to announce only pim routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 67 says hello and bids fair to announce only static routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 zebra[1141]: client 72 says hello and bids fair to announce only bfd routes vrf=0
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: zebra state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: bgpd state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: ripd state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: ripngd state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: ospfd state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: ospf6d state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: pimd state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: staticd state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: bfdd state -> up : connect succeeded
Apr 1 10:45:50 gb-glo-sg4ng1fw34-01 watchfrr[1108]: all daemons up, doing startup-complete notify
Apr 1 11:04:37 gb-glo-sg4ng1fw34-01 zebra[1141]: if_zebra_speed_update: eth1.3201 old speed: 0 new speed: 10000
Apr 1 11:05:34 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.134(Unknown) in vrf default Up
Apr 1 11:05:34 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.135(Unknown) in vrf default Up
Apr 1 11:05:35 gb-glo-sg4ng1fw34-01 bgpd[1146]: [EC 33554503] 109.228.63.136 unrecognized capability code: 70 - ignored
Apr 1 11:05:35 gb-glo-sg4ng1fw34-01 bgpd[1146]: [EC 33554503] 109.228.63.136 unrecognized capability code: 71 - ignored
Apr 1 11:05:35 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.136(Unknown) in vrf default Up
Apr 1 11:05:35 gb-glo-sg4ng1fw34-01 bgpd[1146]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 109.228.63.136 in vrf default
Apr 1 11:05:35 gb-glo-sg4ng1fw34-01 bgpd[1146]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 109.228.63.135 in vrf default
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 bgpd[1146]: %NOTIFICATION: sent to neighbor 109.228.63.134 6/4 (Cease/Administratively Reset) 0 bytes
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 bgpd[1146]: %NOTIFICATION: sent to neighbor 109.228.63.135 6/4 (Cease/Administratively Reset) 0 bytes
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 bgpd[1146]: %NOTIFICATION: sent to neighbor 109.228.63.136 6/4 (Cease/Administratively Reset) 0 bytes
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.134(Unknown) in vrf default Down User reset
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.135(Unknown) in vrf default Down User reset
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.136(Unknown) in vrf default Down User reset
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 zebra[1141]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 ripd[1155]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 ospfd[1163]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 ospf6d[1167]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 11:05:36 gb-glo-sg4ng1fw34-01 bgpd[1146]: snmp[info]: NET-SNMP version 5.7.2.1 AgentX subagent connected
Apr 1 11:05:45 gb-glo-sg4ng1fw34-01 bgpd[1146]: [EC 33554454] 109.228.63.136 [Error] bgp_read_packet error: Connection reset by peer
Apr 1 11:05:45 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.134(Unknown) in vrf default Up
Apr 1 11:05:45 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.135(Unknown) in vrf default Up
Apr 1 11:05:46 gb-glo-sg4ng1fw34-01 bgpd[1146]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 109.228.63.135 in vrf default
Apr 1 11:05:46 gb-glo-sg4ng1fw34-01 bgpd[1146]: %NOTIFICATION: sent to neighbor 109.228.63.134 6/4 (Cease/Administratively Reset) 0 bytes
Apr 1 11:05:46 gb-glo-sg4ng1fw34-01 bgpd[1146]: %NOTIFICATION: sent to neighbor 109.228.63.135 6/4 (Cease/Administratively Reset) 0 bytes
Apr 1 11:05:46 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.134(Unknown) in vrf default Down User reset
Apr 1 11:05:46 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.135(Unknown) in vrf default Down Peer closed the session
Apr 1 11:05:47 gb-glo-sg4ng1fw34-01 bgpd[1146]: [EC 33554454] 109.228.63.136 [Error] bgp_read_packet error: Connection reset by peer
Apr 1 11:05:48 gb-glo-sg4ng1fw34-01 bgpd[1146]: [EC 33554454] 109.228.63.135 [Error] bgp_read_packet error: Connection reset by peer
Apr 1 11:05:51 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.134(Unknown) in vrf default Up
Apr 1 11:05:53 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.135(Unknown) in vrf default Up
Apr 1 11:05:54 gb-glo-sg4ng1fw34-01 bgpd[1146]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 109.228.63.135 in vrf default
Apr 1 11:06:27 gb-glo-sg4ng1fw34-01 bgpd[1146]: [EC 33554503] 109.228.63.136 unrecognized capability code: 70 - ignored
Apr 1 11:06:27 gb-glo-sg4ng1fw34-01 bgpd[1146]: [EC 33554503] 109.228.63.136 unrecognized capability code: 71 - ignored
Apr 1 11:06:27 gb-glo-sg4ng1fw34-01 bgpd[1146]: %ADJCHANGE: neighbor 109.228.63.136(Unknown) in vrf default Up
Apr 1 11:06:27 gb-glo-sg4ng1fw34-01 bgpd[1146]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 109.228.63.136 in vrf default

If the issue is still reproducible, feel free to reopen.

IP routes loss after AgentX master agent failed to respond to pingResolved (N/A)PublicBUGActions

Description

Details

Event Timeline

IP routes loss after AgentX master agent failed to respond to ping
Resolved (N/A)PublicBUG
Actions