We have been getting random freeze ups for the past week or so since upgrading to 1.2.0. Initially we thought it was a hardware issue on our primary firewall, but after bringing the secondary firewall online it has been giving the same errors.
Once this happens, the firewall stops giving out DHCP leases, and /etc/hosts ends up with garbage written to it. Rebooting will bring everything back online, but the DNS recursor won't come back online until the garbage is removed from /etc/hosts
Mar 5 10:06:01 firewall-2 kernel: [ 2176.183169] INFO: task kworker/u16:4:165 blocked for more than 120 seconds. Mar 5 10:06:01 firewall-2 kernel: [ 2176.266599] Not tainted 4.19.12-amd64-vyos #1 Mar 5 10:06:01 firewall-2 kernel: [ 2176.325029] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 5 10:06:01 firewall-2 kernel: [ 2176.418735] Call Trace: Mar 5 10:06:01 firewall-2 kernel: [ 2176.418738] ? 0xffffffff8a5cd1a0 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418739] ? 0xffffffff8a6001a4 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418740] 0xffffffff8a5cd5fd Mar 5 10:06:01 firewall-2 kernel: [ 2176.418742] 0xffffffffc0161b24 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418743] ? 0xffffffff8a0a5130 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418744] 0xffffffffc0167409 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418745] 0xffffffffc0167bde Mar 5 10:06:01 firewall-2 kernel: [ 2176.418747] 0xffffffffc0178227 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418748] 0xffffffffc0178ebc Mar 5 10:06:01 firewall-2 kernel: [ 2176.418749] 0xffffffff8a2831cd Mar 5 10:06:01 firewall-2 kernel: [ 2176.418750] 0xffffffff8a2837a1 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418751] 0xffffffff8a1f1319 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418753] ? 0xffffffff8a1f1962 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418754] 0xffffffff8a1f1962 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418755] 0xffffffff8a07e8ea Mar 5 10:06:01 firewall-2 kernel: [ 2176.418756] 0xffffffff8a07fd42 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418757] ? 0xffffffff8a07fce0 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418758] 0xffffffff8a084495 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418759] ? 0xffffffff8a084390 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418760] 0xffffffff8a600215 Mar 5 10:06:01 firewall-2 kernel: [ 2176.418763] INFO: task md127_raid1:198 blocked for more than 120 seconds. Mar 5 10:06:01 firewall-2 kernel: [ 2176.500057] Not tainted 4.19.12-amd64-vyos #1 Mar 5 10:06:01 firewall-2 kernel: [ 2176.558399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 5 10:06:01 firewall-2 kernel: [ 2176.652105] Call Trace: Mar 5 10:06:01 firewall-2 kernel: [ 2176.652106] ? 0xffffffff8a5cd1a0 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652107] ? 0xffffffff8a6001b0 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652108] 0xffffffff8a5cd5fd Mar 5 10:06:01 firewall-2 kernel: [ 2176.652109] 0xffffffffc0161b24 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652110] ? 0xffffffff8a0a5130 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652111] 0xffffffffc0167409 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652112] ? 0xffffffffc016829b Mar 5 10:06:01 firewall-2 kernel: [ 2176.652113] 0xffffffffc0169f14 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652114] 0xffffffffc0164c4d Mar 5 10:06:01 firewall-2 kernel: [ 2176.652115] 0xffffffffc017cc4b Mar 5 10:06:01 firewall-2 kernel: [ 2176.652117] ? 0xffffffff8a0cd0a8 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652117] ? 0xffffffff8a0cd100 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652118] ? 0xffffffff8a5d177f Mar 5 10:06:01 firewall-2 kernel: [ 2176.652119] ? 0xffffffffc015a0d7 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652120] 0xffffffffc015a0d7 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652121] ? 0xffffffff8a0a5130 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652122] ? 0xffffffffc0159fb0 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652122] 0xffffffff8a084495 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652123] ? 0xffffffff8a084390 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652124] 0xffffffff8a600215 Mar 5 10:06:01 firewall-2 kernel: [ 2176.652126] INFO: task jbd2/md127-8:253 blocked for more than 120 seconds. Mar 5 10:06:01 firewall-2 kernel: [ 2176.734474] Not tainted 4.19.12-amd64-vyos #1 Mar 5 10:06:01 firewall-2 kernel: [ 2176.792814] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 5 10:06:01 firewall-2 kernel: [ 2176.886520] Call Trace: Mar 5 10:06:01 firewall-2 kernel: [ 2176.886521] ? 0xffffffff8a5cd1a0 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886522] ? 0xffffffff8a5cdde0 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886523] 0xffffffff8a5cd5fd Mar 5 10:06:01 firewall-2 kernel: [ 2176.886524] 0xffffffff8a08ed8d Mar 5 10:06:01 firewall-2 kernel: [ 2176.886525] 0xffffffff8a5cdde8 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886526] 0xffffffff8a5cd999 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886527] 0xffffffff8a5cda56 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886528] ? 0xffffffff8a0a5990 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886529] 0xffffffffc0186cd2 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886531] ? 0xffffffffc0189df5 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886532] 0xffffffffc0189df5 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886533] ? 0xffffffff8a0a5130 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886534] ? 0xffffffffc0189d20 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886535] 0xffffffff8a084495 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886536] ? 0xffffffff8a084390 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886536] 0xffffffff8a600215 Mar 5 10:06:01 firewall-2 kernel: [ 2176.886543] INFO: task dhcpd:7207 blocked for more than 120 seconds. Mar 5 10:06:01 firewall-2 kernel: [ 2176.962648] Not tainted 4.19.12-amd64-vyos #1 Mar 5 10:06:01 firewall-2 kernel: [ 2177.020988] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 5 10:06:01 firewall-2 kernel: [ 2177.114695] Call Trace: Mar 5 10:06:01 firewall-2 kernel: [ 2177.114697] ? 0xffffffff8a5cd1a0 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114698] ? 0xffffffff8a0a5832 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114698] 0xffffffff8a5cd5fd Mar 5 10:06:01 firewall-2 kernel: [ 2177.114700] 0xffffffffc01899f9 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114701] ? 0xffffffff8a0a5130 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114702] 0xffffffffc030cb4a Mar 5 10:06:01 firewall-2 kernel: [ 2177.114703] 0xffffffff8a1f5303 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114704] 0xffffffff8a1f55db Mar 5 10:06:01 firewall-2 kernel: [ 2177.114705] 0xffffffff8a003319 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114706] 0xffffffff8a600088 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114708] RIP: 0033:0x00007fea5a4a9a20 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114712] Code: Bad RIP value. Mar 5 10:06:01 firewall-2 kernel: [ 2177.114713] RSP: 002b:00007fff0ff3a888 EFLAGS: 00000246 ORIG_RAX: 000000000000004a Mar 5 10:06:01 firewall-2 kernel: [ 2177.114715] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fea5a4a9a20 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114716] RDX: 0000000000000000 RSI: 000055f737327220 RDI: 0000000000000006 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114716] RBP: 000055f735ff6ae0 R08: 00007fea5b115700 R09: 000055f735d94160 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114717] R10: 706368642d6e6f2f R11: 0000000000000246 R12: 0000000000000001 Mar 5 10:06:01 firewall-2 kernel: [ 2177.114718] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001 Mar 5 10:08:03 firewall-2 kernel: [ 2299.062410] INFO: task kworker/u16:4:165 blocked for more than 120 seconds. Mar 5 10:08:03 firewall-2 kernel: [ 2299.146122] Not tainted 4.19.12-amd64-vyos #1 Mar 5 10:08:03 firewall-2 kernel: [ 2299.204547] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 5 10:08:03 firewall-2 kernel: [ 2299.298356] Call Trace: Mar 5 10:08:03 firewall-2 kernel: [ 2299.298359] ? 0xffffffff8a5cd1a0 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298359] ? 0xffffffff8a6001a4 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298360] 0xffffffff8a5cd5fd Mar 5 10:08:03 firewall-2 kernel: [ 2299.298362] 0xffffffffc0161b24 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298364] ? 0xffffffff8a0a5130 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298365] 0xffffffffc0167409 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298366] 0xffffffffc0167bde Mar 5 10:08:03 firewall-2 kernel: [ 2299.298367] 0xffffffffc0178227 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298369] 0xffffffffc0178ebc Mar 5 10:08:03 firewall-2 kernel: [ 2299.298370] 0xffffffff8a2831cd Mar 5 10:08:03 firewall-2 kernel: [ 2299.298371] 0xffffffff8a2837a1 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298372] 0xffffffff8a1f1319 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298374] ? 0xffffffff8a1f1962 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298374] 0xffffffff8a1f1962 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298381] 0xffffffff8a07e8ea Mar 5 10:08:03 firewall-2 kernel: [ 2299.298382] 0xffffffff8a07fd42 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298389] ? 0xffffffff8a07fce0 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298389] 0xffffffff8a084495 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298395] ? 0xffffffff8a084390 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298400] 0xffffffff8a600215 Mar 5 10:08:03 firewall-2 kernel: [ 2299.298408] INFO: task md127_raid1:198 blocked for more than 120 seconds. Mar 5 10:08:04 firewall-2 kernel: [ 2299.379676] Not tainted 4.19.12-amd64-vyos #1 Mar 5 10:08:04 firewall-2 kernel: [ 2299.438018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 5 10:08:04 firewall-2 kernel: [ 2299.531830] Call Trace: Mar 5 10:08:04 firewall-2 kernel: [ 2299.531833] ? 0xffffffff8a5cd1a0 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531833] ? 0xffffffff8a6001b0 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531834] 0xffffffff8a5cd5fd Mar 5 10:08:04 firewall-2 kernel: [ 2299.531836] 0xffffffffc0161b24 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531842] ? 0xffffffff8a0a5130 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531853] 0xffffffffc0167409 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531861] ? 0xffffffffc016829b Mar 5 10:08:04 firewall-2 kernel: [ 2299.531868] 0xffffffffc0169f14 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531871] 0xffffffffc0164c4d Mar 5 10:08:04 firewall-2 kernel: [ 2299.531873] 0xffffffffc017cc4b Mar 5 10:08:04 firewall-2 kernel: [ 2299.531874] ? 0xffffffff8a0cd0a8 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531875] ? 0xffffffff8a0cd100 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531876] ? 0xffffffff8a5d177f Mar 5 10:08:04 firewall-2 kernel: [ 2299.531877] ? 0xffffffffc015a0d7 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531878] 0xffffffffc015a0d7 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531879] ? 0xffffffff8a0a5130 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531880] ? 0xffffffffc0159fb0 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531880] 0xffffffff8a084495 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531881] ? 0xffffffff8a084390 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531882] 0xffffffff8a600215 Mar 5 10:08:04 firewall-2 kernel: [ 2299.531885] INFO: task jbd2/md127-8:253 blocked for more than 120 seconds.
I managed to pull these messages from a running firewall after it had stopped working. Unfortunately i've not been able to save them out as trying to write to the disk completely freezes up whichever process attempts it.