The service config-sync can freeze the secondary router if it has commit-archive location xxx
To reproduce, there are 2 routes, main and secondary
Main router configuration:
set interfaces ethernet eth0 address '192.168.122.14/24' set service config-sync mode 'load' set service config-sync secondary address '192.168.122.11' set service config-sync secondary key 'foo' set service config-sync section 'nat' set service config-sync section 'firewall'
Secondary router configuration:
set interfaces ethernet eth0 address '192.168.122.11/24' set service https api keys id KID key 'foo' set service https api socket set service https virtual-host vsrv listen-address '192.168.122.11' set system config-management commit-archive location 'scp://vyos:[email protected]/home/vyos/' set system config-management commit-revisions '100'
Set changes in nat or firewall section on the main router:
vyos@r14# commit INFO:vyos_config_sync:Config synchronization: Mode=load, Secondary=192.168.122.11 An error occurred: HTTPSConnectionPool(host='192.168.122.11', port=443): Read timed out. (read timeout=60) ERROR:vyos_config_sync:An error occurred: HTTPSConnectionPool(host='192.168.122.11', port=443): Read timed out. (read timeout=60) An error occurred: HTTPSConnectionPool(host='192.168.122.11', port=443): Read timed out. (read timeout=60) ERROR:vyos_config_sync:An error occurred: HTTPSConnectionPool(host='192.168.122.11', port=443): Read timed out. (read timeout=60) [edit] vyos@r14#
CPU utilization is 100% for the secondary router, and he doesn't answer even for console. It stuck for several minutes.
The log file form the secondary node
Jul 10 16:36:06 r11 vyos-configd[634]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/config_mgmt.py"}
Jul 10 16:36:06 r11 vyos-configd[634]: Sending response 8
Jul 10 16:36:06 r11 sudo[2688]: pam_unix(sudo:session): session closed for user root
Jul 10 16:36:06 r11 systemd[1]: opt-vyatta-config-tmp-new_config_2582.mount: Deactivated successfully.
Jul 10 16:36:06 r11 sudo[2729]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/mv /tmp/config.boot.2722 /opt/vyatta/etc/config/archive/config.boot
Jul 10 16:36:06 r11 sudo[2729]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Jul 10 16:36:06 r11 sudo[2729]: pam_unix(sudo:session): session closed for user root
Jul 10 16:36:06 r11 sudo[2732]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/sbin/logrotate -f -s /opt/vyatta/etc/config/archive/lr.state /opt/vyatta/etc/config/archive/lr.conf
Jul 10 16:36:06 r11 sudo[2732]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Jul 10 16:36:06 r11 sudo[2732]: pam_unix(sudo:session): session closed for user root
Jul 10 16:36:12 r11 commit[2741]: Successful change to active configuration by user vyos on /dev/pts/0
Jul 10 16:36:37 r11 sudo[2774]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/config_mgmt.py'
Jul 10 16:36:37 r11 sudo[2774]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Jul 10 16:36:37 r11 vyos-configd[634]: Received message: {"type": "init"}
Jul 10 16:36:37 r11 vyos-configd[634]: config session pid is 2582
Jul 10 16:36:37 r11 vyos-configd[634]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/config_mgmt.py"}
Jul 10 16:36:37 r11 vyos-configd[634]: Sending response 8
Jul 10 16:36:37 r11 sudo[2774]: pam_unix(sudo:session): session closed for user root
Jul 10 16:36:37 r11 systemd[1]: opt-vyatta-config-tmp-new_config_2582.mount: Deactivated successfully.
Jul 10 16:36:38 r11 sudo[2815]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/mv /tmp/config.boot.2808 /opt/vyatta/etc/config/archive/config.boot
Jul 10 16:36:38 r11 sudo[2815]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Jul 10 16:36:38 r11 sudo[2815]: pam_unix(sudo:session): session closed for user root
Jul 10 16:36:38 r11 sudo[2818]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/sbin/logrotate -f -s /opt/vyatta/etc/config/archive/lr.state /opt/vyatta/etc/config/archive/lr.conf
Jul 10 16:36:38 r11 sudo[2818]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Jul 10 16:36:38 r11 sudo[2818]: pam_unix(sudo:session): session closed for user root
Jul 10 16:36:41 r11 commit[2827]: Successful change to active configuration by user vyos on /dev/pts/0
Jul 10 16:40:01 r11 CRON[2943]: pam_unix(cron:session): session opened for user smmsp(uid=116) by (uid=0)
Jul 10 16:40:01 r11 CRON[2944]: (smmsp) CMD (test -x /etc/init.d/sendmail && test -x /usr/share/sendmail/sendmail && test -x /usr/libexec/sendmail/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul 10 16:40:01 r11 CRON[2943]: pam_unix(cron:session): session closed for user smmsp
Jul 10 16:40:13 r11 sudo[3016]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/nat.py'
Jul 10 16:40:13 r11 sudo[3016]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jul 10 16:40:13 r11 vyos-configd[634]: Received message: {"type": "init"}
Jul 10 16:40:13 r11 vyos-configd[634]: config session pid is 2197
Jul 10 16:40:13 r11 vyos-configd[634]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/nat.py"}
Jul 10 16:40:13 r11 vyos-configd[634]: Sending response 1
Jul 10 16:40:13 r11 sudo[3016]: pam_unix(sudo:session): session closed for user root
Jul 10 16:40:13 r11 systemd[1]: opt-vyatta-config-tmp-new_config_2197.mount: Deactivated successfully.
Jul 10 16:40:13 r11 sudo[3059]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/mv /tmp/config.boot.3052 /opt/vyatta/etc/config/archive/config.boot
Jul 10 16:40:13 r11 sudo[3059]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jul 10 16:40:13 r11 sudo[3059]: pam_unix(sudo:session): session closed for user root
Jul 10 16:40:13 r11 sudo[3061]: root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/logrotate -f -s /opt/vyatta/etc/config/archive/lr.state /opt/vyatta/etc/config/archive/lr.conf
Jul 10 16:40:13 r11 sudo[3061]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jul 10 16:40:13 r11 sudo[3061]: pam_unix(sudo:session): session closed for user root
Jul 10 16:40:47 r11 sudo[3069]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/journalctl -f
Jul 10 16:40:47 r11 sudo[3069]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Jul 10 16:42:26 r11 ospfd[936]: [PHJDC-499N2][EC 100663314] STARVATION: task vtysh_read (7fbff75b07e0) ran for 6408ms (cpu time 316ms)
Jul 10 16:42:30 r11 sudo[3069]: pam_unix(sudo:session): session closed for user root
Jul 10 16:43:22 r11 systemd[1]: frr.service: Watchdog timeout (limit 1min)!
Jul 10 16:43:09 r11 ospfd[936]: [YZRX4-ZXG0C][EC 100663315] Thread Starvation: {(thread *)0x55eaa1499cd0 arg=0x55eaa1498aa0 timer r=-33.707 ospf_lsa_refresh_walker() &ospf->t_lsa_refresher from ../ospfd/ospf_lsa.c:4037} was scheduled to pop greater than 4s ago
Jul 10 16:43:22 r11 systemd[1]: frr.service: Killing process 862 (watchfrr) with signal SIGABRT.
Jul 10 16:43:12 r11 watchfrr[862]: [YZRX4-ZXG0C][EC 100663315] Thread Starvation: {(thread *)0x555678374790 arg=0x55567836b1e0 timer r=-40.445 wakeup_send_echo() &(dmn)->t_wakeup from ../watchfrr/watchfrr.c:705} was scheduled to pop greater than 4s ago
Jul 10 16:43:24 r11 systemd[1]: frr.service: Killing process 902 (zebra) with signal SIGABRT.
Jul 10 16:43:27 r11 BABELD[959]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7f50fc4bea71); aborting...
Jul 10 16:43:24 r11 systemd[1]: frr.service: Killing process 915 (bgpd) with signal SIGABRT.
Jul 10 16:43:27 r11 RIPNG[930]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7fa1ae8cd2e0); aborting...
Jul 10 16:43:25 r11 systemd[1]: frr.service: Killing process 923 (ripd) with signal SIGABRT.
Jul 10 16:43:25 r11 systemd[1]: frr.service: Killing process 930 (ripngd) with signal SIGABRT.
Jul 10 16:43:26 r11 systemd[1]: frr.service: Killing process 936 (ospfd) with signal SIGABRT.
Jul 10 16:43:26 r11 systemd[1]: frr.service: Killing process 939 (ospf6d) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 951 (isisd) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 959 (babeld) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 962 (ldpd) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 963 (ldpd) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 982 (ldpd) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 987 (eigrpd) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 990 (staticd) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Killing process 994 (bfdd) with signal SIGABRT.
Jul 10 16:43:27 r11 systemd[1]: frr.service: Main process exited, code=killed, status=14/ALRM
Jul 10 16:43:27 r11 BFD[994]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7f4836fe003b); aborting...
Jul 10 16:43:27 r11 STATIC[990]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7f7282ebf045); aborting...
Jul 10 16:43:27 r11 OSPF6[939]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7f02d064d190); aborting...
Jul 10 16:43:27 r11 LDP[982]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7fbf3ef22045); aborting...
Jul 10 16:43:27 r11 ISIS[951]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7f7a94147d20); aborting...
Jul 10 16:43:27 r11 EIGRP[987]: Received signal 6 at 1688996607 (si_addr 0x1, PC 0x7f24431b303b); aborting...
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7f24431c901f]
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f24431c9225]
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7f24431f6c91]
Jul 10 16:43:27 r11 EIGRP[987]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7f2442f78f90]
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(rcu_read_lock+0x4b) [0x7f24431b303b]
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_fetch+0x975) [0x7f2443208975]
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xf3) [0x7f24431c1333]
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/frr/eigrpd(main+0x105) [0x555b5d757005]
Jul 10 16:43:27 r11 EIGRP[987]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7f2442f6418a]
Jul 10 16:43:27 r11 EIGRP[987]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f2442f64245]
Jul 10 16:43:27 r11 EIGRP[987]: /usr/lib/frr/eigrpd(_start+0x21) [0x555b5d757251]
Jul 10 16:43:27 r11 EIGRP[987]: no thread information available
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7f7282ed501f]
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f7282ed5225]
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7f7282f02c91]
Jul 10 16:43:27 r11 STATIC[990]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7f7282c71f90]
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(rcu_read_lock+0x55) [0x7f7282ebf045]
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_fetch+0x975) [0x7f7282f14975]
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xf3) [0x7f7282ecd333]
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/frr/staticd(main+0xef) [0x55f25fb49adf]
Jul 10 16:43:27 r11 STATIC[990]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7f7282c5d18a]
Jul 10 16:43:27 r11 STATIC[990]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f7282c5d245]
Jul 10 16:43:27 r11 STATIC[990]: /usr/lib/frr/staticd(_start+0x21) [0x55f25fb49c51]
Jul 10 16:43:27 r11 STATIC[990]: no thread information available
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7f50fc6ab01f]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f50fc6ab225]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7f50fc6d8c91]
Jul 10 16:43:27 r11 BABELD[959]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7f50fc45af90]
Jul 10 16:43:27 r11 BABELD[959]: /lib/x86_64-linux-gnu/libc.so.6(+0x9fa71) [0x7f50fc4bea71]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x73288) [0x7f50fc67b288]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(cmd_execute+0x63) [0x7f50fc67db63]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xe7e77) [0x7f50fc6efe77]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xe8644) [0x7f50fc6f0644]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xeb898) [0x7f50fc6f3898]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f50fc6eab7d]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f50fc6a3328]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/frr/babeld(main+0x178) [0x55c9291b0ab8]
Jul 10 16:43:27 r11 BABELD[959]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7f50fc44618a]
Jul 10 16:43:27 r11 BABELD[959]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f50fc446245]
Jul 10 16:43:27 r11 BABELD[959]: /usr/lib/frr/babeld(_start+0x21) [0x55c9291b0b91]
Jul 10 16:43:27 r11 BABELD[959]: in thread vtysh_read scheduled from ../lib/vty.c:2740 vty_event()
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7fbf3ef3801f]
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7fbf3ef38225]
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7fbf3ef65c91]
Jul 10 16:43:27 r11 LDP[982]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7fbf3ecd4f90]
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(rcu_read_lock+0x55) [0x7fbf3ef22045]
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_fetch+0x975) [0x7fbf3ef77975]
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xf3) [0x7fbf3ef30333]
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/frr/ldpd(main+0x695) [0x559990c3c265]
Jul 10 16:43:27 r11 LDP[982]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7fbf3ecc018a]
Jul 10 16:43:27 r11 LDP[982]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fbf3ecc0245]
Jul 10 16:43:27 r11 LDP[982]: /usr/lib/frr/ldpd(_start+0x21) [0x559990c3c3a1]
Jul 10 16:43:27 r11 LDP[982]: no thread information available
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7f4836ff601f]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7fa1ae8a201f]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7fa1ae8a2225]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7fa1ae8cfc91]
Jul 10 16:43:27 r11 RIPNG[930]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7fa1ae651f90]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(seqlock_acquire_val+0) [0x7fa1ae8cd2e0]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(rcu_read_lock+0x55) [0x7fa1ae88c045]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_fetch+0x975) [0x7fa1ae8e1975]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xf3) [0x7fa1ae89a333]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/frr/ripngd(main+0xf8) [0x5558c6790098]
Jul 10 16:43:27 r11 RIPNG[930]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7fa1ae63d18a]
Jul 10 16:43:27 r11 RIPNG[930]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fa1ae63d245]
Jul 10 16:43:27 r11 RIPNG[930]: /usr/lib/frr/ripngd(_start+0x21) [0x5558c67901f1]
Jul 10 16:43:27 r11 RIPNG[930]: no thread information available
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f4836ff6225]
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7f4837023c91]
Jul 10 16:43:27 r11 BFD[994]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7f4836d92f90]
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(rcu_read_lock+0x4b) [0x7f4836fe003b]
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_fetch+0x975) [0x7f4837035975]
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xf3) [0x7f4836fee333]
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/frr/bfdd(main+0x2a5) [0x5623ade0d075]
Jul 10 16:43:27 r11 BFD[994]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7f4836d7e18a]
Jul 10 16:43:27 r11 BFD[994]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f4836d7e245]
Jul 10 16:43:27 r11 BFD[994]: /usr/lib/frr/bfdd(_start+0x21) [0x5623ade0d4d1]
Jul 10 16:43:27 r11 BFD[994]: no thread information available
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7f7a9410601f]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f7a94106225]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7f7a94133c91]
Jul 10 16:43:28 r11 ISIS[951]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7f7a93ea2f90]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(vector_set_index+0) [0x7f7a94147d20]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(array_to_vector+0x3a) [0x7f7a94147fea]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frrstr_split_vec+0x36) [0x7f7a940f0d16]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(cmd_make_strvec+0x62) [0x7f7a940d7382]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(cmd_execute+0xb6) [0x7f7a940d8bb6]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xe7e77) [0x7f7a9414ae77]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xe8644) [0x7f7a9414b644]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xeb898) [0x7f7a9414e898]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f7a94145b7d]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f7a940fe328]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/frr/isisd(main+0x173) [0x565425e4e233]
Jul 10 16:43:28 r11 ISIS[951]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7f7a93e8e18a]
Jul 10 16:43:28 r11 ISIS[951]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f7a93e8e245]
Jul 10 16:43:28 r11 ISIS[951]: /usr/lib/frr/isisd(_start+0x21) [0x565425e4ea11]
Jul 10 16:43:28 r11 ISIS[951]: in thread vtysh_read scheduled from ../lib/vty.c:2740 vty_event()
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7f02d068601f]
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f02d0686225]
Jul 10 16:43:28 r11 systemd[1]: frr.service: Failed with result 'watchdog'.
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xd0c91) [0x7f02d06b3c91]
Jul 10 16:43:28 r11 OSPF6[939]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7f02d0422f90]
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0x6a190) [0x7f02d064d190]
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_fetch+0x981) [0x7f02d06c5981]
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xf3) [0x7f02d067e333]
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/frr/ospf6d(main+0xcb) [0x55657a96979b]
Jul 10 16:43:28 r11 OSPF6[939]: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7f02d040e18a]
Jul 10 16:43:28 r11 OSPF6[939]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f02d040e245]
Jul 10 16:43:28 r11 OSPF6[939]: /usr/lib/frr/ospf6d(_start+0x21) [0x55657a969d61]
Jul 10 16:43:28 r11 OSPF6[939]: no thread information available
Jul 10 16:43:28 r11 systemd[1]: frr.service: Consumed 14.041s CPU time.
Jul 10 16:43:33 r11 systemd[1]: frr.service: Scheduled restart job, restart counter is at 1.
Jul 10 16:43:33 r11 systemd[1]: Stopped frr.service - FRRouting.
Jul 10 16:43:33 r11 systemd[1]: frr.service: Consumed 14.041s CPU time.
Jul 10 16:43:33 r11 systemd[1]: Starting frr.service - FRRouting...
Jul 10 16:43:33 r11 systemd[1]: Stopping vyos-http-api.service - VyOS HTTP API service...
Jul 10 16:43:34 r11 systemd[1]: [email protected]: Deactivated successfully.
Jul 10 16:43:34 r11 systemd[1]: [email protected]: Consumed 3.603s CPU time.
Jul 10 16:43:34 r11 systemd[1]: [email protected]: Scheduled restart job, restart counter is at 1.
Jul 10 16:43:34 r11 systemd[1]: Stopped [email protected] - Getty on tty1.
Jul 10 16:43:34 r11 systemd[1]: [email protected]: Consumed 3.603s CPU time.
Jul 10 16:43:34 r11 systemd[1]: Started [email protected] - Getty on tty1.
Jul 10 16:43:34 r11 systemd[1]: opt-vyatta-config-tmp-new_config_2197.mount: Deactivated successfully.
Jul 10 16:43:36 r11 kernel: systemctl invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Jul 10 16:43:37 r11 kernel: CPU: 0 PID: 3208 Comm: systemctl Not tainted 6.1.38-amd64-vyos #1
Jul 10 16:43:37 r11 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
Jul 10 16:43:37 r11 kernel: Call Trace:
Jul 10 16:43:37 r11 kernel: <TASK>
Jul 10 16:43:37 r11 kernel: dump_stack_lvl+0x44/0x5c
Jul 10 16:43:37 r11 kernel: dump_header+0x45/0x20c
Jul 10 16:43:37 r11 kernel: oom_kill_process.cold+0xb/0x10
Jul 10 16:43:37 r11 kernel: out_of_memory+0x1df/0x510
Jul 10 16:43:37 r11 kernel: __alloc_pages_slowpath.constprop.0+0xb09/0xdf0
Jul 10 16:43:37 r11 kernel: __alloc_pages+0x213/0x240
Jul 10 16:43:37 r11 kernel: folio_alloc+0x12/0x40
Jul 10 16:43:37 r11 kernel: __filemap_get_folio+0x14f/0x340
Jul 10 16:43:37 r11 kernel: filemap_fault+0x134/0x910
Jul 10 16:43:37 r11 kernel: ? filemap_map_pages+0x141/0x650
Jul 10 16:43:37 r11 kernel: __do_fault+0x2e/0x110
Jul 10 16:43:37 r11 kernel: do_fault+0x1c5/0x490
Jul 10 16:43:37 r11 kernel: __handle_mm_fault+0x611/0x12a0
Jul 10 16:43:37 r11 kernel: ? update_load_avg+0x7e/0x780
Jul 10 16:43:37 r11 kernel: handle_mm_fault+0xcc/0x2b0
Jul 10 16:43:37 r11 kernel: exc_page_fault+0x1a6/0x6f0
Jul 10 16:43:37 r11 kernel: asm_exc_page_fault+0x22/0x30
Jul 10 16:43:37 r11 kernel: RIP: 0033:0x7fe6a3e38d00
Jul 10 16:43:37 r11 kernel: Code: 39 38 0f 84 ea fd ff ff 48 89 f9 0f 1f 80 00 00 00 00 48 83 c2 01 48 39 0c d0 75 f6 e9 d1 fd ff ff 48 8b 45 00 48 85 c0 74 0b <0f> b6 40 04 c0 e8 04 3c 02 74 76 f6 05 8f 7d 02 00 01 75 6d 4d 85
Jul 10 16:43:37 r11 kernel: RSP: 002b:00007ffd0d5f9920 EFLAGS: 00010202
Jul 10 16:43:37 r11 kernel: RAX: 00007fe6a3a5c168 RBX: 00007fe6a3a5d694 RCX: 00007fe6a3a5c168
Jul 10 16:43:37 r11 kernel: RDX: 0000000000000000 RSI: 00007fe6a32d2778 RDI: 0000000000000000
Jul 10 16:43:37 r11 kernel: RBP: 00007ffd0d5f9a78 R08: 0000000002873eda R09: 0000000000000000
Jul 10 16:43:37 r11 kernel: R10: 000000000000000f R11: 00000000a1cfb68c R12: 0000000000000000
Jul 10 16:43:37 r11 kernel: R13: 00007fe6a3cd8a50 R14: 00000000a1cfb68c R15: 00007ffd0d5f9a78
Jul 10 16:43:37 r11 kernel: </TASK>
Jul 10 16:43:37 r11 kernel: Mem-Info:
Jul 10 16:43:37 r11 kernel: active_anon:274 inactive_anon:220616 isolated_anon:0
active_file:9 inactive_file:10 isolated_file:0
unevictable:2736 dirty:0 writeback:0
slab_reclaimable:3824 slab_unreclaimable:4940
mapped:950 shmem:448 pagetables:917
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:12175 free_pcp:109 free_cma:0
Jul 10 16:43:37 r11 kernel: Node 0 active_anon:1096kB inactive_anon:882464kB active_file:36kB inactive_file:40kB unevictable:10944kB isolated(anon):0kB isolated(file):0kB mapped:3800kB dirty:0kB writeback:0kB shmem:1792kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 86016kB writeback_tmp:0kB kernel_stack:2192kB pagetables:3668kB sec_pagetables:0kB all_unreclaimable? yes
Jul 10 16:43:37 r11 kernel: Node 0 DMA free:4436kB boost:0kB min:712kB low:888kB high:1064kB reserved_highatomic:0KB active_anon:0kB inactive_anon:10860kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 10 16:43:37 r11 kernel: lowmem_reserve[]: 0 933 933 933
Jul 10 16:43:37 r11 kernel: Node 0 DMA32 free:44264kB boost:0kB min:44340kB low:55424kB high:66508kB reserved_highatomic:0KB active_anon:1096kB inactive_anon:871616kB active_file:36kB inactive_file:40kB unevictable:10944kB writepending:0kB present:1032048kB managed:989260kB mlocked:10944kB bounce:0kB free_pcp:436kB local_pcp:436kB free_cma:0kB
Jul 10 16:43:37 r11 kernel: lowmem_reserve[]: 0 0 0 0
Jul 10 16:43:37 r11 kernel: Node 0 DMA: 1*4kB (M) 0*8kB 1*16kB (M) 2*32kB (U) 0*64kB 2*128kB (UM) 2*256kB (UM) 1*512kB (U) 1*1024kB (M) 1*2048kB (M) 0*4096kB = 4436kB
Jul 10 16:43:37 r11 kernel: Node 0 DMA32: 716*4kB (UE) 617*8kB (UME) 285*16kB (UE) 247*32kB (UME) 135*64kB (UME) 50*128kB (UME) 13*256kB (UME) 5*512kB (UME) 3*1024kB (M) 0*2048kB 0*4096kB = 44264kB
Jul 10 16:43:37 r11 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jul 10 16:43:37 r11 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 10 16:43:37 r11 kernel: 1418 total pagecache pages
Jul 10 16:43:37 r11 kernel: 262010 pages RAM
Jul 10 16:43:37 r11 kernel: 0 pages HighMem/MovableOnly
Jul 10 16:43:37 r11 kernel: 10855 pages reserved
Jul 10 16:43:37 r11 kernel: 0 pages cma reserved
Jul 10 16:43:37 r11 kernel: 0 pages hwpoisoned
Jul 10 16:43:37 r11 kernel: Tasks state (memory values in pages):
Jul 10 16:43:37 r11 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jul 10 16:43:37 r11 kernel: [ 625] 0 625 12375 654 94208 0 -250 systemd-journal
Jul 10 16:43:37 r11 kernel: [ 634] 0 634 79014 15222 237568 0 0 python3
Jul 10 16:43:37 r11 kernel: [ 636] 0 636 30673 4450 131072 0 0 python3
Jul 10 16:43:37 r11 kernel: [ 640] 0 640 6208 997 73728 0 -1000 systemd-udevd
Jul 10 16:43:37 r11 kernel: [ 673] 0 673 2065 1594 61440 0 0 haveged
Jul 10 16:43:37 r11 kernel: [ 694] 0 694 3280 367 53248 0 -1000 auditd
Jul 10 16:43:37 r11 kernel: [ 786] 0 786 625 223 40960 0 0 acpid
Jul 10 16:43:37 r11 kernel: [ 793] 0 793 1652 460 53248 0 0 cron
Jul 10 16:43:37 r11 kernel: [ 794] 103 794 2010 560 53248 0 -900 dbus-daemon
Jul 10 16:43:37 r11 kernel: [ 800] 0 800 4164 533 69632 0 0 systemd-logind
Jul 10 16:43:37 r11 kernel: [ 811] 0 811 895 213 49152 0 0 atd
Jul 10 16:43:37 r11 kernel: [ 825] 0 825 20058 515 53248 0 0 qemu-ga
Jul 10 16:43:37 r11 kernel: [ 833] 0 833 618 307 45056 0 0 atopacctd
Jul 10 16:43:37 r11 kernel: [ 835] 0 835 2959 2738 61440 0 -1000 atop
Jul 10 16:43:37 r11 kernel: [ 1060] 0 1060 620 277 40960 0 0 netplugd
Jul 10 16:43:37 r11 kernel: [ 1921] 0 1921 55444 1167 86016 0 0 rsyslogd
Jul 10 16:43:37 r11 kernel: [ 2071] 113 2071 4722 547 61440 0 0 chronyd
Jul 10 16:43:37 r11 kernel: [ 2072] 113 2072 2640 506 61440 0 0 chronyd
Jul 10 16:43:37 r11 kernel: [ 2119] 0 2119 3878 746 69632 0 -1000 sshd
Jul 10 16:43:37 r11 kernel: [ 2197] 0 2197 386697 192280 1662976 0 0 vyos-http-api-s
Jul 10 16:43:37 r11 kernel: [ 2236] 0 2236 2663 329 57344 0 0 nginx
Jul 10 16:43:37 r11 kernel: [ 2237] 33 2237 2776 510 61440 0 0 nginx
Jul 10 16:43:37 r11 kernel: [ 2303] 0 2303 1374 155 49152 0 0 agetty
Jul 10 16:43:37 r11 kernel: [ 2312] 0 2312 4400 961 73728 0 0 sshd
Jul 10 16:43:37 r11 kernel: [ 2315] 1002 2315 4702 856 77824 0 100 systemd
Jul 10 16:43:37 r11 kernel: [ 2316] 1002 2316 26086 1139 86016 0 100 (sd-pam)
Jul 10 16:43:37 r11 kernel: [ 2397] 1002 2397 4465 690 69632 0 0 sshd
Jul 10 16:43:37 r11 kernel: [ 2398] 1002 2398 1974 860 49152 0 0 vbash
Jul 10 16:43:37 r11 kernel: [ 2582] 1002 2582 2134 618 57344 0 0 vbash
Jul 10 16:43:37 r11 kernel: [ 2801] 1002 2801 74613 104 73728 0 0 unionfs-fuse
Jul 10 16:43:37 r11 kernel: [ 3200] 0 3200 25893 1107 86016 0 0 (s-config)
Jul 10 16:43:37 r11 kernel: [ 3204] 0 3204 1799 571 53248 0 0 frrinit.sh
Jul 10 16:43:37 r11 kernel: [ 3207] 0 3207 1799 312 53248 0 0 frrinit.sh
Jul 10 16:43:37 r11 kernel: [ 3208] 0 3208 3333 197 65536 0 0 systemctl
Jul 10 16:43:37 r11 kernel: [ 3209] 0 3209 6209 838 61440 0 0 (udev-worker)
Jul 10 16:43:37 r11 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=frr.service,mems_allowed=0,global_oom,task_memcg=/system.slice/vyos-http-api.service,task=vyos-http-api-s,pid=2197,uid=0
Jul 10 16:43:37 r11 kernel: Out of memory: Killed process 2197 (vyos-http-api-s) total-vm:1546788kB, anon-rss:766960kB, file-rss:2160kB, shmem-rss:0kB, UID:0 pgtables:1624kB oom_score_adj:0
Jul 10 16:43:37 r11 kernel: oom_reaper: reaped process 2197 (vyos-http-api-s), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 10 16:43:37 r11 systemd[1]: vyos-http-api.service: A process of this unit has been killed by the OOM killer.
Jul 10 16:43:37 r11 systemd[1]: vyos-http-api.service: Killing process 2197 (vyos-http-api-s) with signal SIGKILL.
Jul 10 16:43:37 r11 systemd[1]: vyos-http-api.service: Main process exited, code=killed, status=9/KILL
Jul 10 16:43:37 r11 systemd[1]: vyos-http-api.service: Failed with result 'oom-kill'.
Jul 10 16:43:37 r11 systemd[1]: Stopped vyos-http-api.service - VyOS HTTP API service.
Jul 10 16:43:37 r11 systemd[1]: vyos-http-api.service: Consumed 2min 13.966s CPU time.
Jul 10 16:43:37 r11 systemd[1]: Stopping vyos-router.service - VyOS Router...
Jul 10 16:43:37 r11 frrinit.sh[3204]: Starting watchfrr with command: ' /usr/lib/frr/watchfrr -d -F traditional zebra bgpd ripd ripngd ospfd ospf6d isisd babeld ldpd eigrpd staticd bfdd'.
Jul 10 16:43:37 r11 vyos-router[3210]: Stopping VyOS router:.
Jul 10 16:43:37 r11 vyos-router[3210]: Un-mounting VyOS Config...
Jul 10 16:43:37 r11 vyos-router[3226]: umount: /opt/vyatta/config: target is busy.
Jul 10 16:43:37 r11 vyos-router[3210]: failed.
Jul 10 16:43:37 r11 watchfrr[3228]: [T83RR-8SM5G] watchfrr 8.5.2 starting: vty@0
Jul 10 16:43:37 r11 watchfrr[3228]: [NG1AJ-FP2TQ] Terminating on signal
Jul 10 16:43:37 r11 systemd[1]: frr.service: Deactivated successfully.
Jul 10 16:43:37 r11 systemd[1]: Stopped frr.service - FRRouting.
Jul 10 16:43:37 r11 systemd[1]: vyos-router.service: Deactivated successfully.
Jul 10 16:43:37 r11 systemd[1]: Stopped vyos-router.service - VyOS Router.
Jul 10 16:43:37 r11 vyos-config[3200]: Configuration success
Jul 10 16:47:50 r11 systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Jul 10 16:47:50 r11 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Jul 10 16:47:50 r11 systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Jul 10 16:47:50 r11 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Jul 10 16:53:41 r11 sudo[3305]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/journalctl -f
Jul 10 16:53:41 r11 sudo[3305]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Jul 10 16:53:47 r11 sudo[3305]: pam_unix(sudo:session): session closed for user root
Jul 10 16:53:53 r11 sudo[3309]: vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/journalctl
Jul 10 16:53:53 r11 sudo[3309]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
[edit]
vyos@r11#