Page MenuHomePhabricator

Some logs are not forwarded to syslog
Closed, ResolvedPublicBUG

Description

Our "run show log" command uses syslog for historical reasons, and we cannot avoid using syslog because all software for collecting logs from remote machines uses it.
Since we are using systemd now, all logs go to journald, which is supposed to forward the messages to syslog. Except sometimes it doesn't.

For example:

dmbaturin@vyos# sudo journalctl -u keepalived
-- Logs begin at Sat 2018-10-13 04:23:10 CEST, end at Sat 2018-10-13 18:33:53 CEST. --
Oct 13 18:32:05 vyos Keepalived_vrrp[3944]: Printing VRRP as json for process(3944) on signal
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: Netlink reports eth1 down
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: (Foo) Entering FAULT STATE
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: (Foo) sent 0 priority
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: VRRP_Group(Bar) Syncing instances to FAULT state
Oct 13 18:32:24 vyos conntrack-tools[20934]: vyatta-vrrp-conntracksync invoked at Sat Oct 13 18:32:24 CEST 2018
Oct 13 18:32:24 vyos conntrack-tools[20937]: vyos transitioning to FAULT state for VRRP sync-group [Bar]
Oct 13 18:32:25 vyos vyos-vrrp-wrapper[20931]: Running transition script /config/scripts/backup.sh testtesttest for VRRP group Foo
Oct 13 18:32:25 vyos vyos-vrrp-wrapper[20931]: Transition script /config/scripts/backup.sh testtesttest executed successfully

...but all we get in syslog is:

Oct 13 18:32:24 vyos conntrack-tools: vyos transitioning to FAULT state for VRRP sync-group [Bar]
Oct 13 18:32:25 vyos vyos-vrrp-wrapper: Running transition script /config/scripts/backup.sh testtesttest for VRRP group Foo

Details

Difficulty level
Unknown (require assessment)
Version
1.2.0-rc1
Why the issue appeared?
Will be filled on close
dmbaturin triaged this task as High priority.
dmbaturin created this object with visibility "Public (No Login Required)".

so far found this:

echo "working" | systemd-cat -t keepalived -p warning
echo "missinginaction" | systemd-cat -t keepalived -p info
EwaldvanGeffen added a comment.EditedOct 14 2018, 12:48 PM

changed

root@vyos:~# cat /etc/rsyslog.d/vyos-rsyslog.conf
[..]
*.debug :omfile:$global
root@vyos:~# grep ^# /etc/systemd/journald.conf  -v

[Journal]
ForwardToSyslog=yes
MaxLevelStore=debug
MaxLevelSyslog=debug

MaxLevelStore because of bug 1370195 but should already be Debug by default. Might not be necessary.

to get it working. But then I thought, isn't this

set system syslog global facility all level notice

fault? ie. PEBCAK

Maybe it merits the larger question on howto migrate this away from rsyslog (if at all) and create sub-tasks.

syncer changed the subtype of this task from "Task" to "Bug".Oct 18 2018, 5:40 AM
syncer reassigned this task from dmbaturin to hagbard.Tue, Oct 23, 1:02 PM

It's not really a bug, the log level is per default set not to log everything, like @EwaldvanGeffen stated above.
'set system syslog global facility all level all' will log everything, embedded users with flash drives will hate us for this if we change it :).
In my opinion, if someone needs to debug anything, he/she can just set the global logging or even to a particular file for debugging.
Otherwise, we can just change the section in the default config to have logging anything enabled per default, which increases the risk for losing information due to disk space limitation for /var/log as well as the quicker rotation (only 5 files are being stored per default).

I'm going to verify @dmbaturin patch to set level to info. Additionally we could also use immark, which leave a 'heartbeat' marker in the logfiles, so one can verify logging at least works. It would be visible to remote logging as well.

hagbard closed this task as Resolved.Wed, Oct 24, 4:31 PM