Page MenuHomeVyOS Platform
Create Task
Maniphest T895

Some logs are not forwarded to syslog
Closed, ResolvedPublicBUG
Actions

Description

Our "run show log" command uses syslog for historical reasons, and we cannot avoid using syslog because all software for collecting logs from remote machines uses it.
Since we are using systemd now, all logs go to journald, which is supposed to forward the messages to syslog. Except sometimes it doesn't.

For example:

dmbaturin@vyos# sudo journalctl -u keepalived
-- Logs begin at Sat 2018-10-13 04:23:10 CEST, end at Sat 2018-10-13 18:33:53 CEST. --
Oct 13 18:32:05 vyos Keepalived_vrrp[3944]: Printing VRRP as json for process(3944) on signal
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: Netlink reports eth1 down
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: (Foo) Entering FAULT STATE
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: (Foo) sent 0 priority
Oct 13 18:32:24 vyos Keepalived_vrrp[3944]: VRRP_Group(Bar) Syncing instances to FAULT state
Oct 13 18:32:24 vyos conntrack-tools[20934]: vyatta-vrrp-conntracksync invoked at Sat Oct 13 18:32:24 CEST 2018
Oct 13 18:32:24 vyos conntrack-tools[20937]: vyos transitioning to FAULT state for VRRP sync-group [Bar]
Oct 13 18:32:25 vyos vyos-vrrp-wrapper[20931]: Running transition script /config/scripts/backup.sh testtesttest for VRRP group Foo
Oct 13 18:32:25 vyos vyos-vrrp-wrapper[20931]: Transition script /config/scripts/backup.sh testtesttest executed successfully

...but all we get in syslog is:

Oct 13 18:32:24 vyos conntrack-tools: vyos transitioning to FAULT state for VRRP sync-group [Bar]
Oct 13 18:32:25 vyos vyos-vrrp-wrapper: Running transition script /config/scripts/backup.sh testtesttest for VRRP group Foo

Details

Difficulty level
Unknown (require assessment)
Version
1.2.0-rc1
Why the issue appeared?
Will be filled on close

Related Objects

Event Timeline

dmbaturin triaged this task as High priority.Oct 13 2018, 4:45 PM
dmbaturin created this task.
dmbaturin created this object with visibility "Public (No Login Required)".
syncer merged a task: T804: snmp error and no firewall log.Oct 13 2018, 7:12 PM
syncer added subscribers: lawrencepan, Maintainers.
EwaldvanGeffen added a subscriber: EwaldvanGeffen.Oct 14 2018, 12:26 PM

so far found this:

echo "working" | systemd-cat -t keepalived -p warning
echo "missinginaction" | systemd-cat -t keepalived -p info
EwaldvanGeffen added a comment.EditedOct 14 2018, 12:48 PM

changed

root@vyos:~# cat /etc/rsyslog.d/vyos-rsyslog.conf
[..]
*.debug :omfile:$global
root@vyos:~# grep ^# /etc/systemd/journald.conf  -v

[Journal]
ForwardToSyslog=yes
MaxLevelStore=debug
MaxLevelSyslog=debug

MaxLevelStore because of bug 1370195 but should already be Debug by default. Might not be necessary.

to get it working. But then I thought, isn't this

set system syslog global facility all level notice

fault? ie. PEBCAK

dmbaturin merged a task: T877: FRR not logging to syslog.Oct 14 2018, 1:44 PM
dmbaturin added a subscriber: c-po.
dmbaturin moved this task from Needs Triage to In Progress on the VyOS 1.2 Crux (VyOS 1.2.0-rc3) board.Oct 14 2018, 8:38 PM
EwaldvanGeffen added a comment.Oct 15 2018, 12:15 AM

Maybe it merits the larger question on howto migrate this away from rsyslog (if at all) and create sub-tasks.

pasik added a subscriber: pasik.Oct 15 2018, 9:52 PM
syncer edited projects, added VyOS 1.2 Crux (VyOS 1.2.0-rc4); removed VyOS 1.2 Crux (VyOS 1.2.0-rc3).Oct 15 2018, 10:15 PM
syncer moved this task from Needs Triage to In Progress on the VyOS 1.2 Crux (VyOS 1.2.0-rc4) board.
syncer changed the subtype of this task from "Task" to "Bug".Oct 18 2018, 5:40 AM
syncer reassigned this task from dmbaturin to hagbard.Oct 23 2018, 1:02 PM
hagbard added a comment.Oct 23 2018, 4:45 PM

It's not really a bug, the log level is per default set not to log everything, like @EwaldvanGeffen stated above.
'set system syslog global facility all level all' will log everything, embedded users with flash drives will hate us for this if we change it :).
In my opinion, if someone needs to debug anything, he/she can just set the global logging or even to a particular file for debugging.
Otherwise, we can just change the section in the default config to have logging anything enabled per default, which increases the risk for losing information due to disk space limitation for /var/log as well as the quicker rotation (only 5 files are being stored per default).

hagbard added a comment.Oct 24 2018, 4:23 PM

I'm going to verify @dmbaturin patch to set level to info. Additionally we could also use immark, which leave a 'heartbeat' marker in the logfiles, so one can verify logging at least works. It would be visible to remote logging as well.

hagbard closed this task as Resolved.Oct 24 2018, 4:31 PM
syncer edited projects, added VyOS 1.2 Crux (VyOS 1.2.0-rc5), VyOS-1.2.0-GA; removed VyOS 1.2 Crux (VyOS 1.2.0-rc4).Oct 24 2018, 4:37 PM
syncer moved this task from Needs Triage to Finished on the VyOS 1.2 Crux (VyOS 1.2.0-rc5) board.
syncer edited projects, added VyOS 1.2 Crux (VyOS 1.2.0-rc4); removed VyOS 1.2 Crux (VyOS 1.2.0-rc5).Oct 24 2018, 4:42 PM
syncer moved this task from In Progress to Finished on the VyOS 1.2 Crux (VyOS 1.2.0-rc4) board.