Page MenuHomeVyOS Platform

Global log facility does not have consistent default settings, and doesn't change when modified
Needs reporter action, HighPublicBUG

Description

Today i noticed repeated messages from rsyslog using journalctl:
Mar 16 19:36:11 router rsyslogd[13285]: file size limit cmd for file '/var/log/messages' did no resolve situation [v8.2102.0]

Wondering what this meant, i went looking.
What is happening is the default size limit for /var/log/messages is 262144 bytes (which matches the docs)

$outchannel global,/var/log/messages,262144,/usr/sbin/logrotate /etc/logrotate.d/vyos-rsyslog

However, the default logrotate for this file is 1M:

vyos@router:/etc/logrotate.d$ cat vyos-rsyslog
/var/log/messages {
    create
    missingok
    nomail
    notifempty
    rotate 10
    size 1M
    postrotate
        # inform rsyslog service about rotation
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

The vyos-rsyslog.tmpl file that is rendered for logrotate isn't even actually written to support file size in kilobytes, it assumes it is in megabytes.
I went down the rabbit hole to see why this happened.

You can see the 'global' facility defaults get set in src/conf_mode/system-syslog.py, but modifying the 'global' facility only affects the rsyslog setting. Nothing you do to the archive settings on the global facility will change the logrotate config for vyos-rsyslog (it works for everything else). This is because the logrotate config for vyos-rsyslog is controlled/rendered only by system-logs.py, and controlled (apparently) by nodes in "system logs" rather than the archive settings for the global facility. This is *only* true for /var/log/messages and atop. Nothing else.

I'm not sure what the right fix is. Should system-syslog re-render the vyos-rsyslog logrotate config if 'global' has archive settings modified?
If so, should system-logs no longer control this node?

These settings have to be consistent and right now they aren't even controlled by the same overall nodes.
I assume they should also have consistent defaults, which means the tmpl file probably needs changing.

I'm happy to make a fix if someone gives me guidance on what they think should happen.

Details

Difficulty level
Unknown (require assessment)
Version
Version: VyOS 1.4-rolling-202203140317
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

My guess, looking now at commit logs, is that T4250 broke this. It looks like we attempted to move the settings to system logs, but the rsyslog part of the config still remains in system syslog, where the default does not match the new logrotate template (and the settings between rsyslog and logrotate can get out of sync).

zsdc changed the task status from Open to Confirmed.Apr 1 2022, 3:03 PM
zsdc added a subscriber: zsdc.

Hi, @dberlin ! Thanks, you are right about the root cause.
I believe that we need to remove the max-size and action-on-max-size from rsyslog.conf. So, leave everything related to rotating logs to logrotate, and to sending logs to rsyslog - UNIX-way. :)

dmbaturin added a project: VyOS 1.5 Circinus.
Viacheslav changed the task status from Confirmed to Needs reporter action.Feb 13 2024, 5:27 PM
Viacheslav added a subscriber: Viacheslav.

@dberlin Can you recheck?