Page MenuHomeVyOS Platform

Removing serial console port from ESXi VM causes flooded syslog
Closed, InvalidPublicBUG

Description

I have deployed a fresh ISO installation into ESXi without a serial interface connected to the VM.

Removing the system console node from the CLI and even after a reboot did not fix the issue. My syslog is spammed.

Dec 12 19:27:32 VMU-02-AZURE systemd[1]: [email protected] holdoff time over, scheduling restart.
Dec 12 19:27:32 VMU-02-AZURE systemd[1]: Stopping Serial Getty on ttyS0...
Dec 12 19:27:32 VMU-02-AZURE systemd[1]: Starting Serial Getty on ttyS0...
Dec 12 19:27:32 VMU-02-AZURE systemd[1]: Started Serial Getty on ttyS0.
Dec 12 19:27:32 VMU-02-AZURE agetty[3580]: /dev/ttyS0: not a tty
Dec 12 19:27:43 VMU-02-AZURE systemd[1]: [email protected] holdoff time over, scheduling restart.
Dec 12 19:27:43 VMU-02-AZURE systemd[1]: Stopping Serial Getty on ttyS0...
Dec 12 19:27:43 VMU-02-AZURE systemd[1]: Starting Serial Getty on ttyS0...
Dec 12 19:27:43 VMU-02-AZURE systemd[1]: Started Serial Getty on ttyS0.
Dec 12 19:27:43 VMU-02-AZURE agetty[3617]: /dev/ttyS0: not a tty
Dec 12 19:27:53 VMU-02-AZURE systemd[1]: [email protected] holdoff time over, scheduling restart.
Dec 12 19:27:53 VMU-02-AZURE systemd[1]: Stopping Serial Getty on ttyS0...
Dec 12 19:27:53 VMU-02-AZURE systemd[1]: Starting Serial Getty on ttyS0...
Dec 12 19:27:53 VMU-02-AZURE systemd[1]: Started Serial Getty on ttyS0.
Dec 12 19:27:53 VMU-02-AZURE agetty[3624]: /dev/ttyS0: not a tty

Rebooting the system has no effect. It is only fixed after deleting the service by hand and rebooting the system again.

$ sudo systemctl disable serial-getty@ttyS0
Removed symlink /etc/systemd/system/getty.target.wants/[email protected].
$ reboot

Details

Difficulty level
Unknown (require assessment)
Version
1.2.3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

I'm experiencing the same issue of the service failing to start on 1.3.
The installation was first started with the default config in a VM that had a serial port. Then the installation was transferred to a physical machine without a serial port, and the whole /config directory was manually copied from the old installation on that machine. The machine was then rebooted. The result were the same errors in syslog/journal.
I believe the issue is that if the config.boot is manually replaced or edited on disk, the script that would normally be triggered on commit when deleting system console is never triggered, thus the service remains enabled, but there is no system console in the config to delete any more.

My proposal for a fix would be to run service setup scripts after each successful config load in the bootup process. These scripts would check the desired service state in the config, check the current enabled/running status via systemctl, then enable/disable and start/stop the service as necessary. The services would remain enabled/disabled across reboots, so even if the startup config load failed, the ssh/console services would still be started via systemd so that the failed load can be fixed.
This would be required foe all services that are started via systemd and are necessary for system access, e.g. ssh, system console (getty). Services that aren't strictly necessary for console access or are started in the config load process wouldn't need to have these scripts. They should be started later in the boot process when their configuration is created (at config load time).

hagbard renamed this task from Removing serial console port from ESXi VM causes flodded syslog to Removing serial console port from ESXi VM causes flooded syslog.Dec 12 2019, 7:11 PM
syncer assigned this task to Unknown Object (User).Jan 1 2020, 1:54 PM
syncer triaged this task as Normal priority.
syncer edited projects, added VyOS 1.3 Equuleus; removed VyOS 1.2 Crux.
Unknown Object (User) added a comment.Jan 2 2020, 6:40 PM

@c-po, it not possible to reproduce in 1.2.4 and 1.3 latest rolling.
I tried also delete system console on deployed 1.2.3, and it also works without issues, syslog clear.

There's a regression here, I can not 'delete system console' in the current rolling build, I get this:

vyos@voip-tunnel-1# delete system console
[edit]
vyos@voip-tunnel-1# commit
[ system console ]
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Make sure you are running the latest version of the code available at
  https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso
- Consult the forum to see how to handle this issue
  https://forum.vyos.io
- Join our community on slack where our users exchange help and advice
  https://vyos.slack.com

When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your
  business policy requires it)
- and include all the information presented below

Report Time:      2020-06-14 08:51:30
Image Version:    VyOS 1.3-rolling-202006120643
Release Train:    equuleus

Built by:         [email protected]
Built on:         Fri 12 Jun 2020 06:43 UTC
Build UUID:       d0f3a48d-a9ba-4aa8-89fc-f7e6ef826ea3
Build Commit ID:  411d48353967f2

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-42 27 ab a0 1e f5 28 c8-7a 79 ae 02 5f 7b 20 cd
Hardware UUID:    a0ab2742-f51e-c828-7a79-ae025f7b20cd

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/system_console.py", line 130, in <module>
    generate(c)
  File "/usr/libexec/vyos/conf_mode/system_console.py", line 77, in generate
    for device in console['device'].keys():
KeyError: 'device'



delete [ system console ] failed
Commit failed
[edit]

T2569 did an entire rewrite of the console command in XML / Pyhton. Bugfix for this problem is also tracked in that task.

Any rolling ISO generated after vyos-1.3-rolling-202006140117-amd64.iso will have the fix.

Confirmed fixed, thanks @c-po!

vyos@voip-tunnel-1# delete system console
[edit]
vyos@voip-tunnel-1# commit
[edit]
vyos@voip-tunnel-1# save
Saving configuration to '/config/config.boot'...
Done
[edit]
vyos@voip-tunnel-1# exit
exit
vyos@voip-tunnel-1:~$
erkin set Issue type to Bug (incorrect behavior).Aug 31 2021, 6:07 PM