Page MenuHomeVyOS Platform

Suppress unsupported interfaces from appearing in messages log by Telegraf
Closed, ResolvedPublicBUG

Description

From https://forum.vyos.io/t/suppress-unsupported-interfaces-from-being-reported-on-by-telegraf/13940

Issue:
Unsupported interfaces (pppoe, tunnel, wireguard) are picked up by Telegraf and appearing /var/log/messages with very frequent occurrence in log file.

Mar  4 21:36:45 <hostname> telegraf[131801]: 2024-03-05T02:36:45Z E! [inputs.ethtool] Error in plugin: "pim6reg" driver: operation not supported
Mar  4 21:36:45 <hostname> telegraf[131801]: 2024-03-05T02:36:45Z E! [inputs.ethtool] Error in plugin: "tun0" stats: operation not supported
Mar  4 21:36:45 <hostname> telegraf[131801]: 2024-03-05T02:36:45Z E! [inputs.ethtool] Error in plugin: "sit0" stats: operation not supported
Mar  4 21:36:45 <hostname> telegraf[131801]: 2024-03-05T02:36:45Z E! [inputs.ethtool] Error in plugin: "wg01" stats: operation not supported
Mar  4 21:36:45 <hostname> telegraf[131801]: 2024-03-05T02:36:45Z E! [inputs.ethtool] Error in plugin: "pppoe0" stats: operation not supported
Mar  4 21:37:03 <hostname> telegraf[131801]: 2024-03-05T02:37:03Z E! [inputs.ethtool] Error in plugin: "pim6reg" driver: operation not supported
Mar  4 21:37:03 <hostname> telegraf[131801]: 2024-03-05T02:37:03Z E! [inputs.ethtool] Error in plugin: "tun0" stats: operation not supported
Mar  4 21:37:03 <hostname> telegraf[131801]: 2024-03-05T02:37:03Z E! [inputs.ethtool] Error in plugin: "wg01" stats: operation not supported
Mar  4 21:37:03 <hostname> telegraf[131801]: 2024-03-05T02:37:03Z E! [inputs.ethtool] Error in plugin: "pppoe0" stats: operation not supported
Mar  4 21:37:03 <hostname> telegraf[131801]: 2024-03-05T02:37:03Z E! [inputs.ethtool] Error in plugin: "sit0" stats: operation not supported

Cause:
Script “/etc/telegraf/custom_scripts/show_interfaces_input_filter.py” at bottom of /run/telegraf/telegraf.conf includes these problematic interfaces.

# Generated by /usr/libexec/vyos/conf_mode/service_monitoring_telegraf.py


InfluxDB2
[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  insecure_skip_verify = true
  token = "$INFLUX_TOKEN"
  organization = "vyos"
  bucket = "bucket_vyos"
End InfluxDB2
...
...
[[inputs.ethtool]]
  interface_include = ['eth0', 'eth1', 'eth2']
...
...
[[inputs.exec]]
  commands = [
    "/etc/telegraf/custom_scripts/show_firewall_input_filter.py",
    "/etc/telegraf/custom_scripts/show_interfaces_input_filter.py",
    "/etc/telegraf/custom_scripts/vyos_services_input_filter.py"
  ]
...
user@<hostname>:~$ python3 /etc/telegraf/custom_scripts/show_interfaces_input_filter.py
show_interfaces,interface=eth0 ip_addresses="xxx.xxx.xxx.xxx/24",state=0i,link=0i,description="WAN" 1709607132000000000
show_interfaces,interface=eth1 ip_addresses="xxx.xxx.xxx.xxx/24",state=0i,link=0i,description="LAN" 1709607132000000000
show_interfaces,interface=eth2 ip_addresses="-",state=0i,link=1i,description="empty" 1709607132000000000
show_interfaces,interface=lo ip_addresses="127.0.0.1/8 ::1/128",state=0i,link=0i,description="empty" 1709607132000000000
show_interfaces,interface=pppoe0 ip_addresses="xxx.xxx.xxx.xxx/32",state=0i,link=0i,description="bell-fibe" 1709607132000000000
show_interfaces,interface=tun0 ip_addresses="2XXX:XXX:XX:XXX::X/64",state=2i,link=1i,description="HE.NET IPv6 Tunnel" 1709607132000000000
show_interfaces,interface=wg01 ip_addresses="XX.XX.XX.XX/24",state=0i,link=0i,description="remote admin" 1709607132000000000
user@<hostname>:~$

Possible Fix:
Either:

Quick and dirty: delete line “/etc/telegraf/custom_scripts/show_interfaces_input_filter.py” from /run/telegraf/telegraf.conf since eth0/1/2 interfaces already mentioned in config file making python script not needed (I’m not sure what the purpose of including that python script was for but if removing it cause needed interfaces from being suppressed then propose solution 2)
-or-
More elegant: modify show_interfaces_input_filter.py to exclude interfaces that aren’t supported by Telegraf (I suspect anything that isn’t an ethX will probably give error)

Details

Difficulty level
Unknown (require assessment)
Version
1.4.0-epa1
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Giggum updated the task description. (Show Details)
Giggum renamed this task from Suppress unsupported interfaces from being reported on by Telegraf to Suppress unsupported interfaces from appearing in messages by Telegraf .Mar 5 2024, 3:33 AM
Giggum renamed this task from Suppress unsupported interfaces from appearing in messages by Telegraf to Suppress unsupported interfaces from appearing in messages log by Telegraf .

@Giggum Can you check it in 1.5?

Yeah sure thing I can do that. Will I be able to roll back from the latest 1.5 to the version of 1.4 rolling I’m on after testing is complete or will the config mess up?

@Giggum yes you can choose which image will be booted

vyos@r4:~$ set system image default-boot 
The following images are available:
	1: 1.5-rolling-202404120636 (running) (default boot)
	2: 1.5-rolling-202404090019
Select an image to set as default:

I don't see those logs:

set service monitoring telegraf influxdb authentication organization 'vyos'
set service monitoring telegraf influxdb authentication token 'lxxx='
set service monitoring telegraf influxdb bucket 'vyos'
set service monitoring telegraf influxdb url 'http://192.168.122.14'

logs:

Apr 14 16:27:24 r4 sudo[56200]:     vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/service_monitoring_telegraf.py'
Apr 14 16:27:24 r4 sudo[56200]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1003)
Apr 14 16:27:25 r4 systemd[1]: Reloading.
Apr 14 16:27:25 r4 systemd[1]: Started telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB.
Apr 14 16:27:25 r4 systemd[1]: Stopping rsyslog.service - System Logging Service...
Apr 14 16:27:25 r4 rsyslogd[51027]: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="51027" x-info="https://www.rsyslog.com"] exiting on signal 15.
Apr 14 16:27:25 r4 systemd[1]: rsyslog.service: Deactivated successfully.
Apr 14 16:27:25 r4 systemd[1]: Stopped rsyslog.service - System Logging Service.
Apr 14 16:27:25 r4 systemd[1]: Starting rsyslog.service - System Logging Service...
Apr 14 16:27:25 r4 rsyslogd[56249]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2302.0]
Apr 14 16:27:25 r4 rsyslogd[56249]: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="56249" x-info="https://www.rsyslog.com"] start
Apr 14 16:27:25 r4 systemd[1]: Started rsyslog.service - System Logging Service.
Apr 14 16:27:25 r4 sudo[56200]: pam_unix(sudo:session): session closed for user root
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Loading config: /run/telegraf/telegraf.conf
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Starting Telegraf 1.28.3 brought to you by InfluxData the makers of InfluxDB
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Loaded inputs: chrony conntrack cpu disk diskio ethtool exec internal interrupts kernel linux_sysctl_fs mem net netstat nstat (2x) processes syslog system systemd_units
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Loaded aggregators:
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Loaded processors:
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Loaded secretstores:
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Loaded outputs: influxdb_v2
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! Tags enabled: host=r4
Apr 14 16:27:25 r4 telegraf[56242]: 2024-04-14T13:27:25Z I! [agent] Config: Interval:15s, Quiet:false, Hostname:"r4", Flush Interval:15s
Apr 14 16:27:25 r4 systemd[1]: opt-vyatta-config-tmp-new_config_5747.mount: Deactivated successfully.
Apr 14 16:27:27 r4 sudo[56293]:     vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/mv /tmp/config.boot.56285 /opt/vyatta/etc/config/archive/config.boot
Apr 14 16:27:27 r4 sudo[56293]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1003)
Apr 14 16:27:27 r4 sudo[56293]: pam_unix(sudo:session): session closed for user root
Apr 14 16:27:27 r4 sudo[56296]:     vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/sbin/logrotate -f -s /opt/vyatta/etc/config/archive/lr.state /opt/vyatta/etc/config/archive/lr.conf
Apr 14 16:27:27 r4 sudo[56296]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1003)
Apr 14 16:27:27 r4 sudo[56296]: pam_unix(sudo:session): session closed for user root
Apr 14 16:27:27 r4 commit[56300]: Successful change to active configuration by user vyos on /dev/pts/0

I see the correct interfaces:

vyos@r4# cat /run/telegraf/telegraf.conf | grep eth
[[inputs.ethtool]]
  interface_include = ['eth0', 'eth1', 'eth2', 'eth3', 'eth4', 'eth5']
[edit]
vyos@r4#

Script gets correct values:

vyos@r4# python3 /etc/telegraf/custom_scripts/show_interfaces_input_filter.py
show_interfaces,interface=dum0 ip_addresses="-",state=0i,link=0i,description="empty" 1713101640000000000
show_interfaces,interface=eth0 ip_addresses="192.168.122.14/24",state=0i,link=0i,description="WAN" 1713101640000000000
show_interfaces,interface=eth1 ip_addresses="192.0.2.1/24",state=0i,link=0i,description="WAN" 1713101641000000000
show_interfaces,interface=eth2 ip_addresses="-",state=0i,link=0i,description="sync" 1713101641000000000
show_interfaces,interface=eth2.100 ip_addresses="-",state=0i,link=0i,description="empty" 1713101641000000000
show_interfaces,interface=eth2.100.200 ip_addresses="10.0.1.1/30",state=0i,link=0i,description="empty" 1713101641000000000
show_interfaces,interface=eth3 ip_addresses="-",state=2i,link=1i,description="empty" 1713101641000000000
show_interfaces,interface=eth4 ip_addresses="-",state=2i,link=1i,description="empty" 1713101641000000000
show_interfaces,interface=eth5 ip_addresses="-",state=2i,link=1i,description="empty" 1713101641000000000
show_interfaces,interface=lo ip_addresses="127.0.0.1/8 ::1/128",state=0i,link=0i,description="empty" 1713101641000000000
show_interfaces,interface=vrrp.10 ip_addresses="10.0.1.1/24 10.0.1.254/32",state=0i,link=0i,description="empty" 1713101641000000000
show_interfaces,interface=vti10 ip_addresses="-",state=2i,link=1i,description="empty" 1713101641000000000
[edit]
vyos@r4#

Checked on VyOS 1.5-rolling-202404140022 and VyOS 1.4-stable-202404120309
Re-check on a more recent version, close the task or describe more steps that need to be done to get it.

Viacheslav changed the task status from Open to Needs reporter action.Sun, Apr 14, 1:38 PM

@Viacheslav concur that it looks to be resolved. The last log entry was at 9:10 and nothing telegraph-related has been logged in almost 30 minutes since. I will close ticket. Thank your help and insight.

$ show version
Version: VyOS 1.5-rolling-202404141045
Release train: current

Built by: [email protected]
Built on: Sun 14 Apr 2024 13:12 UTC
Build UUID: de5e230f-fde2-4998-a0c4-6d86fe65b522
Build commit ID: eeb1a98a776db5

$ cat /var/log/messages | grep "telegraf"
Apr 16 09:10:28 hostname vyos-configd[822]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/service_monitoring_telegraf.py"}
Apr 16 09:10:28 hostname systemd[1]: Started telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB.
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Loading config: /run/telegraf/telegraf.conf
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Starting Telegraf 1.28.3 brought to you by InfluxData the makers of InfluxDB
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Loaded inputs: chrony conntrack cpu disk diskio ethtool exec internal interrupts kernel linux_sysctl_fs mem net netstat nstat (2x) processes syslog system systemd_units
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Loaded aggregators:
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Loaded processors:
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Loaded secretstores:
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Loaded outputs: influxdb_v2
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! Tags enabled: host=hostname
Apr 16 09:10:29 hostname telegraf[258392]: 2024-04-16T13:10:29Z I! [agent] Config: Interval:15s, Quiet:false, Hostname:"hostname", Flush Interval:15s

$ cat /run/telegraf/telegraf.conf | grep eth
inputs.ethtool

interface_include = ['eth0', 'eth1', 'eth2']