Page MenuHomeVyOS Platform

Memory leak in vyos-intfwatchd
Closed, DuplicatePublic

Description

[ Re-raising this as a task, original question is here: https://phabricator.vyos.net/Q32 ]

it seems this service

https://github.com/vyos/vyatta-cfg-system/blob/b4be922411d61216aca52202ec3ee432551a4f89/scripts/vyos-intfwatchd

watches the output of the command "ip monitor link" and takes action when links come back up. The problem is after high uptime or many link state changes (in our case, openvpn interfaces that have a down endpoint, making them reset every 30 seconds), the handle in perl becomes very large. After 2 weeks of uptime on one of our vyos, it was consuming 330MB.

transcript from conversation in IRC about it:

<LesB> anyone here know where i should go to troubleshoot vyos-intfwatchd consuming a large amount of RAM
<LesB> vyos crashed a couple weeks ago, so i bumped it to 1GB, and that process is sitting at 330MB consumed
<LesB> leaps and bounds higher than any other vyos we have running (we have something like 80 in use)
<LesB> best i can tell, vyos-intfwatchd watches the output of "ip monitor link". We have about 30 openvpns, a couple of which are always down at one point or another. Seems like that may be what is bloating this script
<LesB> dmbaturin, i see you were the last contrib to that script, any insight?
<dmbaturin> LesB: Oh. Yes, that's my script, and I have to admit it's rather sloppy.
<dmbaturin> It was made as a fix for the fact that since some version, linux kernel stopped getting IPv6 addresses back after interface flaps...
<LesB> dmbaturin, my perl knowledge is lacking. Any way you know of to have the buffer of "ip monitor link" release so it's not just continuously getting bigger?
<LesB> since we don't have ipv6 on any of our vyos, i just removed execute rights on the script and stopped the service... but i'd like that to not be the perm solution
<dmbaturin> LesB: Agree. So it seems perl garbage collection is giving us troubles. Could you create a task about it in phabricator so that I don't forget about it?

Details

Difficulty level
Normal (likely a few hours)