So for a while I've been thinking there is a memory leak in VyOS somewhere, but I was unable to exactly pinpoint where.
It turns out the more routes you have in your routing table, the more memory each SSH login uses. Logging in too many times, puts VyOS OOM.
In my case, it resulted in about 200MB per login. I tested on a server with a few copies of full tables, and was able to make it take up to a gig per ssh login.
In the following screenshot, all I did was ssh in 4 times.
On logout of the other three, all the memory is instantly returned
When I shut down my large BGP feeds, ssh logins return to taking about 2MB/login
Since that's not a very good solution, I discovered that it's related to this option:
set service ssh disable-host-validation
With this option NOT present, with ONE SSH session, this is my memory usage:
admin@edge:~$ free -m total used free shared buff/cache available Mem: 7976 965 6238 82 772 6663 Swap: 0 0 0
After setting that option and logging out and back in again, this is my memory usage with TWENTY ssh sessions connected to VyOS:
admin@edge:~$ free -m total used free shared buff/cache available Mem: 7976 794 6408 81 772 6834 Swap: 0 0 0
The same fixed result occurs with something else mentioned in the link attached in a separate comment:
Commenting out the myhostname option. I don't know the implications of that though
/etc/nsswitch.conf hosts: files dns #myhostname