Our largest instance has 23,080 lines for config.boot (mostly firewall rule configuration for 15 or so VLANs).
VyOS 1.1.8 to 1.2 configuration migration and boot: 25 min.
Save configuration time: 10 sec.
VyOS 1.2.0 saved config boot time: 19 min.
Commit configuration change time: 3 min 10 sec. (and yes the wait is terrifying)
I've traced back the majority of the boot slowdown to executing iptables for each rule insertion.
Updating the firewall to create a temporary file with changes and calling it using iptables-restore -n < $FILENAME the -n flag being the noflush and important if not replacing the entire ruleset.
Because iptables-restore performs an atomic change there are a few advantages:
- Error in any of the rules will cause the atomic change to fail before applied for all rules making error recovery much easier in terms of VyOS not getting out of sync with itself.
- The speed of the atomic commit is orders of magnitude faster for large rulesets (from 15 min to 30 sec in one example).
- Because iptables-restore with the -n flag allows for insert and removal operations rather than a full flush it is easy to combine add and remove operations such that there is no gap in policy between deletion and insertion of rules making changes safer from a traffic perspective.
For simple operations the syntax is identical to iptables with each line in the file being the same as what would be provided after iptables ending with the word COMMIT (testing needed).
In terms of the commit time I don't know if there is an easy way to address this until the filesystem is no longer used for storing the config tree as a directory structure.
I think adopting the atomic netfilter configuration would be one something easy enough to implement in 1.2.1+ though.
