Page MenuHomePhabricator

Improving Boot Time for Large Firewall Configurations
Confirmed, NormalPublicFEATURE REQUEST

Description

Our largest instance has 23,080 lines for config.boot (mostly firewall rule configuration for 15 or so VLANs).

VyOS 1.1.8 to 1.2 configuration migration and boot: 25 min.

Save configuration time: 10 sec.

VyOS 1.2.0 saved config boot time: 19 min.

Commit configuration change time: 3 min 10 sec. (and yes the wait is terrifying)

I've traced back the majority of the boot slowdown to executing iptables for each rule insertion.

Updating the firewall to create a temporary file with changes and calling it using iptables-restore -n < $FILENAME the -n flag being the noflush and important if not replacing the entire ruleset.

Because iptables-restore performs an atomic change there are a few advantages:

  1. Error in any of the rules will cause the atomic change to fail before applied for all rules making error recovery much easier in terms of VyOS not getting out of sync with itself.
  2. The speed of the atomic commit is orders of magnitude faster for large rulesets (from 15 min to 30 sec in one example).
  3. Because iptables-restore with the -n flag allows for insert and removal operations rather than a full flush it is easy to combine add and remove operations such that there is no gap in policy between deletion and insertion of rules making changes safer from a traffic perspective.

For simple operations the syntax is identical to iptables with each line in the file being the same as what would be provided after iptables ending with the word COMMIT (testing needed).

In terms of the commit time I don't know if there is an easy way to address this until the filesystem is no longer used for storing the config tree as a directory structure.

I think adopting the atomic netfilter configuration would be one something easy enough to implement in 1.2.1+ though.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close

Event Timeline

rps created this task.Feb 4 2019, 8:28 PM
syncer triaged this task as Normal priority.
syncer changed the task status from Open to Confirmed.
pasik added a subscriber: pasik.Feb 5 2019, 7:44 PM
fmayo added a subscriber: fmayo.Feb 12 2019, 4:08 PM
fmayo added a comment.Feb 12 2019, 4:15 PM

I am affected too by this issue.

My configuration file has around 7500 lines, while 5000 of them are related to firewall (rules and address-groups)

With VyOS 1.2 boot time is 12 minutes, while in VyOS 1.8 the same config takes less than 3 minutes. commit times went from 10 sec to nearly one minute.

syncer assigned this task to dmbaturin.
yun added a subscriber: yun.Apr 23 2019, 8:09 PM