Page MenuHomeVyOS Platform

Python validators are slow
Open, Requires assessmentPublic

Description

Python config node validators are slow.

$ time sudo /usr/libexec/vyos/validate-value.py --exec /usr/libexec/vyos/validators/ip-prefix --value '192.0.2.1/24'

real	0m0.133s
user	0m0.097s
sys	0m0.037s

$ time sudo sh -c ' for ((n=0;n<1000;n++)); do /usr/libexec/vyos/validate-value.py --exec /usr/libexec/vyos/validators/ip-prefix --value "192.0.2.1/24"; done'

real	1m58.335s
user	1m34.704s
sys	0m24.341s

That's 118s for 1000 config lines (590s for 5000 lines) just for the validator, not counting other execution time. Also note this is a relatively powerful 2.5Ghz dual-core Pentium E5300.

$ cat /usr/libexec/vyos/validators/ip-prefix
#!/bin/sh

ipaddrcheck --is-any-net $1
$ time ipaddrcheck --is-any-net '192.0.2.1/24'

real	0m0.003s
user	0m0.003s
sys	0m0.000s
$ time sudo sh -c ' for ((n=0;n<1000;n++)); do ipaddrcheck --is-any-net "192.0.2.1/24"; done'

real	0m1.985s
user	0m1.235s
sys	0m0.868s

Only 1.9s for 1000 invocations of the same validator directly, without going through the Python invocation.

It's due to Python startup time. Validation of every config node requires a full startup of Python. It would be better to if the validators were shell scripts or C programs, as they were before, the load time is very small. I think this is also a part of the significant difference in boot and commit times between 1.2 and 1.3.
I think this is a powerful enough argument to rewrite the validators to shell or C (ipaddrcheck is already C). validate-value.py doesn't do anything other than invoke the destination validator or do a simple regex, which can be a sh or C validator too.

This was originally discussed in T2425.

I have rewritten validate-value.py in bash, which is much faster already (12-14 times per validation in my testing):

$ time sudo bash -c ' for ((n=0;n<1000;n++)); do
  ./validate-value.sh
  --exec /usr/libexec/vyos/validators/ip-prefix
  --value "192.0.2.1/24"; done'

real	0m9.606s
user	0m5.576s
sys	0m4.285s

$ time sudo bash -c ' for ((n=0;n<1000;n++)); do
  /usr/libexec/vyos/validate-value.py
  --exec /usr/libexec/vyos/validators/ip-prefix
  --value "192.0.2.1/24"; done'

real	1m58.335s
user	1m34.704s
sys	0m24.341s

$ time sudo bash -c ' for ((n=0;n<1000;n++)); do
  ./validate-value.sh
  --regex "([0-9A-Fa-f]{1,2}[:])*([0-9A-Fa-f]{1,2})"
  --value "ff:ff:ff:ff:ff:ff"; done'

real	0m7.455s
user	0m4.264s
sys	0m3.900s

$ time sudo bash -c ' for ((n=0;n<1000;n++)); do
  /usr/libexec/vyos/validate-value.py
  --regex "([0-9A-Fa-f]{1,2}[:])*([0-9A-Fa-f]{1,2})"
  --value "ff:ff:ff:ff:ff:ff"; done'

real	1m45.868s
user	1m28.573s
sys	0m17.378s

This seems to have reduced my boot time a bit from approx. 220-250s before to 194s after. I'm sure rewriting all the individual validators will bring similar orders of magnitude improvements.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible

Event Timeline

jjakob created this task.May 7 2020, 6:50 PM
jjakob created this object in space S1 VyOS Public.
jjakob updated the task description. (Show Details)
jjakob added a comment.EditedMay 9 2020, 12:07 PM

New tests with the OCaml validate-value (same machine as above)

$ time sudo sh -c 'for ((n=0;n<1000;n++)); do ipaddrcheck --is-any-net "192.0.2.1/24"; done'

real	0m2.088s
user	0m1.230s
sys	0m0.956s

baseline - still the same

$ time sudo sh -c 'for ((n=0;n<1000;n++)); do /usr/libexec/vyos/validators/ip-prefix "192.0.2.1/24"; done'

real	0m5.599s
user	0m3.282s
sys	0m2.436s

validators/ip-prefix is 2.7x slower than ipaddrcheck (it's a one-liner shell script that just executes ipaddrcheck)

$ time sudo sh -c 'for ((n=0;n<1000;n++)); do ./validate-value --exec /usr/libexec/vyos/validators/ip-prefix --value "192.0.2.1/24";done'

real	0m11.353s
user	0m5.263s
sys	0m6.284s

OCaml validate-value: 2.0x slower than ip-prefix, 5.4x slower than ipaddrcheck, 18% slower than my bash validate-value.sh, 10.4x faster than the python validate-value.py

$ time sudo sh -c 'for ((n=0;n<1000;n++)); do ./validate-value --regex "([0-9A-Fa-f]{1,2}[:])*([0-9A-Fa-f]{1,2})" --value "ff:ff:ff:ff:ff:ff"; done'

real	0m3.492s
user	0m1.467s
sys	0m2.130s

OCaml validate-value: 2.1x faster than my bash validate-value.sh, 30x faster than the python validate-value.py

So in essence, the OCaml validate-value is significantly faster than my bash script for regexes (which are done via grep in my bash script) but my script beats it by 18% for --exec. It's at least an order of magnitude faster than the old python validator. All in all, much in favour of OCaml.

Now, if we could rewrite all validators in OCaml or bash, and get rid of the intermediate shell invocation for calling ipaddrcheck, we could improve things significantly more again.

I'm already noticing a significant shell responsiveness improvement and faster boot time and commit time.

Comparison of the numeric validator:

$ time sudo sh -c 'for ((n=0;n<1000;n++)); do /usr/libexec/vyos/validators/numeric 1; done'

real	1m25.518s
user	1m11.249s
sys	0m14.353s

Baseline - old python

$ time sudo sh -c 'for ((n=0;n<1000;n++)); do ./numeric 1; done'

real	0m2.897s
user	0m1.474s
sys	0m1.500s

New OCaml numeric - 29x faster than the old python validator

related to T2088 where performance is also being discussed.