Page MenuHomeVyOS Platform

Python CLI
Open, Requires assessmentPublicFEATURE REQUEST

Description

Once the migration to Python complete and that nothing is left from Vyatta, then a new CLI will be required to replace the legacy code.
The CLI will need to load the xml.in files and provide the same interface than the users, including auto-complete and data validation.

The same code could be used and tested as a remote CLI using the HTTP API too (it would however only provide validation and auto-completion for the known XML).

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Event Timeline

thomas-mangin created this task.
pasik added a subscriber: pasik.May 27 2020, 5:14 PM
c-po added a subscriber: c-po.May 30 2020, 12:54 PM

That is what VyConf is for

I just had a look at VyConf and it is excellent, I fear that no one but @dmbaturin can maintain participate to it.
Also, VyConf will still need to fork all the python code and unless we have a resolution to T2088 - I am not sure what the best path forward will be

POC: https://github.com/thomas-mangin/vyos-1x/tree/T2522

The cli is mostly functional. I am able to validate the data as it is typed in conf mode (the CLI has both completion and validation working), as soon as delete and show (the current show is "show configuration commands") are implemented, it will be mostly usable. The code can already load a configuration from file, allow some "set" edit and then allow the use of save (the config format is a number of "set" commands, one per line), respect the initialisation order of the XML.

I implemented a new pure python Config class (by using del Config, from .. import Config so not changing the config file ATM. This monkey patching mean that all the code in src/conf works without change.
The configuration is handled in memory only, so if we want to allow multiple edit, it will need to be moved to a long lived process (otherwise each action will require a full reload).
In order to be able to import the conf_mode code and l link it to the owners (as the string is a path, not a python module name), I created a symlink from src/conf_mod to python/vyos/modules/configuration. It allows me to use the code without touching the repo, but ultimately a repo restructuration to move all the completion/validation/operation and configuration code into the pyhton/vyos folder would help me.

The code is good enough to be used to configure interfaces (I have not tested all type of data and commands, so there is surely plenty of bugs).

On commit, all the CPU is now spent running the commands and setting 100 dummy went from 42 to 8 seconds on my test VirtualBox VyOS instance. It could even be speed-up even more on multi-core machines by running multiprocess but I am not planning to go there ATM/yet. Plenty of other optimisations are possible but would be premature ATM.

I am not just sure how the python prompt toolkit will work on console (vs normal VT100) and I may have to re-implement the part I use in a way which works with the console, this again can be looked into later. It is also a pain as it has to be pip installed as root vs the apt packaged version which is missing some features it seems.

thomas-mangin added a comment.EditedJun 2 2020, 2:10 PM

Configuring 100 dummy interfaces (no thread) on a local VirtualBox, all tests done from boot

42 seconds for the current code.

1 CORE
without threading

real	0m 8.90s
user	0m 2.60s
sys	0m 1.61s

with threading

real	0m 8.93s
user	0m 2.50s
sys	0m 2.07s

with multiprocessing (to be expected with one core this is not as fast)

real	0m 10.56s
user	0m 3.11s
sys	0m 2.39s

4 CORES

without threading

real	0m 4.60s
user	0m 2.03s
sys	0m 2.01s

with threading

real	0m 3.70s
user	0m 2.38s
sys	0m 4.85s

with multiprocessing

real	0m 3.50s
user	0m 2.75s
sys	0m 4.01s

Removing the debug printing of vyos.ifconfig.debug
Threading

real	0m 3.32s
user	0m 2.33s
sys	0m 3.69s

multiprocessing

real	0m 3.29s
user	0m 2.65s
sys	0m 3.64s

same but without any printing

real	0m 2.77s
user	0m 2.24s
sys	0m 2.55s

load empty configuration (first time)

real	0m 0.97s
user	0m 0.34s
sys	0m 0.27s

empty configuration (second time)

real	0m 0.41s
user	0m 0.33s
sys	0m 0.07s

I see the POC as complete and conclusive, it would only make sense to spend more time on things once this is done (or to add support for control via HTTP/API).

The patched code will continue to work as expected for the live production (ie: the patch does not modify/break the repository, just adds to it).

A new class Config is monkey patched by the vyos/cli/main.py when it runs. It is a full in-memory implementation (plus a few more function as the new code does not have a ConfigTree class anymore.
But nothing would prevent to monkey patch a new one which would interface with VyConf instead.

For the code to work, the xml is parsed and converted to a dict which is added as a python file to the repository. The code could be optimised quite a bit but I see no point right now as it is already quite fast !

The cli implements, set, delete, load, save and show (with diff view).
It can load and apply a full configuration and load and apply to memory only (so you can edit from the changed state from the first use).

For it to work the python prompt-toolkit must be installed using sudo python -m pip install prompt-toolkit which mean that apt get install python-pip must have been done before hand.

It requires to run as sudo to be able to write to the proc filesystem.
sudo python3 /vyos/cli/main.py

to load an apply a config
sudo time python3 /usr/lib/python3/dist-packages/vyos/cli/main.py --load config --commit --no-cli

to load an apply a config, and then give you a shell to continue editing:
sudo time python3 ./vyos/cli/main.py --load config --commit

to load a config and then modify it to only have the modification done
sudo time python3 /usr/lib/python3/dist-packages/vyos/cli/main.py --load config

The configuration file is a list of set command (the output of show configuration commands) and # lines are ignored.

If the POC is accepted, it could be modified it so that it can load the data from the REST API and therefore provide a remote CLI. This should be "trivial" to add (just take time).
The REST API could also be extended to provide the XML schema to make sure that the schema used is the same as the router.

As a side note, if there is a need to migrate between backends, it would be possible to change the code to have an option to indicate which storage should be used and allow loading from one, saving to another as long as the XML schema is the same.

jjakob added a subscriber: jjakob.Jun 2 2020, 7:51 PM

A significant part of the old config system is the bash-completion integration as well. I assume this is not integrated with bash but is a separate console that you start and takes over all stdin/stdout? Is it possible to implement the same completion output as there is now?

I would make a separate branch for this so that we still have a proven working 1.3-rolling while the new CLI is being worked on, as it's a drastic change and I personally would like it to be at least on feature parity with the Vyatta CLI (that means command completion, help text, etc.) before getting into current.

@jjakob you are right, there is no shell integration and this is using the python promt-toolkit library to handle input/output.

It is much more like a Juniper CLI than what VyOS does ATM, but if there is support for the concept, I am willing to work on it until we get feature parity - I am unsure what I am signing up for when I say that tho :-)

The "run" command is passed through using _vyatta_op_run (without completion - which could be added) but it is not vbash. I am not sure how vbash works but how script-template set the search for the first few letters.

Before shell integration, multi-user support may be more important (which is again nothing hard, but it will require a single source of truth for the config, and currently, this is just kept in memory and not distributed).

I still have to make the basic "edit" command work too but that would not be the hardest ;-D

For a separate branch, I would rather not diverge from current, and as the code is all "additive", it is ok. There is a number of repository re-structure which would make no difference to what is done but would make it easier to work with (like moving many things in src/ to a good home in python/vyos) but it should be a separate discussion.

It should not be too hard to convert the current parser to read.
https://gist.github.com/thomas-mangin/17a450a3e26a4cc41902475c0a1dfe5f

Obviously all the code generation in src/conf_mode relies on the current xml but it should be possible to have a different get_config() like function for openconfig.
This is obviously a totally different discussion but I thought I should mention here how it becomes easier to do so.

jjakob added a comment.EditedJun 5 2020, 1:10 PM

Writing the config parsing, editing, committing and saving shouldn't be that hard.
Re-implementing completion with the current set of features would be very hard.

To list the features I personally like and use (most are bash):

  • command history (up/down arrow keys)
  • reverse history search (bash: ctrl-r)
  • word jumping between words in the current line (ctrl-left arrow, ctrl-right arrow)
  • ctrl-c to delete the current typed line

Features added to bash that are purely vyatta-cfg is obviously the whole set of tab-completion of commands, command help output, command lists, config node string completion, interface name completion, etc.

These would need to work within the config session. For example, to run a lot of nearly identical commands, one can simply type the full command once, then use the history to find the old one, quickly jump to the thing you want to change with word jumping, change it, run it, repeat. Without the history features I'd need to copy that command into the local clipboard and paste it each time, and I still wouldn't have the word jumping to speed up editing.

Another consideration is the handling of terminal types. Do we support just VT100? Does the library we use have the escape sequences for jumping to certain positions on the screen built in or do we need to reimplement it? We'd need to then determine the x-y positions of the cursor, where each word starts and ends, and send corresponding escape sequences to move the cursor. Also an important part is different terminals have different backspace, delete, arrow, ... characters. Do we keep a list of all of those and do the mapping in code or does the library do that for us?
Bash currently handles all of that transparently to the backend code.

IMO the quickest way with the least work would be to reuse the current CLI code from vyatta: vbash, bash-completion, everything. It's all written around calls to cli-shell-api. It wouldn't be that hard to make our own cli-shell-api that returned output in the way it expects. AFAIK from when I last worked on it, cli-shell-api is ran with options corresponding to the data it wants, then cli-shell-api returns a string that contains shell-escaped variable assignments that are then evaluated with shell's 'eval' to bring them into the shell local variables. AFAIK it's pretty simple. Then the only complexity on our side would be to look up and generate the completion in our code and put it into the correct shell-escaped variables string. To implement completion in our own CLI we'd need those completion functions anyway.

The alternative would be a massive amount of work to basically duplicate what's already there. There may be some things that'd need fixing in the vyatta bash code, but that'd be much easier, and we'd have a working product much quicker than rewriting our own shell (which is basically what we'd be doing: writing our own shell) from scratch.

@jjakob the code can be installed on a router (using my vyos update tool - after running vyos setup router )

I agree that to be used as a replacement we need to provide feature parity with the current CLI. The python prompt-toolkit library provides what is required for the features you listed above.

Command history, validation and tab-completion, interface completion, are done part of the POC, which is also coming with a fish-like preview of the command completion. Reverse history search works by typing the start of the command and up, word jump and line erasure should require just a little bit of work but were not considered implemented (this is only a POC).
https://python-prompt-toolkit.readthedocs.io/en/master/pages/asking_for_input.html#adding-custom-key-bindings

I did not want to go into much detail but it should also be possible to integrate the code with https://xon.sh which I have been using as my day-to-day shell for two weeks now (eating my own dog-food) to see if it could be the bases for a vbash replacement. Up to now, I am happy enough with it that I am not planning to go back to fish or zsh.

As I only connect to VyOS via ssh so I could not test more than bare "VT100" by changing the TERM env. It is supported out of the box:
https://github.com/prompt-toolkit/python-prompt-toolkit/tree/master/prompt_toolkit/output/

For anything, else, it needs testing, the first use of the CLI code may however only be to provide remote administration using the HTTP API and not on the router.
The CLI used on the router may well not change. I just have shown that it is possible to do. For the POC it was "simpler" for me to just rewrite the lot, which provides me with the "remote CLI" feature at the same time.

I do agree that writing new code to replace some working code requires development time and that the project is not rich in contributors. Since VyOS move to python, more people have been able to help and I believe moving more code to python will make the project easier to gain more contributors.

Plus this is the kind of code I like to write on my "free" time :-)

jjakob added a comment.Jun 5 2020, 3:23 PM

@thomas-mangin that's great if the POC already has the above - I'm on board with making it the replacement for vbash in that case. I can live without word jumping and line deletion (for now) if it has tab completion and history (I will miss reverse history search a lot as it doesn't search just the beginning of the line but the whole line for the pattern, I'm assuming prompt-toolkit just searches the beginning?). I do need to test it when I get some time to see if anything is still missing.

@jjakob control-R should be implemented. It is a feature I use too and expect it - just did not think it was worth a demo 0.0 release - there is plenty to improve with the code in the branch - it surely has bugs -as I said POC :-)