Re-reading the entry, I am now unsure what you believe should be different.

Yes, I should parse the tagNode and insert them into the default data.

Thank you for this explaining what is happening. I would indeed rather tag were used as keys.

Yes, so there is no way to know at parsing time (unless you have two elements or check the XML) that this is a multi-element. Something "computer-friendly" would generate:

test {
    something [one]
}

The data returned by get_config_dict should always have the same format to prevent any function using it to have to check if one element is a list or a string.

Agreeing and defining the name of the keys to be used in the dict passed to verify/generate/apply, would be very beneficial for the project.
The current code does this by calling a number of helper function (so that all interface have the same keys) but this is not defined or/and enforced.

Should we provide the script which perform the change and define what action can be performed in the xml?

The work to perform the get_config/verify/generate/apply in python is already available in this tree:
https://github.com/thomas-mangin/vyos-1x/tree/T2522

@cpo, you can change the defaults from True to False in the patch if this is what you want to do.

https://github.com/vyos/vyos-1x/pull/479

I propose to provide alternatives is_ functions using the new XML code. I will provide a patch for review.

I agree that this kind of changes are better done at the start of a development cycle.

There is really no difference between calling Config() and using it functions in the code - as is done by every module - vs having the class inherit from it as I do not use any private ( _ prefixed ) functions. I can modify the code to have self.config as an object of the class instead but IMO this is cosmetic and does not change anything with the API and coding guideline.

not willing to take the lead on this task but happy to help.

@jestabro could you please clarify how interfaces-tunnel.py is not following the guideline. The class it uses to generate the dict is internal to the get_config() function and the dict API is respected.

The problem also probably exists with vif_s .. it needs to be investigated.

https://github.com/vyos/vyos-1x/pull/474/files

was done part of T2633

could have the range 68-65536 but it may be a bit on the extreme side.

https://github.com/vyos/vyos-1x/pull/473 was merged so now need to agree sane limits for the XML.

I have a PR for this (not changing the XML limiting range) for review ATM.

related to T2630

https://github.com/vyos/vyos-1x/pull/472

Breaking user existing configs should be a no-no. If the options can be used that way under Linux, then we should not restrict it if it is not invalid. If we intend to prevent it then we would need a way to warn users clearly and we have no framework for this ATM.

Need to add max MTU to operational mode and create a new validator using it and applying it to the xml. The only question being if the information is always available.

I see no issue with the proposed solution.

LOL - I could have commited :-) Thanks !

@Dmitry correct same bug - thank you. resolved.

https://github.com/vyos/vyos-1x/pull/457

The patch was merged and the issue should be resolved with the next ISO.

https://github.com/vyos/vyos-1x/pull/456

Thank you very much for the POC. Very useful to understand the proposed design.

hello Fabio, could you please show me how the vti interfaces are presented under Linux so I can fix the code. I thought I had properly ported the code from Perl to Python must I must have misunderstood something.

Yes, we need to try/except the apply section (the other should never fail but we could still catch errors to not leave the system in an unknown state) but when applying the reverse configuration (ie: invert effective and new and re-apply) one must then be careful if that fails too (we do not want a forever loop :p). The code already runs all the get_dict and all the verify first, so we will only apply if all is ok, but still issues could occur.

@jjakob control-R should be implemented. It is a feature I use too and expect it - just did not think it was worth a demo 0.0 release - there is plenty to improve with the code in the branch - it surely has bugs -as I said POC :-)

generate should make a backup of the previous file before generating the new one. It will then make it possible to create the rollback as a file move and service reload.

@jjakob the code can be installed on a router (using my vyos update tool - after running vyos setup router )

It should not be too hard to convert the current parser to read.
https://gist.github.com/thomas-mangin/17a450a3e26a4cc41902475c0a1dfe5f

@jjakob you are right, there is no shell integration and this is using the python promt-toolkit library to handle input/output.

Thank you for reporting this issue, it looks like that parser allows ranges of IP address (IP hyphen IP) but the parser does not. You could get around using CIDR notation but this indeed need looking into.

should help to go further in the testing, but it is still failing on set_state but I do not know why it should be done if the interface is managed by openvpn. The relevant code is:

As a side note, if there is a need to migrate between backends, it would be possible to change the code to have an option to indicate which storage should be used and allow loading from one, saving to another as long as the XML schema is the same.

I see the POC as complete and conclusive, it would only make sense to spend more time on things once this is done (or to add support for control via HTTP/API).

Configuring 100 dummy interfaces (no thread) on a local VirtualBox, all tests done from boot

Thank you for this long answer @jjakob. I want to demonstrate that a full python solution can provide the performance we need. I appreciate that changing the Vyatta code need to be done carefully with many consideration about backward compatibility. What I am doing is surely 1.4 material. However I do not believe this is as hard to achieve as everyone may think, and as working code is the best way to discuss code design, that is what I am doing.

The cli is mostly functional. I am able to validate the data as it is typed in conf mode (the CLI has both completion and validation working), as soon as delete and show (the current show is "show configuration commands") are implemented, it will be mostly usable. The code can already load a configuration from file, allow some "set" edit and then allow the use of save (the config format is a number of "set" commands, one per line), respect the initialisation order of the XML.

I used @runar code on my branch working on T2522 .. Configuring 100 dummy interface went from 42 to 8 seconds.

I just had a look at VyConf and it is excellent, I fear that no one but @dmbaturin can maintain participate to it.
Also, VyConf will still need to fork all the python code and unless we have a resolution to T2088 - I am not sure what the best path forward will be

I tried to get a flamegraph showing what I was wanting to say but .. do not look very clear :-(

I believe it was git pull —tags which fixed it for me ..

Many scripts also so do implement if name == “main”

Also related to T2088

https://github.com/vyos/vyos-1x/pull/423

The called code can return 3 - in that case in that case _run should return an empty string

https://github.com/vyos/vyos-1x/pull/422/files

It causes this:

9140 ?        Ss     0:00 /bin/bash /usr/libexec/vyos/init/vyos-router start
 9427 ?        S      0:00  \_ /bin/bash /usr/libexec/vyos/init/vyos-router start
 9428 ?        S      0:00      \_ python3 /usr/libexec/vyos/vyos-boot-config-loader.py /opt/vyatta/etc/config/config.boot
 9451 ?        S      0:00          \_ /opt/vyatta/sbin/my_commit
 9522 ?        S      0:00              \_ sudo sh -c VYOS_TAGNODE_VALUE='eth0' /usr/libexec/vyos/conf_mode/interfaces-ethernet.py
 9523 ?        Sl     0:00                  \_ python3 /usr/libexec/vyos/conf_mode/interfaces-ethernet.py
 9531 ?        R      0:00                      \_ /bin/cli-shell-api --show-active-only --show-show-defaults --show-ignore-edit showConfig

related to T2088 where performance is also being discussed.

waiting for a decision on T2485 before doing this work

https://github.com/vyos/vyos-1x/pull/418

I have worked and provided a patch for T2485 .. It may be the right place to move it in.

https://docs.google.com/document/d/1pKkxwq5eBm9eSTmyrSluDU1cx8NYqQRkQNdTB1tCSdk/edit?usp=sharing

Every call to /bin/cli-shell-api --show-working-only --show-show-defaults --show-ignore-edit showConfig takes multiples seconds (6?)
/usr/libexec/vyos/conf_mode/system-timezone.py call it twice.
/usr/libexec/vyos/conf_mode/nat.py call it twice
/usr/libexec/vyos/conf_mode/interfaces-loopback.py call it twice ... etc.

0-18 kernel boot
18-41 system starting inc FRR
70-120 the python most of the time is spend/wasted in cli-shell-api - so I would think reading the configuration file. If we can optimise / reduce this number of calls it would be very good.
140-220 is more or less firewall setup with vyatta-firewall / vyatta-upset.pl / ip6tables

I need to double-check (and may not get to it) but if you use a vyos-build to build current and then try to build crux, make iso is not happy. I am now building crux using the command in my post above, the only difference: clean vyos-build install before the git checkout crux.

ediing vyos make iso -r crux -d
https://github.com/thomas-mangin/vyos-hacker-toolkit

Last time I generate crux I used the docker current to do it ...
(but I could not this time...)

https://github.com/vyos/vyos-1x/pull/412

@runar just created this as I can not create a dev env without a phabricator entry. answer in 20 minutes :-)