As the code changes and we integrate the conf mode and operation mode from independent programs which are forked to code which is imported a number of repository change could be considered.
Move the code from /usr/libexec/vyos to the python vyos folder.
It is possible to import the configuration code using env PYTHONPATH=/usr/libexec/vyos python3 and then import conf_mode. It is however "hacky" as it requires to change the environment.
It would surely be preferable to have the code installed in the python folder, and symlinked from the old location (file by file or as a folder).
I will propose python/vyos/modules/configuration for /usr/libexec/vyos/conf_mode (for discussion sake) and then symlink that folder in libexec to preserve compatibility with any code which may have expected the file there. It should then be possible at a next release to remove the liecex folder. Giving enough warning to users (or even keeping it as a legacy symlink).
This should/could also be done for the op_mode, completion and validation. In the case of completion and validation, not all the code is however in python and some could be rewritten.
Rename some of the files / programs
The file in theses folders also have names with "-" which may it painful to import using the standard python keyword. It is still possible using importlib but clumsy. Therefore I would also suggest that any filename in the vyos-1x repository should use "_". Again, if symlink are used, it is possible to keep the name in libexec as they were.
If individual symlinks are done, it would be possible to write a single program which would introspect sys.argv and then import and run the configuration mode previously under that name, allowing to change the name of the file in the repository while keeping backward compatibility (if this is required).
Define an API/Interface for Configuration
The configuration code is made of a number of classes , Config and ConfigTree being the main one. I would suggest that we should look at this code in term of "API", for otherw part of the code, in particular what function should be used / available from the conf_mode modules and should be considered as part of the "conf_mode" "API" and which one are not.
Also many of the CLI command do map exactly to some function in the ConfigTree. If you consider the concept of Interface (like Java or Go) then I believe we should define what should be implemented and how, so that multiple implementation can co-exist ( the current Vyatta, VyConf and an in-memory one) as they would all have different use case.
Some of the code should/could be refactored when shared. I will attempt to document this "visually" and update this task with something later on.
This new API is very useful and should be used more. I would suggest that an improved version of ConfigurationState (which would not be a subclass of configuration but instead be returned by a factory function in Configuration) could become a better object to pass between the verify / generate and apply.