Page MenuHomeVyOS Platform

Script for remote file transfers
In progress, LowPublicFEATURE REQUEST

Description

The current way we handle network file transfers is rather messy. There are multiple scripts that independently wrap curl and ssh for file transfers:

  1. vyatta-remote-copy.pl (used exclusively for tech support file uploads).
  2. vyatta-image-tools.pl contains its own ad hoc routines for curl and rsync for some strange reason even though vyatta-remote-copy.pl is available.
  3. Erstwhile remote.py (used for config management).
  4. vyatta-save-config.pl (also used for config management).
  5. vyatta-load-user-key.pl for loadkey, which also needs to be more robust.
  6. vyatta-commit-push.pl.

All of them spend a lot of effort figuring out the given protocol and fiddling with host keys.
The solution is writing a single all-Python script as the one-stop solution for all remote transfers in the codebase. Python standard library comes with ftplib, which is an adequate FTP client for our needs, although there is nothing for SSH. Paramiko is a well-known, reliable SSH library for Python, suited exactly for this purpose. It would save us from relying on curl and herding host keys.
loadkey currently breaks on some keys and the rewrite needs to tolerate both bare keys and keys with user/host info.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change

Event Timeline

erkin changed the task status from Open to In progress.Feb 25 2021, 3:43 PM
erkin claimed this task.
erkin created this task.

remote.py declaring support for TFTP and HTTP(S) might complicate things further. Since it's only for downloading, the latter shouldn't be too complicated to handle with urllib but if we need HTTP(S) uploading capabilities, an external library like requests would be preferable. As for TFTP, there's no support for it in the standard library, but instead of pulling in a third-party library for TFTP alone, we can just write an ad hoc wrapper for curl dedicated to TFTP. It wouldn't be as complicated since TFTP has no form of authentication so there's no need to worry about host keys and such.

erkin triaged this task as Low priority.Mar 1 2021, 9:20 AM

Turns out vyatta-remote-copy.pl has been broken for a while. It fails to exchange encryption keys when it tries to establish an SSH connection (for SFTP or SCP), possibly because of the old libssh it relies on.