Page MenuHomeVyOS Platform

Script for remote file transfers
Closed, ResolvedPublicFEATURE REQUEST

Description

The current way we handle network file transfers is rather messy. There are multiple scripts that independently wrap curl and ssh for file transfers:

  1. vyatta-remote-copy.pl (used exclusively for tech support file uploads). Completely removed.
  2. vyatta-image-tools.pl contains its own ad hoc routines for curl and rsync for some strange reason even though vyatta-remote-copy.pl is available. Deserves to be removed eventually.
  3. Erstwhile remote.py (used for config management). Rewritten.
  4. vyatta-save-config.pl (also used for config management).
  5. vyatta-load-user-key.pl for loadkey, which also needs to be more robust. Deprecated in favour of generate public-key-command. Deserves to be removed eventually.
  6. vyatta-commit-push.pl.
  7. install-image.

All of them spend a lot of effort figuring out the given protocol and fiddling with host keys.
The solution is writing a single all-Python script as the one-stop solution for all remote transfers in the codebase. Python standard library comes with ftplib, which is an adequate FTP client for our needs, although there is nothing for SSH. Paramiko is a well-known, reliable SSH library for Python, suited exactly for this purpose. It would save us from relying on curl and herding host keys.
loadkey currently breaks on some keys and the rewrite needs to tolerate both bare keys and keys with user/host info.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
erkin changed the task status from Open to In progress.Feb 25 2021, 3:43 PM
erkin claimed this task.

remote.py declaring support for TFTP and HTTP(S) might complicate things further. Since it's only for downloading, the latter shouldn't be too complicated to handle with urllib but if we need HTTP(S) uploading capabilities, an external library like requests would be preferable. As for TFTP, there's no support for it in the standard library, but instead of pulling in a third-party library for TFTP alone, we can just write an ad hoc wrapper for curl dedicated to TFTP. It wouldn't be as complicated since TFTP has no form of authentication so there's no need to worry about host keys and such.

erkin triaged this task as Low priority.Mar 1 2021, 9:20 AM

Turns out vyatta-remote-copy.pl has been broken for a while. It fails to exchange encryption keys when it tries to establish an SSH connection (for SFTP or SCP), possibly because of the old libssh it relies on.

It seems other commands like save, copy which depend on scp and tftp are also broken.

vyos@R3:~$ copy file /config/archive/config.boot.1.gz to tftp://192.168.0.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 176, in upload
    upload_tftp(local_path, url.hostname, url.path, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 117, in upload_tftp
    stderr=None, input=file.read()).encode()
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
OSError: [Errno 71] failed to run command: curl  -s -T - tftp://192.168.0.1:69/
returned:
exit code: 71
vyos@R3# save tftp://192.168.0.1
Saving configuration to 'tftp://192.168.0.1'...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 176, in upload
    upload_tftp(local_path, url.hostname, url.path, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 117, in upload_tftp
    stderr=None, input=file.read()).encode()
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
OSError: [Errno 71] failed to run command: curl  -s -T - tftp://192.168.0.1:69/
returned:
exit code: 71
Done
vyos@R3:~$ copy file /config/archive/config.boot.1.gz to scp://vyos:vyos@192.168.0.1/tmp
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  m.add_string(self.Q_C.public_numbers().encode_point())
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.curve, Q_S_bytes
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  hm.add_string(self.Q_C.public_numbers().encode_point())
Host '192.168.0.1' not found in known hosts.
Fingerprint: 1b70f4247188eca6fe2281293920a162
Do you wish to continue? [y/N] y
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 174, in upload
    upload_sftp(local_path, url.hostname, url.path, url.username, url.password, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 104, in upload_sftp
    transfer_sftp('upload', *args, **kwargs)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 93, in transfer_sftp
    sftp.put(local_path, remote_path)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 759, in put
    return self.putfo(fl, remotepath, file_size, callback, confirm)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 714, in putfo
    with self.file(remotepath, "wb") as fr:
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 372, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 813, in _request
    return self._read_response(num)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 865, in _read_response
    self._convert_status(msg)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 898, in _convert_status
    raise IOError(text)
OSError: Failure

copy file still depends on vyatta-image-tools.pl. I think it merits a rewrite, maybe a simple file transfer script that uses a couple of basic routines for file:// and running:// and remote.py for everything else.

Now that all major instances of curl have been replaced with the in-house script, we can begin to backport these changes to v1.3 in small pieces ahead of the first stable version.