Page MenuHomeVyOS Platform

Script for remote file transfers
Closed, ResolvedPublicFEATURE REQUEST

Description

(Metatask)

The current way we handle network file transfers is rather messy. There are multiple scripts that independently wrap curl and ssh for file transfers:

  1. vyatta-remote-copy.pl (used exclusively for tech support file uploads). Completely removed.
  2. vyatta-image-tools.pl contains its own ad hoc routines for curl and rsync for some strange reason even though vyatta-remote-copy.pl is available. Deserves to be removed eventually.
  3. Erstwhile remote.py (used for config management). Rewritten.
  4. vyatta-save-config.pl (also used for config management).
  5. vyatta-load-user-key.pl for loadkey, which also needs to be more robust. Deprecated in favour of generate public-key-command. Deserves to be removed eventually.
  6. vyatta-commit-push.pl.
  7. install-image.
  8. get_latest_iso.py uses requests directly.

All of them spend a lot of effort figuring out the given protocol and fiddling with host keys.
The solution is writing a single all-Python script as the one-stop solution for all remote transfers in the codebase. Python standard library comes with ftplib, which is an adequate FTP client for our needs, although there is nothing for SSH. Paramiko is a well-known, reliable SSH library for Python, suited exactly for this purpose. It would save us from relying on curl and herding host keys.
loadkey currently breaks on some keys and the rewrite needs to tolerate both bare keys and keys with user/host info.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change
Issue type
Improvement (missing useful functionality)

Related Objects

StatusSubtypeAssignedTask
In progressFEATURE REQUESTNone
ResolvedFEATURE REQUESTerkin
ResolvedBUGerkin
On holderkin
ResolvedBUGchaya2z
ResolvedBUGerkin
ResolvedBUGerkin
Resolvederkin
ResolvedBUGerkin
ResolvedENHANCEMENTerkin
ResolvedBUGerkin
ResolvedBUGerkin
Resolved N/ABUGerkin
InvalidFEATURE REQUESTNone
Resolved N/AFEATURE REQUESTc-po
Resolvederkin
ResolvedFEATURE REQUESTc-po
ResolvedBUGerkin
Resolvederkin
Wontfixerkin
Openerkin

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

It seems other commands like save, copy which depend on scp and tftp are also broken.

vyos@R3:~$ copy file /config/archive/config.boot.1.gz to tftp://192.168.0.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 176, in upload
    upload_tftp(local_path, url.hostname, url.path, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 117, in upload_tftp
    stderr=None, input=file.read()).encode()
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
OSError: [Errno 71] failed to run command: curl  -s -T - tftp://192.168.0.1:69/
returned:
exit code: 71
vyos@R3# save tftp://192.168.0.1
Saving configuration to 'tftp://192.168.0.1'...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 176, in upload
    upload_tftp(local_path, url.hostname, url.path, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 117, in upload_tftp
    stderr=None, input=file.read()).encode()
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
OSError: [Errno 71] failed to run command: curl  -s -T - tftp://192.168.0.1:69/
returned:
exit code: 71
Done
vyos@R3:~$ copy file /config/archive/config.boot.1.gz to scp://vyos:vyos@192.168.0.1/tmp
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  m.add_string(self.Q_C.public_numbers().encode_point())
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.curve, Q_S_bytes
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  hm.add_string(self.Q_C.public_numbers().encode_point())
Host '192.168.0.1' not found in known hosts.
Fingerprint: 1b70f4247188eca6fe2281293920a162
Do you wish to continue? [y/N] y
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 174, in upload
    upload_sftp(local_path, url.hostname, url.path, url.username, url.password, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 104, in upload_sftp
    transfer_sftp('upload', *args, **kwargs)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 93, in transfer_sftp
    sftp.put(local_path, remote_path)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 759, in put
    return self.putfo(fl, remotepath, file_size, callback, confirm)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 714, in putfo
    with self.file(remotepath, "wb") as fr:
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 372, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 813, in _request
    return self._read_response(num)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 865, in _read_response
    self._convert_status(msg)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 898, in _convert_status
    raise IOError(text)
OSError: Failure

copy file still depends on vyatta-image-tools.pl. I think it merits a rewrite, maybe a simple file transfer script that uses a couple of basic routines for file:// and running:// and remote.py for everything else.

Now that all major instances of curl have been replaced with the in-house script, we can begin to backport these changes to v1.3 in small pieces ahead of the first stable version.

erkin set Issue type to Unspecified (please specify).
erkin updated the task description. (Show Details)
erkin reopened this task as In progress.Dec 2 2021, 2:44 PM
erkin updated the task description. (Show Details)
erkin changed Issue type from Unspecified (please specify) to Improvement (missing useful functionality).
Viacheslav changed the status of subtask T4029: Broken SFTP uploads from Open to Needs testing.Dec 11 2021, 9:38 AM