Script for remote file transfers
Closed, ResolvedPublicFEATURE REQUEST
Actions

Assigned To

Authored By

	erkin
	Feb 25 2021, 3:43 PM

Description

(Metatask)

The current way we handle network file transfers is rather messy. There are multiple scripts that independently wrap curl and ssh for file transfers:

~~vyatta-remote-copy.pl (used exclusively for tech support file uploads).~~ Completely removed.
~~vyatta-image-tools.pl contains its own ad hoc routines for curl and rsync for some strange reason even though vyatta-remote-copy.pl is available.~~ Deserves to be removed eventually.
~~Erstwhile remote.py (used for config management).~~ Rewritten.
~~vyatta-save-config.pl (also used for config management).~~
~~vyatta-load-user-key.pl for loadkey, which also needs to be more robust.~~ Deprecated in favour of generate public-key-command. Deserves to be removed eventually.
~~vyatta-commit-push.pl.~~
~~install-image~~.
get_latest_iso.py uses requests directly.

All of them spend a lot of effort figuring out the given protocol and fiddling with host keys.
The solution is writing a single all-Python script as the one-stop solution for all remote transfers in the codebase. Python standard library comes with ftplib, which is an adequate FTP client for our needs, although there is nothing for SSH. Paramiko is a well-known, reliable SSH library for Python, suited exactly for this purpose. It would save us from relying on curl and herding host keys.
loadkey currently breaks on some keys and the rewrite needs to tolerate both bare keys and keys with user/host info.

Details

Difficulty level: Unknown (require assessment)
Version: -
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Behavior change
Issue type: Improvement (missing useful functionality)

Related Objects
Search...

Status	Subtype	Assigned	Task
In progress	FEATURE REQUEST	None	T3355 Remove all remaining legacy Vyatta code
Resolved	FEATURE REQUEST	erkin	T3356 Script for remote file transfers
Resolved	BUG	erkin	T3439 Commit-archive location not working for scp
Resolved		erkin	T3506 Migrate loadkey command to op-mode
Resolved	BUG	chaya2z	T3865 loadkey command help text missing escape sequence
Resolved	BUG	erkin	T3508 Check if there's enough drive space for an upgrade before downloading an image
Resolved	BUG	erkin	T3556 Commit-archive via scp causes 100% CPU on boot
Resolved		erkin	T3563 commit-archive breaks with IPv6 source addresses
Resolved	BUG	erkin	T3518 Warning messages when using SCP commit-archive
Resolved	ENHANCEMENT	erkin	T1506 commit-archive scp/sftp public key authentication
Resolved	BUG	erkin	T3378 commit-archive source-address broken for IPv6 addresses
Resolved	BUG	erkin	T1866 Commit archive over SFTP doesn't work with non-standard ports
Resolved N/A	BUG	erkin	T3628 commit-archive source-address Interface Broken
Invalid	FEATURE REQUEST	None	T2736 Change curl to aria2c
Resolved N/A	FEATURE REQUEST	c-po	T2651 Generate CLI abstraction for options passed to CURL and SSH client
Resolved		erkin	T2615 Provide an explicit option for server fingerprint in commit archive, and make insecure the default
Resolved	FEATURE REQUEST	c-po	T2596 Allow specifying source IP for 'add system image'
Resolved	FEATURE REQUEST	c-po	T4922 Add ssh-client source-interface CLI option
Resolved	BUG	erkin	T3950 CLI backtrace on update if DNS not defined
Resolved		erkin	T4029 Broken SFTP uploads
Wontfix		erkin	T4090 Source port and interface support for `commit-archive`
Open		erkin	T4091 Progress bar support for HTTP uploads
Resolved		erkin	T5650 Progressbars suffer from staircasing effect
Open		erkin	T5985 Downloaded files end up with unwieldy file attributes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

SrividyaA mentioned this in T3556: Commit-archive via scp causes 100% CPU on boot.May 17 2021, 1:43 PM

erkin added a subtask: T3508: Check if there's enough drive space for an upgrade before downloading an image.May 18 2021, 11:18 AM

erkin added a subtask: T3556: Commit-archive via scp causes 100% CPU on boot.

SrividyaA added a subscriber: SrividyaA.May 19 2021, 6:08 PM

Viacheslav added a subtask: T3563: commit-archive breaks with IPv6 source addresses.May 20 2021, 7:53 AM

Viacheslav added a subtask: T3518: Warning messages when using SCP commit-archive.

Viacheslav added a subtask: T1506: commit-archive scp/sftp public key authentication.May 20 2021, 7:57 AM

Viacheslav added a subtask: T3378: commit-archive source-address broken for IPv6 addresses.

It seems other commands like save, copy which depend on scp and tftp are also broken.

vyos@R3:~$ copy file /config/archive/config.boot.1.gz to tftp://192.168.0.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 176, in upload
    upload_tftp(local_path, url.hostname, url.path, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 117, in upload_tftp
    stderr=None, input=file.read()).encode()
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
OSError: [Errno 71] failed to run command: curl  -s -T - tftp://192.168.0.1:69/
returned:
exit code: 71

vyos@R3# save tftp://192.168.0.1
Saving configuration to 'tftp://192.168.0.1'...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 176, in upload
    upload_tftp(local_path, url.hostname, url.path, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 117, in upload_tftp
    stderr=None, input=file.read()).encode()
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
OSError: [Errno 71] failed to run command: curl  -s -T - tftp://192.168.0.1:69/
returned:
exit code: 71
Done

vyos@R3:~$ copy file /config/archive/config.boot.1.gz to scp://vyos:[email protected]/tmp
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  m.add_string(self.Q_C.public_numbers().encode_point())
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.curve, Q_S_bytes
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  hm.add_string(self.Q_C.public_numbers().encode_point())
Host '192.168.0.1' not found in known hosts.
Fingerprint: 1b70f4247188eca6fe2281293920a162
Do you wish to continue? [y/N] y
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 174, in upload
    upload_sftp(local_path, url.hostname, url.path, url.username, url.password, source=source)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 104, in upload_sftp
    transfer_sftp('upload', *args, **kwargs)
  File "/usr/lib/python3/dist-packages/vyos/remote.py", line 93, in transfer_sftp
    sftp.put(local_path, remote_path)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 759, in put
    return self.putfo(fl, remotepath, file_size, callback, confirm)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 714, in putfo
    with self.file(remotepath, "wb") as fr:
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 372, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 813, in _request
    return self._read_response(num)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 865, in _read_response
    self._convert_status(msg)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 898, in _convert_status
    raise IOError(text)
OSError: Failure

erkin added a subtask: T1866: Commit archive over SFTP doesn't work with non-standard ports.May 30 2021, 9:54 AM

erkin updated the task description. (Show Details)

erkin closed subtask T3518: Warning messages when using SCP commit-archive as Resolved.

erkin closed subtask T1506: commit-archive scp/sftp public key authentication as Resolved.May 30 2021, 9:59 AM

erkin changed the status of subtask T3563: commit-archive breaks with IPv6 source addresses from Open to Needs testing.

erkin closed subtask T3508: Check if there's enough drive space for an upgrade before downloading an image as Resolved.May 30 2021, 10:03 AM

erkin changed the status of subtask T3506: Migrate loadkey command to op-mode from Open to In progress.May 30 2021, 10:21 AM

erkin changed the status of subtask T3563: commit-archive breaks with IPv6 source addresses from Needs testing to In progress.Jun 2 2021, 9:53 AM

erkin changed the status of subtask T3556: Commit-archive via scp causes 100% CPU on boot from Open to Needs testing.Jun 2 2021, 9:57 AM

erkin closed subtask T3563: commit-archive breaks with IPv6 source addresses as Resolved.Jun 8 2021, 2:20 PM

erkin changed the status of subtask T3378: commit-archive source-address broken for IPv6 addresses from Open to Needs testing.Jun 8 2021, 2:31 PM

trae32566 reopened subtask T3563: commit-archive breaks with IPv6 source addresses as Open.Jun 11 2021, 9:45 AM

trae32566 added a subtask: T3628: commit-archive source-address Interface Broken.Jun 16 2021, 7:03 AM

copy file still depends on vyatta-image-tools.pl. I think it merits a rewrite, maybe a simple file transfer script that uses a couple of basic routines for file:// and running:// and remote.py for everything else.

erkin updated the task description. (Show Details)Jun 17 2021, 2:08 PM

erkin updated the task description. (Show Details)Jun 20 2021, 2:30 PM

Now that all major instances of curl have been replaced with the in-house script, we can begin to backport these changes to v1.3 in small pieces ahead of the first stable version.

erkin changed the status of subtask T3628: commit-archive source-address Interface Broken from Open to In progress.Jun 20 2021, 3:48 PM

erkin closed subtask T3563: commit-archive breaks with IPv6 source addresses as Resolved.Jun 21 2021, 4:55 AM

erkin closed subtask T3628: commit-archive source-address Interface Broken as Resolved.

erkin changed the status of subtask T3506: Migrate loadkey command to op-mode from In progress to On hold.Jun 23 2021, 8:49 AM

erkin closed this task as Resolved.Jun 23 2021, 9:11 AM

erkin closed subtask T3556: Commit-archive via scp causes 100% CPU on boot as Resolved.Jun 23 2021, 9:51 AM

FileGo reopened subtask T3556: Commit-archive via scp causes 100% CPU on boot as Open.Jun 23 2021, 10:06 AM

erkin changed the status of subtask T3556: Commit-archive via scp causes 100% CPU on boot from Open to In progress.Jul 1 2021, 2:48 PM

trae32566 reopened subtask T3628: commit-archive source-address Interface Broken as Open.Jul 7 2021, 5:51 AM

erkin changed the status of subtask T3628: commit-archive source-address Interface Broken from Open to In progress.Jul 7 2021, 9:03 AM

erkin added a subtask: T2736: Change curl to aria2c.Aug 29 2021, 1:42 PM

erkin added a subtask: T2651: Generate CLI abstraction for options passed to CURL and SSH client.Aug 29 2021, 2:05 PM