When using:
/configure
{"op": "set", "path": ["service", "https", "api", "keys","id","MYKEY","key","password22"]}
The API returns with:
{
"error": "service https api unavailable at this proxy address: set service https api-restrict virtual-host"
}
However, using the new key works perfectly.
Changing the api settings (for example: keys, port) will cause a disruptive reload of the http-api service; it will of course succeed, but the response for that set command will reflect the reloading. This is known, if misleading behaviour. I will lower this to normal priority, and consider if we can improve the error/success message.
As discussed in above comment, this is understandable behaviour, but will be re-investigated after the move to fastapi, re T2397.
I'm also getting the same error when calling the https API from localhost. In my case it only happens occasionally.
service https api unavailable at this proxy address: set service https api-restrict virtual-host
The APIs that I was trying to call when I got the error:
set [system flow-accounting interface br2]
show [interfaces vxlan vxlan1308]
delete [interfaces ethernet eth4]
https configs:
service {
https {
api {
keys {
id vyos-tester {
key ****************
}
}
}
api-restrict {
virtual-host localhost
}
virtual-host localhost {
}
}
}VyOS version:
Version: VyOS 1.4-rolling-202202230317 Release train: sagitta Built by: [email protected] Built on: Wed 23 Feb 2022 03:17 UTC Build UUID: 008815ba-b343-4d5b-a86d-950ef7a843c2 Build commit ID: 7c82c5c7104675 Architecture: x86_64 Boot via: installed image System type: KVM guest Hardware vendor: QEMU Hardware model: Standard PC (i440FX + PIIX, 1996) Hardware S/N: Hardware UUID: 75151ed2-890b-4a78-b485-a9b57477caf7
I never touch https configs again after setting up the VyOS VM.
I have tried this on different VMs and the same behavior still occurs.
I dug a little deeper, it appears that calling 2 http APIs in parallel results in vyos-http-api library crash.
Jul 5 08:47:39 cxr vyos-http-api[107198]: Configuration modified via HTTP API using key 'ccube-dev' Jul 5 08:47:39 cxr vyos-http-api[107198]: INFO: None:0 - "POST /configure HTTP/1.0" 200 OK Jul 5 08:47:39 cxr netplugd[907]: br4: can't get flags: No such device Jul 5 08:47:39 cxr vyos-http-api[107198]: processing form data Jul 5 08:47:39 cxr netplugd[907]: br4: can't get flags: No such device Jul 5 08:47:39 cxr netplugd[907]: message repeated 3 times: [ br4: can't get flags: No such device] Jul 5 08:47:39 cxr netplugd[907]: br5: can't get flags: No such device Jul 5 08:47:39 cxr netplugd[907]: br4: can't get flags: No such device Jul 5 08:47:39 cxr netplugd[907]: message repeated 27 times: [ br4: can't get flags: No such device] Jul 5 08:47:40 cxr vyos-http-api[107198]: INFO: None:0 - "POST /config-file HTTP/1.0" 200 OK Jul 5 08:47:40 cxr vyos-http-api[107198]: processing form data Jul 5 08:47:40 cxr vyos-http-api[107198]: INFO: None:0 - "POST /retrieve HTTP/1.0" 400 Bad Request Jul 5 08:47:40 cxr vyos-http-api[107198]: processing form data Jul 5 08:47:40 cxr vyos-http-api[107198]: processing form data Jul 5 08:47:40 cxr ntpd[3893]: Listen normally on 87 vti1 169.254.231.46:123 Jul 5 08:47:40 cxr ntpd[3893]: new interface(s) found: waking up resolver Jul 5 08:47:41 cxr kernel: [104872.825731] vyos-http-api-s[107280]: segfault at 1020 ip 00007f792d30391d sp 00007f792dce93e0 error 4 in libvyosconfig.so.0[7f792d2ae000+10c000] Jul 5 08:47:41 cxr kernel: [104872.825745] Code: 20 48 83 c4 08 c3 e8 f2 d9 fa ff eb c6 48 83 ec 48 48 8b 40 10 48 89 44 24 18 48 8b 40 20 48 8b 58 20 48 8b 5b 20 48 8b 7b 20 <48> 8b 77 20 48 89 74 24 10 48 8b 56 20 48 89 54 24 20 48 8b 7f 08 Jul 5 08:47:41 cxr kernel: [104872.833057] net_ratelimit: 24 callbacks suppressed Jul 5 08:47:41 cxr kernel: [104872.833059] IPv4: martian source 10.10.10.18 from 10.10.10.1, on dev eth2 Jul 5 08:47:41 cxr kernel: [104872.833061] ll header: 00000000: ff ff ff ff ff ff 6a 2c d7 cd 51 fd 08 06 Jul 5 08:47:41 cxr systemd[1]: vyos-http-api.service: Main process exited, code=killed, status=11/SEGV
The behaviour is quite different depending on what is done. In my case another API key (not the one my application uses) should be deleted and fails, the service restarts afterwards and the config is still the same. Is there something like a hot reload missing? Also, if the API-server dies without restart (had that case several times, would have to find out, which command does that) even a change using cli did not recover the service.
root@rz0-rt1:/home/wornet# show version Version: VyOS 1.3.2 Release train: equuleus Built by: Sentrium S.L. Built on: Mon 05 Sep 2022 09:23 UTC Build UUID: 1ceaab3a-4f4e-4692-b551-7c05e1da0a77 Build commit ID: 7ce86511888635 Architecture: x86_64 Boot via: installed image System type: bare metal Hardware vendor: HP Hardware model: ProLiant MicroServer Gen8 Hardware S/N: <redacted> Hardware UUID: <redacted> Copyright: VyOS maintainers and contributors
Command sent: [{"op": "delete", "path": ["service", "https", "api", "keys", "id", "routerdeployment"]}]
Log:
Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: ConfigSessionError: Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: Traceback (most recent call last): Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: File "/usr/libexec/vyos/services/vyos-http-api-server", line 459, in configure_op Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: session.commit() Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 165, in commit Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: out = self.__run_command([COMMIT]) Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 137, in __run_command Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: raise ConfigSessionError(output) Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: vyos.configsession.ConfigSessionError Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: INFO: 212.11.<redacted>:0 - "POST /configure HTTP/1.0" 400 Bad Request Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: processing form data Nov 25 09:19:27 rz0-rt1 vyos-http-api[1539]: INFO: Shutting down
Self-configuration of the http-api calls a service restart from the config mode script: some re-configuration should be possible without restart; the remaining should provide an explanatory 'success' response. Move to high-priority to address.
The similar bug with load if we change something in service https api
curl -k --location 192.168.122.11 --request POST 'https://192.168.122.11/config-file' --form data='{"op": "load", "file": "config.boot"}' --form key='foo'
{"success": false, "error": "", "data": null}Logs from the VyOS node VyOS 1.4-rolling-202309280309
Oct 05 20:11:55 r1 vyos-http-api[4425]: processing form data
Oct 05 20:11:55 r1 sudo[4522]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/http-api.py'
Oct 05 20:11:55 r1 sudo[4522]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Oct 05 20:11:55 r1 vyos-configd[641]: Received message: {"type": "init"}
Oct 05 20:11:56 r1 vyos-configd[641]: config session pid is 4425
Oct 05 20:11:56 r1 vyos-configd[641]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/http-api.py"}
Oct 05 20:11:56 r1 vyos-configd[641]: Sending response 8
Oct 05 20:11:56 r1 systemd[1]: Reloading.
Oct 05 20:11:56 r1 systemd[1]: Stopping vyos-http-api.service - VyOS HTTP API service...
Oct 05 20:11:56 r1 sudo[4522]: pam_unix(sudo:session): session closed for user root
Oct 05 20:11:56 r1 vyos-http-api[4425]: INFO: 192.168.122.1:0 - "POST /config-file HTTP/1.0" 400 Bad Request
Oct 05 20:11:56 r1 systemd[1]: opt-vyatta-config-tmp-new_config_4425.mount: Deactivated successfully.
Oct 05 20:11:56 r1 vyos-http-api[4425]: INFO: Shutting down
Oct 05 20:11:56 r1 vyos-http-api[4425]: INFO: Waiting for application shutdown.
Oct 05 20:11:56 r1 vyos-http-api[4425]: INFO: Application shutdown complete.
Oct 05 20:11:56 r1 vyos-http-api[4425]: INFO: Finished server process [4425]
Oct 05 20:11:56 r1 systemd[1]: vyos-http-api.service: Deactivated successfully.
Oct 05 20:11:56 r1 systemd[1]: Stopped vyos-http-api.service - VyOS HTTP API service.
Oct 05 20:11:56 r1 systemd[1]: Started vyos-http-api.service - VyOS HTTP API service.
Oct 05 20:11:56 r1 vyos-http-api[4558]: INFO: Started server process [4558]
Oct 05 20:11:56 r1 vyos-http-api[4558]: INFO: Waiting for application startup.
Oct 05 20:11:56 r1 vyos-http-api[4558]: INFO: Application startup complete.
Oct 05 20:11:56 r1 vyos-http-api[4558]: INFO: Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)Is it possible to at least have the correct error message?
Final testing before PR, the following corrects behavior when configuring the http-api using the http-api, for example:
curl -k -X POST -Fkey=qwerty -Fdata='{"op": "set", "path": ["service", "https", "api", "socket"]}' https://192.168.122.67/configure
{"success": true, "data": "Requested HTTP API server configuration change; commit will be called in the background", "error": null}
curl -k -X POST -Fkey=qwerty -Fdata='{"op": "set", "path": ["service", "https", "api", "keys", "id", "jestabro", "key", "newkey"]}' https://192.168.122.67/configure
{"success": true, "data": "Requested HTTP API server configuration change; commit will be called in the background", "error": null}
curl -k -X POST -Fkey=newkey -Fdata='{"op": "showConfig", "path": []}' https://192.168.122.67/retrieve
{"success": true, "data": {"interfaces": {"ethernet": {"eth0": {"address": "dhcp", "description": "TEST113", "hw-id": "52:54:00:34:42:89"}}, "loopback": {"lo": {}}}, "service": {"https": {"api": {"keys": {"id": {"jestabro": {"key": "newkey"}}}, "socket": {}}}, "ntp": {"allow-client": {"address": ["0.0.0.0/0", "::/0"]}, "server": {"time1.vyos.net": {}, "time2.vyos.net": {}, "time3.vyos.net": {}}}, "ssh": {}}, "system": {"config-management": {"commit-revisions": "100"}, "conntrack": {"modules": {"ftp": {}, "h323": {}, "nfs": {}, "pptp": {}, "sip": {}, "sqlnet": {}, "tftp": {}}}, "console": {"device": {"ttyS0": {"speed": "115200"}}}, "host-name": "vyos", "login": {"user": {"vyos": {"authentication": {"encrypted-password": "$6$BXlhO/0.p45zKhHL$jK2Un68xHTQRMJd.7oLMd7H5EgAOsly/8fWDw1tCWEHUSiq/UqCMR7r8W2HVSOdBH3slhGhpPPRStnni7ttkw1", "plaintext-password": ""}}}}, "syslog": {"global": {"facility": {"all": {"level": "info"}, "local7": {"level": "debug"}}}}}}, "error": null}https://github.com/vyos/vyos-1x/compare/current...jestabro:api-self-config