This actually has nothing to do with the http api, but rather is a deadlock coming from ConfigSession show_config(), namely, the use of Popen.wait() in __run_command().

This issue has been fixed in this commit: https://github.com/vyos/vyos-1x/commit/0fb5cc02c6c16ef410c1b1f746c1fbfbc40a0da0

@dmbaturin @zsdc Perhaps we could discuss the possibility of changing the dhcp(v6) server to a better server (and what feature changes might exist), which I hope will address the older ISC's better support for interfaces such as PPPOE and the latest dhcp(v6) standard

DHCPd shold be vonsistent for both v4 and v6 - running different daemons is simply bad.

GDB however tells a different story:

vyos@vyos# ps aux | grep cli
root      1624  0.1  1.9  35212 19508 ?        S    12:11   0:00 /bin/cli-shell-api showConfig
root      1684  0.0  0.5   9492  5720 ?        Ss   Feb16   0:00 /sbin/dhclient -4 -nw -cf /var/lib/dhcp/dhclient_eth0.conf -pf /var/lib/dhcp/dhclient_eth0.pid -lf /var/lib/dhcp/dhclient_eth0.leases eth0
vyos      2534  0.0  0.0   6084   892 pts/0    S+   12:22   0:00 grep cli

Seems like the issue manifests in cnode-algorithm.cpp:_diff_print_indent:

Perfect, thanks

Sorry. Yep. Can confirm, problem fixed.

Here is some more testing. Currently on FRR 7.5:

Currently, vyos-configd will explicitly report ConfigError to the console; however, non-fatal warnings or information will not be reported. An example of a non-fatal warning is:

SolarFlare modules present in the kernel for 1.3 and 1.4

vyos@vyos:~$ sudo modinfo sfc
filename:       /lib/modules/5.10.14-amd64-vyos/kernel/drivers/net/ethernet/sfc/sfc.ko
license:        GPL
description:    Solarflare network driver
author:         Solarflare Communications and Michael Brown <[email protected]>
alias:          pci:v00001924d00001B03sv*sd*bc*sc*i*
alias:          pci:v00001924d00000B03sv*sd*bc*sc*i*
alias:          pci:v00001924d00001A03sv*sd*bc*sc*i*
alias:          pci:v00001924d00000A03sv*sd*bc*sc*i*
alias:          pci:v00001924d00001923sv*sd*bc*sc*i*
alias:          pci:v00001924d00000923sv*sd*bc*sc*i*
alias:          pci:v00001924d00001903sv*sd*bc*sc*i*
alias:          pci:v00001924d00000903sv*sd*bc*sc*i*
alias:          pci:v00001924d00000813sv*sd*bc*sc*i*
alias:          pci:v00001924d00000803sv*sd*bc*sc*i*
alias:          pci:v000010EEd00001100sv*sd*bc*sc*i*
alias:          pci:v000010EEd00000100sv*sd*bc*sc*i*
depends:        mdio
retpoline:      Y
intree:         Y
name:           sfc
vermagic:       5.10.14-amd64-vyos SMP mod_unload modversions 
parm:           vf_max_tx_channels:Limit the number of TX channels VFs can use (uint)
parm:           max_vfs:Reduce the number of VFs initialized by the driver (int)
parm:           mcdi_logging_default:Enable MCDI logging on newly-probed functions (bool)
parm:           rx_refill_threshold:RX descriptor ring refill threshold (%) (uint)
parm:           irq_adapt_low_thresh:Threshold score for reducing IRQ moderation (uint)
parm:           irq_adapt_high_thresh:Threshold score for increasing IRQ moderation (uint)
parm:           interrupt_mode:Interrupt mode (0=>MSIX 1=>MSI 2=>legacy) (uint)
parm:           rss_cpus:Number of CPUs to use for Receive-Side Scaling (uint)
parm:           efx_separate_tx_channels:Use separate channels for TX and RX (bool)
parm:           phy_flash_cfg:Set PHYs into reflash mode initially (bool)
parm:           debug:Bitmapped debugging message enable value (uint)

As a solution proposed to use TTL by default equivalent 16, but also add the possibility to change it via VyOS CLI

Tried to mess around/instrument https://github.com/vyos/vyatta-cfg/blob/current/src/cnode/cnode-algorithm.cpp#L924,
ended up loosing the active config and /opt/vyatta/config/active becoming empty.

Looks good on 1.4-rolling-202102162107 (including migration from self-built 1.2.0-rolling+202102162120).

It seems that the stall happens in ConfigSession.show_config() which calls /bin/cli-shell-api showConfig.

@c-po That seemed to more than double my RX!

@mathiashedberg could you try and enable RPS set interfaces ethernet eth0 offload rps and see if this does any good on utilisation / drop rate? I had a similar issue with a PPPoE link which behaved super bad under preasure.

So after a week of running, and comparing to performance with the LTS, i know that something is wrong.

@plett VyOS 1.4 now does an automatic fallback to the 1.3 behavior which used FRR 7.3 until we decide how to handle this, w.g. by adding a new CLI option which is added by a migration script to keep older systems running "as is" when upgrading.

Yeah.. I don't think it's a timeout issue really.. because of this:

In T2100#72560, @jestabro wrote:

FRR 7.4 has been released, and the default behaviour has been changed, commit 62282e8379. @Viacheslav, when we update to this version, I can work with you to update the migration script.

@Viacheslav , good point, and unfortunately, that's not an exact science: cf. T2252, wherein the timeout was raised to 10 minutes, the time of the timeout reported above.

It possible nginx limitation, need to change/set/tunning some timeouts.

@pasik this is not a known issue, and your report is much appreciated. I have not drilled down into the problem yet; any debugging that you have a chance to do is welcome !

Any comments here? Is this a known issue/limitation? I can look into the internals and do some debugging if this should be working in the current state..

The translation is working properly now but it is not showing in the command output:

Maybe we should consider using a better proxy server to complete the dhcpv6 agent

Somehow this was lost in translation in my git repo....