@dmbaturin
Confirmed: the problem is not reproducible anymore in 1.2.0-rolling+201908010337 with keepalived 1:2.0.17+vyos1.2.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Aug 1 2019
@ekim, we have never met with such a problem and cannot reproduce it in our environments. The better way to continue investigation would be getting access to this installation. If this would be possible, we could continue debugging.
Jul 18 2019
@mb300sd, yes - nothing newer yet. Just test, please, when a new build will be available. :)
The problem, which leads to the malformed hostname in Hostname Capability was fixed in T1531. I am marking this as "Resolved", because the problem with DNS servers was resolved also, according to feedback.
Thank you, @ekim!
What exactly you were doing at the moment of this log record? I see a lot of scripts, which are almost permanently do something in the system. Is it possible that this system contains some custom scripts (in /config/scripts folder, for example)?
If yes, you should check the schedule of execution, and requirements of modification for 1.2 version command syntax.
Jul 16 2019
Jul 13 2019
Hello, @ekim! I see now. This is more looks like the waiting due to I/O. Do, please, the next:
- Run sudo atop -w /tmp/atop-mon.log -a 5 60 in dedicated terminal.
- Try to work several minutes in the terminal. It must freezing at this moment, otherwise, collected test data will be wrong.
- Wait until atop finish its work (~ 5 min from the start).
- Copy the file /tmp/atop-mon.log from the router and send to us for analysis.
Jul 10 2019
@ekim, could you make a screen record with this CLI delay?
Hello, all!
I have prepared the pull requests for fixing this bug. They add hooks for two situations:
- if VRRP configuration changed;
- if firewall settings for interface changed.
Jul 8 2019
Hello, @ekim!
Such a significant increase of boot time with the same configuration is very strange, but still possible - even small changes can easily cause such behavior if you have a huge or some specific configuration. But what is more strange - CLI response time. Regardless of configuration, CLI should work without visible delays, except for autocompleting or commit operations.
Could you check the current load of the host at that moment, when CLI is slow? We must be sure that the system is not overloaded.
Jun 28 2019
I am confirming that the problem is not reproducing in the 2.0.17. We should upgrade keepalived distribution.
Jun 26 2019
I have checked behavior in 4.3.5 and 4.4.1 versions. The information about hostname is still not synced from primary to secondary.
As I see from the information in the Debian bug report, it is about the other bug - when hostname not rewritten after offering lease. From the ISC-DHCP changelog:
In T1416#38750, @dongjunbo wrote:
@dongjunbo, show please the configuration of this router so we could check why gcdomestic pool does not count correctly.
Jun 24 2019
Provided configuration from the first message was successfully loaded in 1.2.0-rolling+201906240337.
@csalcedo, could you test new rolling to check if the problem is solved for you too?
Hello, @dongjunbo!
Have you tried current rolling releases to check if leases information view work correctly now?
The safest solution will be waiting for 2.0.17, test compatibility with VyOS again, and then update keepalived package inside the VyOS.
As I see, from current VyOS scripts, keepalived restart only at router startup or if all VRRP groups were deleted. In case of configuration change we use reload, which is correct.
This means that we get nothing from the keeping state in case of the restart - there is no sense to keep states of deleted groups, and we have nothing to keep at first startup.
Jun 23 2019
Jun 22 2019
I agree with you on many arguments. I just wanted to say my point of view.
There are many differences between vendors: in terminology, in behavior, in technologies. This is a normal situation. The same applying to engineers - we are very diversified, and this is beautiful :).
I strongly disagree about mixing dummy/loopback at any of level (CLI or under the hood). Now we have two different types of interfaces: dummy and loopback with the according to names. And they are equal only if you use a dummy for the /32 addresses. In general situation:
- Loopback interface will be used to reach any of the address inside the configured network;
- If the IP address assigned to a dummy interface, the system will respond only to this address, not for the whole network.
Example, to be more precise:
set interfaces loopback lo address 192.168.8.1/24 set interfaces dummy dum1 address 192.168.9.1/24
Jun 21 2019
Jun 20 2019
Example of the output when value is below 10000000:
vyos@test-06:~$ show firewall name TESTFW rule 50
The problem was fixed in https://phabricator.vyos.net/R6:97c5ad3dca756635e83eb3bf667f742457d85d74.
Jun 19 2019
Jun 5 2019
Jun 4 2019
This bug must be fixed in current rolling, because of T1416. @dongjunbo, check this, please.
May 30 2019
Problems with DHCP server status viewing can be fixed with the next patch for show_dhcp.py :
--- orig/show_dhcp.py 2019-05-30 22:45:01.625708032 +0300 +++ T1416/show_dhcp.py 2019-05-30 22:40:33.302777881 +0300 @@ -55,15 +55,28 @@ return data
May 22 2019
May 16 2019
The solution was tested and fully worked.
@hagbard, everything works fine now. Thank you!
May 15 2019
@hagbard @UnicronNL, please make a new one PR. This is correct patch:
diff -Naur origin/dhclient-script pull2/dhclient-script --- origin/dhclient-script 2019-05-15 19:32:59.001598203 +0300 +++ pull2/dhclient-script 2019-05-15 19:33:47.533181873 +0300 @@ -39,7 +39,6 @@ echo " " > $new_resolv_conf fi
May 14 2019
This is not the same.
set protocols static table 100 interface-route 10.100.100.0/24 next-hop-interface eth1
generate:
test-06# show running-config staticd Building configuration...
May 13 2019
@hagbard solution works. Please, add it also to the stable branch and to the set protocols static table section.
May 3 2019
Apr 18 2019
Unfortunately, current Linux GRE and network stack implementations don't support Cisco-style of GRE keepalives (GRE inside GRE, with spoofed IP addresses). From the Linux point of view, those packets look like martians, and the kernel drop them, information about what you can see inside a log.
Try to disable the keepalive at the Cisco side, after this tunnel must be fully functional.
Apr 5 2019
Sorry, I must reopen this task. Absolutely the same situation with multiple "lower" interfaces:
OPTIONS="-6 -l ::%eth1.100-l ::%eth1.102 -u 2001:db8:0:feed::2%eth2.88 -u 2001:db8:0:feed::3%eth2.88 " ^^ here
Apr 3 2019
Mar 21 2019
Mar 12 2019
Mar 6 2019
Thank you! This was made the situation much more apparent. Provide, please, an output of the next command:
sudo ipset list
Mar 4 2019
Feb 28 2019
Feb 27 2019
Can you disable this GRE tunnel and make a dump of a moment when a tunnel being established?
Also, send output of:
sudo ip6tables -t raw -L -n -v sudo ip6tables -t filter -L -n -v sudo ip6tables -t mangle -L -n -v sudo ip6tables -t nat -L -n -v
Unfortunately, this works only for default route, but not for any other.
Feb 26 2019
@jjcordon can you provide the full configuration and test again with a latest rolling version?
In our test with 1.2.0-rolling+201902250337 everything works fine:
routing table: C>* 2001:XXXX:YYYY:11::/64 is directly connected, eth1, 00:27:06 C>* 2001:XXXX:YYYY:12::/64 is directly connected, eth2, 00:14:28
According to pmacct configuration, this looks good. We need to check code, and if all is correct I propose to merge this into rolling for testing.
Just the one nuance. Currently, VyOS CLI doesn't allow to set a BGP neighbor to the local IP address. If we accept this patch, then it will be good to remove this restriction.
I'm closing this ticket, because we don't have feedback from author for a long time nor configuration for reproducing this by ourselves.
If someone will meet with this bug with FRRouting-based versions of VyOS, please reopen.
Hello!
Bug confirmed in 1.2.0-rolling+201902250337. Current way of checking values accept only that names, which is returned by cat /etc/iproute2/rt_dsfield (source). Currently with kernel 4.19.20-amd64-vyos this is:
# Differentiated field values # These include the DSCP and unused bits 0x0 default # Newer RFC2597 values 0x28 AF11 0x30 AF12 0x38 AF13 0x48 AF21 0x50 AF22 0x58 AF23 0x68 AF31 0x70 AF32 0x78 AF33 0x88 AF41 0x90 AF42 0x98 AF43 # Older values RFC2474 0x20 CS1 0x40 CS2 0x60 CS3 0x80 CS4 0xA0 CS5 0xC0 CS6 0xE0 CS7 # RFC 2598 0xB8 EF
Feb 25 2019
Confirmed in 1.2.0-rolling+201902250337. A hw-id is assigned to only one of interfaces per boot.
If this will not lead to a race condition situation, maybe try to implement the solution from T577#12637 proposed by @rps?
Confirmed in 1.2.0-rolling+201902250337.
I think we must allow adding overlapped networks/address into the same set, as ipset allows to do this. Maybe situations, where this will be necessary, are rare (for example, resizing network size defined in set), but there is no clear reason to deny such combinations.
Feb 14 2019
With new package all works fine.
Feb 13 2019
Need to reopen this task.
Version: 1.2.0-LTS.
Running configuration:
vyos@test-01# show interfaces { ethernet eth0 { address 192.168.55.18/30 duplex auto hw-id 08:00:27:95:bb:f6 smp-affinity auto speed auto } ethernet eth1 { address 192.168.56.3/24 duplex auto hw-id 08:00:27:8e:d6:fb smp-affinity auto speed auto } ethernet eth2 { duplex auto hw-id 08:00:27:8c:27:04 smp-affinity auto speed auto } loopback lo { } } service { ssh { } } system { config-management { commit-revisions 100 } console { device ttyS0 { speed 9600 } } host-name test-01 login { user vyos { authentication { encrypted-password $6$7X4XbQJ2xVMZ8$NmISPmyC1f88cIfcKig01pkjePNTVeeWwULrHgich6wB0A1TH/b31Jywpsde8Mv4/B8Qa5CxFM.rlXmfOQT0Z0 plaintext-password "" } level admin } } name-server 1.1.1.1 ntp { server 0.pool.ntp.org { } server 1.pool.ntp.org { } server 2.pool.ntp.org { } } syslog { global { facility all { level info } facility protocols { level debug } } } time-zone UTC }
Feb 12 2019
Feb 8 2019
This bug can't be reproduced in 1.2.0-LTS and 1.2.0-rolling+201902080337, so seems that it was fixed in some of previous releases. Closing ticket.
Feel free to reopen it if the the same behavior will be spotted in one of current releases.
@danhusan , you can send the configuration to [email protected] with the theme "Phabricator T1148". Also, please check if a remote side of BGP peering run in active or passive mode?
Feb 7 2019
Hello @danhusan!
How big is your configuration at all? Can you provide depersonalized config? Which hardware or virtual machine using for VyOS? Can you provide full log of booting?
We can't confidently reproduce this bug. Looks like configuration can't load quickly enough or something like this.
Bug confirmed. The problem is in FRRouting CLI (FRRouting 7.1-dev).
Will see what we can do with this.
Hello, @adestis!
You can use values from 5 to 100. 600 is unsupported in current FRRouting.
Hello, @MrXermon!
We can't reproduce this bug. Maybe, there are some other errors, which can affect route-maps?
If you have the old configuration, can you check update procedure with an update to 1.2.0-LTS and current rolling?
Hello, @rizkidtn!
Is any problems still exist with your configuration or we can close this issue?
Hello, @dongjunbo!
Can you check behavior with new versions? If the problem still exist show us an example of such big configuration, or at least count of objects and their types in this config.
Hello, @bjtangseng!
Can you recheck this with fresh versions? Seems that in 1.2.0-LTS and rolling everything is OK. Also, if there is no VPN logs at all (for example, VPN is not configured), you can see output like in first message and then this will be not a issue.
Hello, @TrueTechy!
As I understand, this situation can exist only if interface is in down state? Can you check this behavior with fresh VyOS version?
Jan 21 2019
OK, things is more clearly now.
If you don't have any L2-filters between eth1 interfaces of VyOS instances I could recommend you first to change configuration to something like this (based on your configuration from first message):
Router 1:
Jan 15 2019
Confirmed. If we try to change variable with multiple values, will be applied only first of them. For example:
net.ipv4.tcp_wmem = 4098 16384 1643392
Hi, @bmtauer!
To be honest, it's looks like you have used bug or some non-typical behavior in 1.1.8 as feature. Your configuration looks strange from the start, so I propose to start investigation of this from detailed description of your task, which you want to solve by this all.
If you can, please, provide information about:
- Connections at eth interfaces on both (master and backup) routers. As I understand, all interfaces is connected to the same L2 segment of network? Explain why connections was made like this - this is not obvious for us now.
- What exactly you want to reach by VRRP? Just make reserved router or this is part of more complex task?
The more we understand your task, the faster we will can help to solve this problem.
Jan 8 2019
I doubt that it will be unecrypted
It can be at 100% unencrypted. If sender accept redirects, then traffic can be routed through another router. :)
By default, more optimal will be leaving send_redirects in enabled state.
I think, that better will be preventing to commit something in vpn ipsec if send_redirects is enabled for any interface, as we can't predict at 100% from which interface will be received traffic, that need to be encrypted with IPSec.
Per-interface option can help in any case, definitely. But we need to leave at least warning to user, where will be clearly said, that with enabled send_redirects some of traffic from interface with this option can be leaked through unencrypted channels.
Can you reproduce this with some other emulator, except SecureCRT and make video with this bug?
Jan 7 2019
OK. So, for now, anyone can use workarounds provided in T1011 or here. And wait for permanent fix in further builds.
Hi, @knozzle !
Provide, please, output of next command, executed in the same terminal as defected show ip bgp summary:
tput cols ; tput lines
Hi, @rizkidtn!
Policy route wouldn't work if it will be assigned to any other interface, except those from which incoming traffic will be received.
Why do you can't set policy to eth1.2400 interface? Is there some problems or errors related with this occurs?
Dec 30 2018
I can confirm, that problem with connection tracking is exist. Reason in this change in Linux kernel. Now, by default, all connection helpers is disabled. You may try to search in your log files something like:
kernel: nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
If you want, you may read more about this here.
So, we need to add all helpers by hand. You may try next workaround. Add this to /config/scripts/vyatta-postconfig-bootup.script:
sleep 10 iptables -t raw -I VYATTA_CT_HELPER 1 -p tcp --dport 1723 -j CT --helper pptp iptables -t raw -I VYATTA_CT_HELPER 2 -p tcp --dport 21 -j CT --helper ftp
Then reboot or, if you want tot apply it without rebooting, just execute all commands in root shell.
Dec 28 2018
I've made some tests...
I have build a lab with next configuration:
In test PC gateway to 10.2.1.0/24 is R2.
In R2 we have next routing tables:
vyos@vyos:~$ show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route
Dec 27 2018
I found. This is VPN settings.
Based on information from Linux IP stack flow diagrams, IPSec policy applying after route decision, and ICMP redirects doing before this. So we can't leave send_redirects=1 on interface, where we receive unencrypted traffic for IPSec.
But, we can:
- Check for firewall send-redirects 'enable' and prevent to commiting vpn ipsec options, when send_redirects is enabled.
- Disable send_redirects only on interfaces, where we expect incoming unencrypted IPSec traffic.
I'm not sure, what is better.
Dec 25 2018
If you just enable and reboot it works too? I've seen this problem at different routers with RC3, RC11 and rolling, but I can't find obvious reason for it.