Page MenuHomeVyOS Platform

IPSec VTI tunnels are deleted after rekey and dangling around as A/D
Closed, ResolvedPublicBUG

Description

I use two VyOS Appliances which connect to an Active-Active AZURE VPNGw type1.

In total I have 8 IPSec tunnels per VyOS Appliance (two tunnels per AZURE VPN Gateway) and 4 VPN Gateways in total.

I consulated not only the Azure example configuration for Cisco and Ubiquity EdgeOS but also the guide at https://cloudnetworking.io/2019/08/21/azure-vpn-vyos/

There are several posts describing this issue

I now work-arount the problem using a custom shell script which checks the tunnel A/D state and resets it on demand

set system task-scheduler task azure-reset executable path '/root/reset_azure.sh'
set system task-scheduler task azure-reset interval '1m'
vyos@vyos:~$ cat /root/reset_azure.sh
#!/bin/vbash
source /opt/vyatta/etc/functions/script-template

for vti in $(run show interfaces vti | grep "A/D" | awk {'print $1}')
do
  tunnel=$(run show configuration commands | grep "bind '$vti'" | awk {'print $6}')
  logger "Resetting IPSec tunnel $vti -> $tunnel"
  run reset vpn ipsec-peer $tunnel
done

exit

When looking at the logs when the hangup occured the last time one can see that reauth works perfect on vti42 but vti22 does not come up again and the script above kicks in:

Dec 14 13:46:01 VMU-02-AZURE CRON[27183]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 14 13:46:01 VMU-02-AZURE CRON[27184]: (root) CMD (sg vyattacfg "/root/reset_azure.sh")
Dec 14 13:46:01 VMU-02-AZURE sg[27184]: user 'root' (login 'root' on ???) switched to group 'vyattacfg'
Dec 14 13:46:01 VMU-02-AZURE sg[27184]: user 'root' (login 'root' on ???) returned to group 'root'
Dec 14 13:46:01 VMU-02-AZURE CRON[27183]: pam_unix(cron:session): session closed for user root
Dec 14 13:46:25 VMU-02-AZURE charon[2247]: 07[IKE] reauthenticating IKE_SA peer-xxx.xxx.229.18-tunnel-vti[71]
Dec 14 13:46:25 VMU-02-AZURE charon[2247]: 07[IKE] deleting IKE_SA peer-xxx.xxx.229.18-tunnel-vti[71] between zzz.zzz.32.190[zzz.zzz.32.190]...xxx.xxx.229.18[xxx.xxx.229.18]
Dec 14 13:46:25 VMU-02-AZURE charon[2247]: 06[IKE] IKE_SA deleted
Dec 14 13:46:25 VMU-02-AZURE sudo[27376]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/ip link set vti42 down
Dec 14 13:46:25 VMU-02-AZURE sudo[27376]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 14 13:46:25 VMU-02-AZURE netplugd[974]: vti42: ignoring event
Dec 14 13:46:25 VMU-02-AZURE sudo[27376]: pam_unix(sudo:session): session closed for user root
Dec 14 13:46:25 VMU-02-AZURE charon[2247]: 06[IKE] initiating IKE_SA peer-xxx.xxx.229.18-tunnel-vti[78] to xxx.xxx.229.18
Dec 14 13:46:25 VMU-02-AZURE charon[2247]: 05[IKE] establishing CHILD_SA peer-xxx.xxx.229.18-tunnel-vti{101720} reqid 5
Dec 14 13:46:25 VMU-02-AZURE charon[2247]: 04[IKE] IKE_SA peer-xxx.xxx.229.18-tunnel-vti[78] established between zzz.zzz.32.190[zzz.zzz.32.190]...xxx.xxx.229.18[xxx.xxx.229.18]
Dec 14 13:46:25 VMU-02-AZURE charon[2247]: 04[IKE] CHILD_SA peer-xxx.xxx.229.18-tunnel-vti{101720} established with SPIs c8023ec0_i b41943aa_o and TS 0.0.0.0/0 === 0.0.0.0/0
Dec 14 13:46:25 VMU-02-AZURE sudo[27388]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/ip link set vti42 up
Dec 14 13:46:25 VMU-02-AZURE sudo[27388]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 14 13:46:25 VMU-02-AZURE netplugd[974]: vti42: ignoring event
Dec 14 13:46:25 VMU-02-AZURE sudo[27388]: pam_unix(sudo:session): session closed for user root
Dec 14 13:46:39 VMU-02-AZURE charon[2247]: 15[IKE] reauthenticating IKE_SA peer-yyy.yyy.89.238-tunnel-vti[74] actively
Dec 14 13:46:39 VMU-02-AZURE charon[2247]: 15[IKE] deleting IKE_SA peer-yyy.yyy.89.238-tunnel-vti[74] between zzz.zzz.32.190[zzz.zzz.32.190]...yyy.yyy.89.238[yyy.yyy.89.238]
Dec 14 13:46:39 VMU-02-AZURE charon[2247]: 08[IKE] IKE_SA deleted
Dec 14 13:46:39 VMU-02-AZURE sudo[27398]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/ip link set vti22 down
Dec 14 13:46:39 VMU-02-AZURE sudo[27398]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 14 13:46:39 VMU-02-AZURE netplugd[974]: vti22: ignoring event
Dec 14 13:46:39 VMU-02-AZURE sudo[27398]: pam_unix(sudo:session): session closed for user root
Dec 14 13:46:39 VMU-02-AZURE charon[2247]: 08[IKE] initiating IKE_SA peer-yyy.yyy.89.238-tunnel-vti[79] to yyy.yyy.89.238
Dec 14 13:46:39 VMU-02-AZURE charon[2247]: 14[IKE] establishing CHILD_SA peer-yyy.yyy.89.238-tunnel-vti{101721} reqid 7
Dec 14 13:46:40 VMU-02-AZURE charon[2247]: 10[IKE] IKE_SA peer-yyy.yyy.89.238-tunnel-vti[79] established between zzz.zzz.32.190[zzz.zzz.32.190]...yyy.yyy.89.238[yyy.yyy.89.238]
Dec 14 13:46:40 VMU-02-AZURE charon[2247]: 10[IKE] CHILD_SA peer-yyy.yyy.89.238-tunnel-vti{101721} established with SPIs ce67c986_i 7890fcd6_o and TS 0.0.0.0/0 === 0.0.0.0/0
Dec 14 13:46:40 VMU-02-AZURE sudo[27410]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/ip link set vti22 up
Dec 14 13:46:40 VMU-02-AZURE sudo[27410]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 14 13:46:40 VMU-02-AZURE netplugd[974]: vti22: ignoring event
Dec 14 13:46:40 VMU-02-AZURE sudo[27410]: pam_unix(sudo:session): session closed for user root
Dec 14 13:46:40 VMU-02-AZURE charon[2247]: 06[IKE] closing CHILD_SA peer-yyy.yyy.89.238-tunnel-vti{101712} with SPIs c24cfd41_i (364 bytes) 58386366_o (2338 bytes) and TS 0.0.0.0/0 === 0.0.0.0/0
Dec 14 13:46:40 VMU-02-AZURE sudo[27420]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/ip link set vti22 down
Dec 14 13:46:40 VMU-02-AZURE sudo[27420]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 14 13:46:40 VMU-02-AZURE netplugd[974]: vti22: ignoring event
Dec 14 13:46:40 VMU-02-AZURE sudo[27420]: pam_unix(sudo:session): session closed for user root
Dec 14 13:46:40 VMU-02-AZURE charon[2247]: 08[IKE] deleting IKE_SA peer-yyy.yyy.89.238-tunnel-vti[73] between zzz.zzz.32.190[zzz.zzz.32.190]...yyy.yyy.89.238[yyy.yyy.89.238]
Dec 14 13:46:40 VMU-02-AZURE charon[2247]: 08[IKE] IKE_SA deleted
Dec 14 13:46:41 VMU-02-AZURE ntpd[1634]: Deleting interface #26 vti22, fe80::200:5efe:50f6:20be#123, interface stats: received=0, sent=0, dropped=0, active_time=106678 secs
Dec 14 13:46:41 VMU-02-AZURE ntpd[1634]: peers refreshed
Dec 14 13:47:01 VMU-02-AZURE CRON[27422]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 14 13:47:01 VMU-02-AZURE CRON[27423]: (root) CMD (sg vyattacfg "/root/reset_azure.sh")
Dec 14 13:47:01 VMU-02-AZURE sg[27423]: user 'root' (login 'root' on ???) switched to group 'vyattacfg'
Dec 14 13:47:02 VMU-02-AZURE root[27718]: Resetting IPSec tunnel vti22 -> yyy.yyy.89.238

I have labbed the issue (and hooked a tcpdump on it) but unfortunately no success yet.

On first glance it looks like a race condition as ip link down is called again after the new link is up.

Details

Difficulty level
Unknown (require assessment)
Version
1.2.3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

c-po triaged this task as Unbreak Now! priority.Dec 14 2019, 1:55 PM
c-po updated the task description. (Show Details)

@syncer why remove 1.2? Bug should be fixed in 1.2.x bugfix releases

@c-po this is IKEv2?
can you increase log level?

set vpn ipsec logging log-level '2'
set vpn ipsec logging log-modes 'ike'
set vpn ipsec logging log-modes 'net'
set vpn ipsec logging log-modes 'knl'

Its IKEv2, correct. Loglevel increased

The issue can be reproduced by the following configuration:

VPN GW1 (initiator)

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall ip-src-route 'disable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall log-martians 'enable'
set firewall options interface vti0 adjust-mss '1350'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'

set interfaces dummy dum0 address '172.18.254.202/32'
set interfaces ethernet eth0 address '172.18.202.10/24'
set interfaces vti vti0

set protocols bgp 65000 address-family ipv4-unicast network 10.0.0.0/24
set protocols bgp 65000 neighbor 172.18.254.203 remote-as '65001'
set protocols bgp 65000 neighbor 172.18.254.203 update-source '172.18.254.202'
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 timers holdtime '30'
set protocols bgp 65000 timers keepalive '2'
set protocols static interface-route 172.18.254.203/32 next-hop-interface vti0
set protocols static route 0.0.0.0/0 next-hop 172.18.202.254 distance '10'

set vpn ipsec esp-group AZURE compression 'disable'
set vpn ipsec esp-group AZURE lifetime '90'
set vpn ipsec esp-group AZURE mode 'tunnel'
set vpn ipsec esp-group AZURE pfs 'disable'
set vpn ipsec esp-group AZURE proposal 1 encryption 'aes256'
set vpn ipsec esp-group AZURE proposal 1 hash 'sha1'
set vpn ipsec ike-group AZURE dead-peer-detection action 'restart'
set vpn ipsec ike-group AZURE dead-peer-detection interval '2'
set vpn ipsec ike-group AZURE dead-peer-detection timeout '20'
set vpn ipsec ike-group AZURE ikev2-reauth 'yes'
set vpn ipsec ike-group AZURE key-exchange 'ikev2'
set vpn ipsec ike-group AZURE lifetime '90'
set vpn ipsec ike-group AZURE proposal 1 dh-group '2'
set vpn ipsec ike-group AZURE proposal 1 encryption 'aes256'
set vpn ipsec ike-group AZURE proposal 1 hash 'sha1'

set vpn ipsec ipsec-interfaces interface 'eth0'

set vpn ipsec site-to-site peer 172.18.203.10 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer 172.18.203.10 authentication pre-shared-secret 'asdf1234567890'
set vpn ipsec site-to-site peer 172.18.203.10 connection-type 'initiate'
set vpn ipsec site-to-site peer 172.18.203.10 ike-group 'AZURE'
set vpn ipsec site-to-site peer 172.18.203.10 ikev2-reauth 'yes'
set vpn ipsec site-to-site peer 172.18.203.10 local-address '172.18.202.10'
set vpn ipsec site-to-site peer 172.18.203.10 vti bind 'vti0'
set vpn ipsec site-to-site peer 172.18.203.10 vti esp-group 'AZURE'

VPN GW2 (responder)

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall ip-src-route 'disable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall log-martians 'enable'
set firewall options interface vti0 adjust-mss '1350'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'

set interfaces dummy dum0 address '172.18.254.203/32'
set interfaces ethernet eth0 address '172.18.203.10/24'
set interfaces vti vti0

set protocols bgp 65001 address-family ipv4-unicast network 10.0.1.0/24
set protocols bgp 65001 neighbor 172.18.254.202 remote-as '65000'
set protocols bgp 65001 neighbor 172.18.254.202 update-source '172.18.254.203'
set protocols bgp 65001 parameters log-neighbor-changes
set protocols bgp 65001 timers holdtime '30'
set protocols bgp 65001 timers keepalive '2'
set protocols static interface-route 172.18.254.202/32 next-hop-interface vti0
set protocols static route 0.0.0.0/0 next-hop 172.18.203.254 distance '10'

set service ssh disable-host-validation
set service ssh port '22'

set vpn ipsec esp-group AZURE compression 'disable'
set vpn ipsec esp-group AZURE lifetime '1800'
set vpn ipsec esp-group AZURE mode 'tunnel'
set vpn ipsec esp-group AZURE pfs 'disable'
set vpn ipsec esp-group AZURE proposal 1 encryption 'aes256'
set vpn ipsec esp-group AZURE proposal 1 hash 'sha1'
set vpn ipsec ike-group AZURE dead-peer-detection action 'restart'
set vpn ipsec ike-group AZURE dead-peer-detection interval '15'
set vpn ipsec ike-group AZURE dead-peer-detection timeout '30'
set vpn ipsec ike-group AZURE ikev2-reauth 'yes'
set vpn ipsec ike-group AZURE key-exchange 'ikev2'
set vpn ipsec ike-group AZURE lifetime '1800'
set vpn ipsec ike-group AZURE proposal 1 dh-group '2'
set vpn ipsec ike-group AZURE proposal 1 encryption 'aes256'
set vpn ipsec ike-group AZURE proposal 1 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec site-to-site peer 172.18.202.10 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer 172.18.202.10 authentication pre-shared-secret 'asdf1234567890'
set vpn ipsec site-to-site peer 172.18.202.10 connection-type 'respond'
set vpn ipsec site-to-site peer 172.18.202.10 ike-group 'AZURE'
set vpn ipsec site-to-site peer 172.18.202.10 ikev2-reauth 'inherit'
set vpn ipsec site-to-site peer 172.18.202.10 local-address '172.18.203.10'
set vpn ipsec site-to-site peer 172.18.202.10 vti bind 'vti0'
set vpn ipsec site-to-site peer 172.18.202.10 vti esp-group 'AZURE'

After some time and re-key retires you will see interfacte vti0 beeing A/D when running show interfaces op-mode command

Maybe enabling "Make-before-break" re-keying can solve this? https://wiki.strongswan.org/projects/strongswan/wiki/ExpiryRekey

@Dmitry I had it once more on the production system - see attached log with the above loggin features enabled.

Need to check again with 1.3, as may be solved by: https://phabricator.vyos.net/T1291

I have tested the fix in https://github.com/vyos/vyatta-cfg-vpn/pull/31 successfully on VyOS 1.2.5 with the hotfix mentiones. Happy to see this in 1.2.6

erkin set Issue type to Bug (incorrect behavior).Aug 31 2021, 6:06 PM