Page MenuHomePhabricator

VPN/IPSEC/BGP/DPD - unknown bug, tunnel and interfaces up, but no traffic
Closed, WontfixPublicBUG

Description

Hi,

We were using vyos v1.1.7 almost for 1,5 years with no problems at all to connect to AWS VPN. We have 2 vyos in cluster, with each one have 1 connection to AWS, 4 tunnels and configured BGP.

From last month, every 2/3/5 days (random) we experienced that traffic through VPN tunnels just stops, so we upgraded to v1.1.8., thinking that we resolved our problem. But everything is the same, and there are no logs what is wrong (or I don't see it), so I would need some help.

The symptoms are:

  • IPSEC proceses are ok, both VTI's are up/up
  • BGP drops routes for some reason
  • ping doesn't work

We checked with our ISP provider and every time traffic stops, there is an route path calculation on one of links that they connect to AWS (it lasts for few seconds). And few minutes (10, sometimes 30 minutes later) we experience problems.

But after that when we restart ipesc service everything comes up and traffic goes as nothing happen.

I think that Dead-peer-detection is not working as it should or BGP, but can not confirm that.

Can someone help?

Logs before restart:

Feb  1 16:45:35 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5701: replacing stale IPsec SA
Feb  1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #5701 {using isakmp#5699}
Feb  1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: Dead Peer Detection (RFC 3706) enabled
Feb  1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: sent QI2, IPsec SA established {ESP=>0x9bd614a3 <0xc4ac7fe5}
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:45:46 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:46:01 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb  1 16:46:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb  1 16:46:01 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:46:12 vyos01 pluto[8741]: forgetting secrets
Feb  1 16:46:12 vyos01 pluto[8741]: loading secrets from "/etc/ipsec.secrets"
Feb  1 16:46:12 vyos01 pluto[8741]:   loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb  1 16:46:12 vyos01 pluto[8741]:   loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb  1 16:46:12 vyos01 pluto[8741]: loading secrets from "/etc/dmvpn.secrets"
Feb  1 16:46:12 vyos01 pluto[8741]: Changing to directory '/etc/ipsec.d/crls'
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:46:16 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:46:22 vyos01 bgpd[2459]: Performing BGP general scanning
Feb  1 16:46:22 vyos01 bgpd[2459]: scanning IPv4 Unicast routing tables
Feb  1 16:46:22 vyos01 bgpd[2459]: scanning IPv6 Unicast routing tables
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:46:31 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb  1 16:46:31 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb  1 16:46:31 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:46:46 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:46:57 vyos01 pluto[8741]: forgetting secrets
Feb  1 16:46:57 vyos01 pluto[8741]: loading secrets from "/etc/ipsec.secrets"
Feb  1 16:46:57 vyos01 pluto[8741]:   loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb  1 16:46:57 vyos01 pluto[8741]:   loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb  1 16:46:57 vyos01 pluto[8741]: loading secrets from "/etc/dmvpn.secrets"
Feb  1 16:46:57 vyos01 pluto[8741]: Changing to directory '/etc/ipsec.d/crls'
Feb  1 16:46:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5695: IPsec SA expired (superseded by #5703)
Feb  1 16:46:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: ignoring Delete SA payload: PROTO_IPSEC_ESP SA(0x2b4ad0ed) not found (maybe expired)
Feb  1 16:47:01 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb  1 16:47:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb  1 16:47:01 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:47:16 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:47:22 vyos01 pluto[8741]: "[AWS_vti_01]" #5686: ISAKMP SA expired (superseded by #5699)
Feb  1 16:47:22 vyos01 pluto[8741]: packet from 52.58.104.97:500: Informational Exchange is for an unknown (expired?) SA
Feb  1 16:47:22 vyos01 bgpd[2459]: Performing BGP general scanning
Feb  1 16:47:22 vyos01 bgpd[2459]: scanning IPv4 Unicast routing tables
Feb  1 16:47:22 vyos01 bgpd[2459]: scanning IPv6 Unicast routing tables
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:47:31 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb  1 16:47:31 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb  1 16:47:31 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:47:42 vyos01 pluto[8741]: forgetting secrets
Feb  1 16:47:42 vyos01 pluto[8741]: loading secrets from "/etc/ipsec.secrets"
Feb  1 16:47:42 vyos01 pluto[8741]:   loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb  1 16:47:42 vyos01 pluto[8741]:   loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb  1 16:47:42 vyos01 pluto[8741]: loading secrets from "/etc/dmvpn.secrets"
Feb  1 16:47:42 vyos01 pluto[8741]: Changing to directory '/etc/ipsec.d/crls'
Feb  1 16:47:45 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:45 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:47:45 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:47:45 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:45 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:47:45 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:47:46 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:47:47 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA(0xc98c8b65) payload: deleting IPSEC State #5696
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 3, length (excl. header) 2
Feb  1 16:47:55 vyos01 bgpd[2459]: %NOTIFICATION: received from neighbor 169.254.40.61 4/0 (Hold Timer Expired) 0 bytes
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Receive_NOTIFICATION_message (Established->Clearing)
Feb  1 16:47:55 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.61 Down BGP Notification received
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 went from Established to Clearing
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 3, length (excl. header) 2
Feb  1 16:47:55 vyos01 bgpd[2459]: %NOTIFICATION: received from neighbor 169.254.40.169 4/0 (Hold Timer Expired) 0 bytes
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Receive_NOTIFICATION_message (Established->Clearing)
Feb  1 16:47:55 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.169 Down BGP Notification received
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 went from Established to Clearing
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Clearing_Completed (Clearing->Idle)
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 went from Clearing to Idle
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Clearing_Completed (Clearing->Idle)
Feb  1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 went from Clearing to Idle
Feb  1 16:47:55 vyos01 bgpd[2459]: Zebra send: IPv4 route delete 10.50.0.0/16 nexthop 169.254.40.169 metric 100
Feb  1 16:47:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5703: replacing stale IPsec SA
Feb  1 16:47:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5705: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #5703 {using isakmp#5694}
Feb  1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (start timer expire).
Feb  1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] BGP_Start (Idle->Connect)
Feb  1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [Event] Connect start to 169.254.40.61 fd 8
Feb  1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Non blocking connect waiting result
Feb  1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 went from Idle to Connect
Feb  1 16:48:01 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA payload: replace IPSEC State #5704 in 10 seconds
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA(0x5902e492) payload: deleting IPSEC State #5701
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA(0x2f42a949) payload: deleting IPSEC State #5700
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA payload: deleting ISAKMP State #5699
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA payload: replace IPSEC State #5703 in 10 seconds
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA(0x368ef2ed) payload: deleting IPSEC State #5702
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA(0x12b03fa2) payload: deleting IPSEC State #5698
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA(0x29562d00) payload: deleting IPSEC State #5697
Feb  1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA payload: deleting ISAKMP State #5694
Feb  1 16:48:06 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: DPD: Could not find newest phase 1 state

After restart:

Feb  1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (start timer expire).
Feb  1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [FSM] BGP_Start (Idle->Connect)
Feb  1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [Event] Connect start to 169.254.40.169 fd 11
Feb  1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Non blocking connect waiting result
Feb  1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 went from Idle to Connect
Feb  1 16:48:10 vyos01 pluto[8741]: shutting down
Feb  1 16:48:10 vyos01 pluto[8741]: forgetting secrets
Feb  1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_01]": deleting connection
Feb  1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: deleting state (STATE_QUICK_I2)
Feb  1 16:48:10 vyos01 zebra[2449]: interface vti1 index 6 changed <POINTOPOINT,NOARP>.
Feb  1 16:48:10 vyos01 bgpd[2459]: Zebra rcvd: interface vti1 down
Feb  1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_02]": deleting connection
Feb  1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_02]" #5705: deleting state (STATE_QUICK_I1)
Feb  1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_02]" #5703: deleting state (STATE_QUICK_I2)
Feb  1 16:48:10 vyos01 zebra[2449]: interface vti0 index 5 changed <POINTOPOINT,NOARP>.
Feb  1 16:48:10 vyos01 bgpd[2459]: Zebra rcvd: interface vti0 down
Feb  1 16:48:10 vyos01 pluto[8741]: shutting down interface lo/lo ::1
Feb  1 16:48:10 vyos01 pluto[8741]: shutting down interface lo/lo 127.0.0.1
Feb  1 16:48:10 vyos01 pluto[8741]: shutting down interface eth3/eth3 192.168.0.190
Feb  1 16:48:10 vyos01 pluto[8741]: shutting down interface eth2/eth2 192.168.10.190
Feb  1 16:48:10 vyos01 ipsec_starter[8740]: pluto stopped after 160 ms
Feb  1 16:48:10 vyos01 charon: 00[DMN] signal of type SIGINT received. Shutting down
Feb  1 16:48:10 vyos01 ipsec_starter[8740]: charon stopped after 200 ms
Feb  1 16:48:10 vyos01 ipsec_starter[8740]: ipsec starter stopped
Feb  1 16:48:11 vyos01 ipsec_starter[17266]: Starting strongSwan 4.5.2 IPsec [starter]...
Feb  1 16:48:11 vyos01 pluto[17275]: Starting IKEv1 pluto daemon (strongSwan 4.5.2) THREADS SMARTCARD VENDORID CISCO_QUIRKS
Feb  1 16:48:11 vyos01 pluto[17275]: failed to load pkcs11 module '/usr/lib/opensc-pkcs11.so'
Feb  1 16:48:11 vyos01 ipsec_starter[17274]: pluto (17275) started after 20 ms
Feb  1 16:48:11 vyos01 charon: 00[DMN] Starting IKEv2 charon daemon (strongSwan 4.5.2)
Feb  1 16:48:11 vyos01 charon: 00[KNL] listening on interfaces:
Feb  1 16:48:11 vyos01 charon: 00[KNL]   eth3
Feb  1 16:48:11 vyos01 charon: 00[KNL]     192.168.0.190
Feb  1 16:48:11 vyos01 charon: 00[KNL]     fe80::250:56ff:feb7:2648
Feb  1 16:48:11 vyos01 charon: 00[KNL]   eth2
Feb  1 16:48:11 vyos01 charon: 00[KNL]     192.168.10.190
Feb  1 16:48:11 vyos01 charon: 00[KNL]     fe80::250:56ff:feb7:950
Feb  1 16:48:11 vyos01 charon: 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Feb  1 16:48:11 vyos01 charon: 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Feb  1 16:48:11 vyos01 charon: 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Feb  1 16:48:11 vyos01 charon: 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Feb  1 16:48:11 vyos01 charon: 00[CFG] loading crls from '/etc/ipsec.d/crls'
Feb  1 16:48:11 vyos01 charon: 00[CFG] loading secrets from '/etc/ipsec.secrets'
Feb  1 16:48:11 vyos01 charon: 00[CFG]   loaded IKE secret for 192.168.0.190 [AWS_vti_01]
Feb  1 16:48:11 vyos01 charon: 00[CFG]   loaded IKE secret for 192.168.0.190 [AWS_vti_02]
Feb  1 16:48:11 vyos01 charon: 00[CFG] loading secrets from '/etc/dmvpn.secrets'
Feb  1 16:48:11 vyos01 charon: 00[CFG] sql plugin: database URI not set
Feb  1 16:48:11 vyos01 charon: 00[LIB] plugin 'sql': failed to load - sql_plugin_create returned NULL
Feb  1 16:48:11 vyos01 charon: 00[CFG] loaded 0 RADIUS server configurations
Feb  1 16:48:11 vyos01 charon: 00[CFG] HA config misses local/remote address
Feb  1 16:48:11 vyos01 charon: 00[LIB] plugin 'ha': failed to load - ha_plugin_create returned NULL
Feb  1 16:48:11 vyos01 charon: 00[DMN] loaded plugins: test-vectors curl ldap aes des sha1 sha2 md5 random x509 revocation constraints pubkey pkcs1 pgp pem openssl fips-prf gmp agent pkcs11 xcbc hmac ctr ccm gcm attr kernel-netlink resolve socket-raw farp stroke updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc dhcp led addrblock
Feb  1 16:48:11 vyos01 charon: 00[JOB] spawning 16 worker threads
Feb  1 16:48:11 vyos01 pluto[17275]: Changing to directory '/etc/ipsec.d/crls'
Feb  1 16:48:11 vyos01 pluto[17275]: listening for IKE messages
Feb  1 16:48:11 vyos01 pluto[17275]: adding interface eth2/eth2 192.168.10.190:500
Feb  1 16:48:11 vyos01 pluto[17275]: adding interface eth3/eth3 192.168.0.190:500
Feb  1 16:48:11 vyos01 pluto[17275]: adding interface lo/lo 127.0.0.1:500
Feb  1 16:48:11 vyos01 pluto[17275]: adding interface lo/lo ::1:500
Feb  1 16:48:11 vyos01 pluto[17275]: loading secrets from "/etc/ipsec.secrets"
Feb  1 16:48:11 vyos01 pluto[17275]:   loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb  1 16:48:11 vyos01 pluto[17275]:   loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb  1 16:48:11 vyos01 pluto[17275]: loading secrets from "/etc/dmvpn.secrets"
Feb  1 16:48:11 vyos01 ipsec_starter[17274]: charon (17341) started after 20 ms
Feb  1 16:48:11 vyos01 charon: 07[CFG] received stroke: add connection '[AWS_vti_02]'
Feb  1 16:48:11 vyos01 charon: 07[CFG] added configuration '[AWS_vti_02]'
Feb  1 16:48:11 vyos01 charon: 07[CFG] received stroke: add connection '[AWS_vti_01]'
Feb  1 16:48:11 vyos01 pluto[17275]: added connection description "[AWS_vti_02]"
Feb  1 16:48:11 vyos01 charon: 07[CFG] added configuration '[AWS_vti_01]'
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: perl: warning: Setting locale failed.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: perl: warning: Please check that your locale settings:
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: \011LANGUAGE = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: \011LC_ALL = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: \011LANG = "en_US.UTF-8"
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output:     are supported and installed on your system.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: perl: warning: Falling back to the standard locale ("C").
Feb  1 16:48:11 vyos01 zebra[2449]: interface vti0 index 5 changed <UP,POINTOPOINT,RUNNING,NOARP>.
Feb  1 16:48:11 vyos01 bgpd[2459]: Zebra rcvd: interface vti0 up
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: initiating Main Mode
Feb  1 16:48:11 vyos01 pluto[17275]: added connection description "[AWS_vti_01]"
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: perl: warning: Setting locale failed.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: perl: warning: Please check that your locale settings:
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: \011LANGUAGE = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: \011LC_ALL = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: \011LANG = "en_US.UTF-8"
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output:     are supported and installed on your system.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: perl: warning: Falling back to the standard locale ("C").
Feb  1 16:48:11 vyos01 zebra[2449]: interface vti1 index 6 changed <UP,POINTOPOINT,RUNNING,NOARP>.
Feb  1 16:48:11 vyos01 bgpd[2459]: Zebra rcvd: interface vti1 up
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: received Vendor ID payload [Dead Peer Detection]
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: initiating Main Mode
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: received Vendor ID payload [Dead Peer Detection]
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: Peer ID is ID_IPV4_ADDR: '[AWS_vti_01]'
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: ISAKMP SA established
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP {using isakmp#1}
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: Peer ID is ID_IPV4_ADDR: '[AWS_vti_02]'
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: ISAKMP SA established
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP {using isakmp#2}
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: perl: warning: Setting locale failed.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: perl: warning: Please check that your locale settings:
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: \011LANGUAGE = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: \011LC_ALL = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: \011LANG = "en_US.UTF-8"
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output:     are supported and installed on your system.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: perl: warning: Falling back to the standard locale ("C").
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: Dead Peer Detection (RFC 3706) enabled
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: sent QI2, IPsec SA established {ESP=>0x4ff64e1e <0xcbcc8420}
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: perl: warning: Setting locale failed.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: perl: warning: Please check that your locale settings:
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: \011LANGUAGE = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: \011LC_ALL = (unset),
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: \011LANG = "en_US.UTF-8"
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output:     are supported and installed on your system.
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: perl: warning: Falling back to the standard locale ("C").
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: Dead Peer Detection (RFC 3706) enabled
Feb  1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: sent QI2, IPsec SA established {ESP=>0xf52b5d00 <0xc442253d}
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 [FSM] TCP_connection_open (Connect->OpenSent)
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 open active, local address 169.254.40.170
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 sending OPEN, version 4, my as 65000, holdtime 32, id 192.168.0.190
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 send message type 1, length (incl. header) 53
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 went from Connect to OpenSent
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 1, length (excl. header) 34
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv OPEN, version 4, remote-as (in open) 7224, holdtime 30, id 169.254.40.169
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv OPEN w/ OPTION parameter len: 24
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has MultiProtocol Extensions capability (1), length 4
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has MP_EXT CAP for afi/safi: 1/1
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb  1 16:48:12 vyos01 bgpd[2459]: message index 128 [Route Refresh (Old)] found in capcode_str at position 6 (max is 8)
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has Route Refresh (Old) capability (128), length 0
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has Route Refresh capability (2), length 0
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb  1 16:48:12 vyos01 bgpd[2459]: message index 65 [4-octet AS number] found in capcode_str at position 4 (max is 8)
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has 4-octet AS number capability (65), length 4
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm)
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 went from OpenSent to OpenConfirm
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established)
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 went from OpenConfirm to Established
Feb  1 16:48:12 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.169 Up
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:48:13 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb  1 16:48:13 vyos01 bgpd[2459]: 169.254.40.169 send UPDATE 10.0.0.0/26
Feb  1 16:48:13 vyos01 bgpd[2459]: 169.254.40.169 rcvd UPDATE w/ attr: nexthop 169.254.40.169, origin i, metric 100, path 7224
Feb  1 16:48:14 vyos01 kernel: [1316316.879526] e1000 0000:02:01.0 eth2: Reset adapter
Feb  1 16:48:14 vyos01 netplugd[2371]: eth2: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
Feb  1 16:48:14 vyos01 zebra[2449]: interface eth2 index 3 changed <UP,BROADCAST,MULTICAST>.
Feb  1 16:48:14 vyos01 bgpd[2459]: Zebra rcvd: interface eth2 down
Feb  1 16:48:14 vyos01 netplugd[17642]: /etc/netplug/netplug eth2 out -> pid 17642
Feb  1 16:48:14 vyos01 conntrack-tools[7886]: no dedicated links available!
Feb  1 16:48:14 vyos01 netplugd[2371]: eth2: state OUTING pid 17642 exited status 0
Feb  1 16:48:14 vyos01 zebra[2449]: interface eth2 index 3 changed <UP,BROADCAST,RUNNING,MULTICAST>.
Feb  1 16:48:14 vyos01 bgpd[2459]: Zebra rcvd: interface eth2 up
Feb  1 16:48:14 vyos01 netplugd[2371]: eth2: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
Feb  1 16:48:14 vyos01 netplugd[17645]: /etc/netplug/netplug eth2 in -> pid 17645
Feb  1 16:48:14 vyos01 netplugd[2371]: eth2: state INNING pid 17645 exited status 0
Feb  1 16:48:15 vyos01 heartbeat: [8225]: WARN: Late heartbeat: Node 192.168.10.198: interval 12000 ms
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 [FSM] TCP_connection_open (Connect->OpenSent)
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 open active, local address 169.254.40.62
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 sending OPEN, version 4, my as 65000, holdtime 32, id 192.168.0.190
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 send message type 1, length (incl. header) 53
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 went from Connect to OpenSent
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 1, length (excl. header) 34
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv OPEN, version 4, remote-as (in open) 7224, holdtime 30, id 169.254.40.61
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv OPEN w/ OPTION parameter len: 24
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has MultiProtocol Extensions capability (1), length 4
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has MP_EXT CAP for afi/safi: 1/1
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb  1 16:48:16 vyos01 bgpd[2459]: message index 128 [Route Refresh (Old)] found in capcode_str at position 6 (max is 8)
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has Route Refresh (Old) capability (128), length 0
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has Route Refresh capability (2), length 0
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb  1 16:48:16 vyos01 bgpd[2459]: message index 65 [4-octet AS number] found in capcode_str at position 4 (max is 8)
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has 4-octet AS number capability (65), length 4
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm)
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 went from OpenSent to OpenConfirm
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established)
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 went from OpenConfirm to Established
Feb  1 16:48:16 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.61 Up
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:48:16 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:48:17 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb  1 16:48:17 vyos01 bgpd[2459]: 169.254.40.61 send UPDATE 10.0.0.0/26
Feb  1 16:48:17 vyos01 bgpd[2459]: 169.254.40.61 rcvd UPDATE w/ attr: nexthop 169.254.40.61, origin i, metric 200, path 7224
Feb  1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:48:22 vyos01 bgpd[2459]: Performing BGP general scanning
Feb  1 16:48:22 vyos01 bgpd[2459]: scanning IPv4 Unicast routing tables
Feb  1 16:48:22 vyos01 bgpd[2459]: scanning IPv6 Unicast routing tables
Feb  1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:48:31 vyos01 bgpd[2459]: Import timer expired.
Feb  1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb  1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb  1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb  1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb  1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb  1 16:48:43 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb  1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb  1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb  1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb  1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb  1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb  1 16:48:46 vyos01 bgpd[2459]: Import timer expired.

Our configuration:

cluster {
    dead-interval 10000
    group ClusterGroup1 {
        auto-failback false
        primary vyos01
        secondary vyos02
    }
    interface eth0
    interface eth1
    keepalive-interval 2000
    monitor-dead-interval 15000
}


protocols {
    bgp LOCAL_AS {
        maximum-paths {
            ebgp 2
        }
        neighbor 169.254.40.5 {
            ebgp-multihop 2
            nexthop-self
            remote-as REMOTE_AS
            route-map {
            }
            soft-reconfiguration {
                inbound
            }
            timers {
                holdtime 32
                keepalive 10
            }
        }
        neighbor 169.254.41.21 {
            ebgp-multihop 2
            nexthop-self
            remote-as REMOTE_AS
            route-map {
            }
            soft-reconfiguration {
                inbound
            }
            timers {
                holdtime 32
                keepalive 10
            }
        }
        parameters {
            log-neighbor-changes
        }
    }


vpn {
    ipsec {
        auto-update 45
        esp-group AWS {
            compression disable
            lifetime 900
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes128
                hash sha1
            }
        }
        ike-group AWS {
            dead-peer-detection {
                action restart
                interval 15
                timeout 32
            }
            ikev2-reauth no
            key-exchange ikev1
            lifetime 1800
            proposal 1 {
                dh-group 2
                encryption aes128
                hash sha1
            }
        }
        ipsec-interfaces {
            interface eth0
        }
        logging {
            log-modes all
        }
        nat-traversal disable
        site-to-site {
            peer [AWS_IP_VPN2] {
                authentication {
                    mode pre-shared-secret
                }
                connection-type initiate
                ike-group AWS
                ikev2-reauth inherit
                local-address 192.168.x.y
                vti {
                    bind vti0
                    esp-group AWS
                }
            }
            peer [AWS_IP_VPN1] {
                authentication {
                    mode pre-shared-secret
                }
                connection-type initiate
                ike-group AWS
                ikev2-reauth inherit
                local-address 192.168.x.z
                vti {
                    bind vti1
                    esp-group AWS
                }
            }
        }
    }
}

Details

Difficulty level
Unknown (require assessment)
Version
vyos 1.1.8 1.1.7
Why the issue appeared?
Will be filled on close

Event Timeline

mario created this task.Feb 1 2018, 4:55 PM
mario updated the task description. (Show Details)Feb 1 2018, 4:58 PM

So just to be clear,

When the problem happens, you cannot ping directly from one vti interface to the other vti interface across the tunnel?

I had similar issues on 1.1.7.
Could you try increasing the dead peer detection interval and timeout? I used int 60 and timeout 90.

Does running -
reset vpn ipsec-peer x.x.x.x

Fix the issue for a particular tunnel?

mario added a comment.Feb 5 2018, 7:41 AM

Hi xrpixer,

Yes, both vti interface are not able to ping through tunnel.

I have tried that, I experiment with the DPD interval and timeout, from 15-120 s interval, and from 30-240 timeout. The symptoms are the same.

No, "reset vpn ipsec-peer xxx" and "reset vpn" does not help. Only thing that help is when reseting the ipsec service with sudo. After that both tunnels are operational.

syncer triaged this task as Low priority.Feb 27 2018, 2:22 PM
syncer added a subscriber: syncer.

I will suggest perform testing on 1.2 since we not going to patch 1.1.x line anymore

mario added a comment.Feb 27 2018, 2:31 PM

OK, thx

I think that this is a bug that has been resolved in newer versions, but can not confirm.

Regards

We are runing VyOS 1.1.8 with AWS tunnels based on AWS provided config.
It's running for months !

vpn {
    ipsec {
        esp-group AWS {
            compression disable
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes128
                hash sha1
            }
        ike-group AWS {
            dead-peer-detection {
                action restart
                interval 15
                timeout 30
            }
            ikev2-reauth no
            key-exchange ikev1
            lifetime 28800
            proposal 1 {
                dh-group 2
                encryption aes128
                hash sha1
            }
        }
    }
}
syncer changed the task status from Open to On hold.Oct 13 2018, 9:03 AM

Can this be reproduced in 1.2?

mario added a comment.Oct 17 2018, 1:13 PM

Don't know, but we are still experiencing this in 1.1.8. The only solution is to restart IPSEC service. The logs are as above.

We have the same config as provided by AWS.

@mario we not going to fix this in 1.1.x
It will be required to retest with 1.2 same config

@mario is your ike-lifetime correct? That looks really short for an aws tunnel.
Otherwise yeah, I'd try with 1.2.

mario added a comment.Oct 19 2018, 6:51 AM

Hi,
we returned ike-lifetime to AWS specs. Logs above are when we tshoot it.

OK, so I'll upgrade production vyos to 1.2 and retest it.

hagbard claimed this task.Sep 10 2019, 10:05 PM
hagbard added a subscriber: hagbard.

@mario Did you manage to upgrade to 1.2 and if so, do you still have that issue?

mario added a comment.Sep 11 2019, 6:54 AM

Hi @hagbard,

tried with the first version 1.2, problem was still present. After that, decided to get us a physical router/fw because ipsec would stopped without any obvious reason.
It was a long time ago, almost year and a half...

Hi @mario ,

Thanks for your response, did you test a newer image already? There was a lot of work done meanwhile.

mario added a comment.Sep 12 2019, 6:34 AM

No, we don't use vyos in production any more, so I can't tell.

syncer closed this task as Wontfix.Sep 12 2019, 10:08 AM
syncer edited projects, added Rejected; removed VyOS 1.1.x (1.1.8).

Ok, closing as wontfix