Hi,
We were using vyos v1.1.7 almost for 1,5 years with no problems at all to connect to AWS VPN. We have 2 vyos in cluster, with each one have 1 connection to AWS, 4 tunnels and configured BGP.
From last month, every 2/3/5 days (random) we experienced that traffic through VPN tunnels just stops, so we upgraded to v1.1.8., thinking that we resolved our problem. But everything is the same, and there are no logs what is wrong (or I don't see it), so I would need some help.
The symptoms are:
- IPSEC proceses are ok, both VTI's are up/up
- BGP drops routes for some reason
- ping doesn't work
We checked with our ISP provider and every time traffic stops, there is an route path calculation on one of links that they connect to AWS (it lasts for few seconds). And few minutes (10, sometimes 30 minutes later) we experience problems.
But after that when we restart ipesc service everything comes up and traffic goes as nothing happen.
I think that Dead-peer-detection is not working as it should or BGP, but can not confirm that.
Can some
one help?
Logs before restart:
```
Feb 1 16:45:35 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5701: replacing stale IPsec SA
Feb 1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #5701 {using isakmp#5699}
Feb 1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: Dead Peer Detection (RFC 3706) enabled
Feb 1 16:45:36 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: sent QI2, IPsec SA established {ESP=>0x9bd614a3 <0xc4ac7fe5}
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:45:45 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:45:46 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:45:55 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:46:01 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb 1 16:46:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb 1 16:46:01 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:46:05 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:46:12 vyos01 pluto[8741]: forgetting secrets
Feb 1 16:46:12 vyos01 pluto[8741]: loading secrets from "/etc/ipsec.secrets"
Feb 1 16:46:12 vyos01 pluto[8741]: loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb 1 16:46:12 vyos01 pluto[8741]: loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb 1 16:46:12 vyos01 pluto[8741]: loading secrets from "/etc/dmvpn.secrets"
Feb 1 16:46:12 vyos01 pluto[8741]: Changing to directory '/etc/ipsec.d/crls'
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:46:15 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:46:16 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:46:22 vyos01 bgpd[2459]: Performing BGP general scanning
Feb 1 16:46:22 vyos01 bgpd[2459]: scanning IPv4 Unicast routing tables
Feb 1 16:46:22 vyos01 bgpd[2459]: scanning IPv6 Unicast routing tables
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:46:25 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:46:31 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb 1 16:46:31 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb 1 16:46:31 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:46:35 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:46:45 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:46:46 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:46:55 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:46:57 vyos01 pluto[8741]: forgetting secrets
Feb 1 16:46:57 vyos01 pluto[8741]: loading secrets from "/etc/ipsec.secrets"
Feb 1 16:46:57 vyos01 pluto[8741]: loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb 1 16:46:57 vyos01 pluto[8741]: loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb 1 16:46:57 vyos01 pluto[8741]: loading secrets from "/etc/dmvpn.secrets"
Feb 1 16:46:57 vyos01 pluto[8741]: Changing to directory '/etc/ipsec.d/crls'
Feb 1 16:46:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5695: IPsec SA expired (superseded by #5703)
Feb 1 16:46:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: ignoring Delete SA payload: PROTO_IPSEC_ESP SA(0x2b4ad0ed) not found (maybe expired)
Feb 1 16:47:01 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb 1 16:47:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb 1 16:47:01 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:47:05 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:47:15 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:47:16 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:47:22 vyos01 pluto[8741]: "[AWS_vti_01]" #5686: ISAKMP SA expired (superseded by #5699)
Feb 1 16:47:22 vyos01 pluto[8741]: packet from 52.58.104.97:500: Informational Exchange is for an unknown (expired?) SA
Feb 1 16:47:22 vyos01 bgpd[2459]: Performing BGP general scanning
Feb 1 16:47:22 vyos01 bgpd[2459]: scanning IPv4 Unicast routing tables
Feb 1 16:47:22 vyos01 bgpd[2459]: scanning IPv6 Unicast routing tables
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:47:25 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:47:31 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb 1 16:47:31 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb 1 16:47:31 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:47:35 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:47:42 vyos01 pluto[8741]: forgetting secrets
Feb 1 16:47:42 vyos01 pluto[8741]: loading secrets from "/etc/ipsec.secrets"
Feb 1 16:47:42 vyos01 pluto[8741]: loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb 1 16:47:42 vyos01 pluto[8741]: loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb 1 16:47:42 vyos01 pluto[8741]: loading secrets from "/etc/dmvpn.secrets"
Feb 1 16:47:42 vyos01 pluto[8741]: Changing to directory '/etc/ipsec.d/crls'
Feb 1 16:47:45 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:45 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:47:45 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:47:45 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:45 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:47:45 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:47:46 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:47:47 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA(0xc98c8b65) payload: deleting IPSEC State #5696
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 3, length (excl. header) 2
Feb 1 16:47:55 vyos01 bgpd[2459]: %NOTIFICATION: received from neighbor 169.254.40.61 4/0 (Hold Timer Expired) 0 bytes
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Receive_NOTIFICATION_message (Established->Clearing)
Feb 1 16:47:55 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.61 Down BGP Notification received
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 went from Established to Clearing
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 3, length (excl. header) 2
Feb 1 16:47:55 vyos01 bgpd[2459]: %NOTIFICATION: received from neighbor 169.254.40.169 4/0 (Hold Timer Expired) 0 bytes
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Receive_NOTIFICATION_message (Established->Clearing)
Feb 1 16:47:55 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.169 Down BGP Notification received
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 went from Established to Clearing
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Clearing_Completed (Clearing->Idle)
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.61 went from Clearing to Idle
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Clearing_Completed (Clearing->Idle)
Feb 1 16:47:55 vyos01 bgpd[2459]: 169.254.40.169 went from Clearing to Idle
Feb 1 16:47:55 vyos01 bgpd[2459]: Zebra send: IPv4 route delete 10.50.0.0/16 nexthop 169.254.40.169 metric 100
Feb 1 16:47:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5703: replacing stale IPsec SA
Feb 1 16:47:58 vyos01 pluto[8741]: "[AWS_vti_02]" #5705: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #5703 {using isakmp#5694}
Feb 1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (start timer expire).
Feb 1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] BGP_Start (Idle->Connect)
Feb 1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [Event] Connect start to 169.254.40.61 fd 8
Feb 1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Non blocking connect waiting result
Feb 1 16:48:01 vyos01 bgpd[2459]: 169.254.40.61 went from Idle to Connect
Feb 1 16:48:01 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA payload: replace IPSEC State #5704 in 10 seconds
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA(0x5902e492) payload: deleting IPSEC State #5701
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA(0x2f42a949) payload: deleting IPSEC State #5700
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_01]" #5699: received Delete SA payload: deleting ISAKMP State #5699
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA payload: replace IPSEC State #5703 in 10 seconds
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA(0x368ef2ed) payload: deleting IPSEC State #5702
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA(0x12b03fa2) payload: deleting IPSEC State #5698
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA(0x29562d00) payload: deleting IPSEC State #5697
Feb 1 16:48:02 vyos01 pluto[8741]: "[AWS_vti_02]" #5694: received Delete SA payload: deleting ISAKMP State #5694
Feb 1 16:48:06 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: DPD: Could not find newest phase 1 state
```
After restart:
```
Feb 1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (start timer expire).
Feb 1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [FSM] BGP_Start (Idle->Connect)
Feb 1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [Event] Connect start to 169.254.40.169 fd 11
Feb 1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Non blocking connect waiting result
Feb 1 16:48:09 vyos01 bgpd[2459]: 169.254.40.169 went from Idle to Connect
Feb 1 16:48:10 vyos01 pluto[8741]: shutting down
Feb 1 16:48:10 vyos01 pluto[8741]: forgetting secrets
Feb 1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_01]": deleting connection
Feb 1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_01]" #5704: deleting state (STATE_QUICK_I2)
Feb 1 16:48:10 vyos01 zebra[2449]: interface vti1 index 6 changed <POINTOPOINT,NOARP>.
Feb 1 16:48:10 vyos01 bgpd[2459]: Zebra rcvd: interface vti1 down
Feb 1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_02]": deleting connection
Feb 1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_02]" #5705: deleting state (STATE_QUICK_I1)
Feb 1 16:48:10 vyos01 pluto[8741]: "[AWS_vti_02]" #5703: deleting state (STATE_QUICK_I2)
Feb 1 16:48:10 vyos01 zebra[2449]: interface vti0 index 5 changed <POINTOPOINT,NOARP>.
Feb 1 16:48:10 vyos01 bgpd[2459]: Zebra rcvd: interface vti0 down
Feb 1 16:48:10 vyos01 pluto[8741]: shutting down interface lo/lo ::1
Feb 1 16:48:10 vyos01 pluto[8741]: shutting down interface lo/lo 127.0.0.1
Feb 1 16:48:10 vyos01 pluto[8741]: shutting down interface eth3/eth3 192.168.0.190
Feb 1 16:48:10 vyos01 pluto[8741]: shutting down interface eth2/eth2 192.168.10.190
Feb 1 16:48:10 vyos01 ipsec_starter[8740]: pluto stopped after 160 ms
Feb 1 16:48:10 vyos01 charon: 00[DMN] signal of type SIGINT received. Shutting down
Feb 1 16:48:10 vyos01 ipsec_starter[8740]: charon stopped after 200 ms
Feb 1 16:48:10 vyos01 ipsec_starter[8740]: ipsec starter stopped
Feb 1 16:48:11 vyos01 ipsec_starter[17266]: Starting strongSwan 4.5.2 IPsec [starter]...
Feb 1 16:48:11 vyos01 pluto[17275]: Starting IKEv1 pluto daemon (strongSwan 4.5.2) THREADS SMARTCARD VENDORID CISCO_QUIRKS
Feb 1 16:48:11 vyos01 pluto[17275]: failed to load pkcs11 module '/usr/lib/opensc-pkcs11.so'
Feb 1 16:48:11 vyos01 ipsec_starter[17274]: pluto (17275) started after 20 ms
Feb 1 16:48:11 vyos01 charon: 00[DMN] Starting IKEv2 charon daemon (strongSwan 4.5.2)
Feb 1 16:48:11 vyos01 charon: 00[KNL] listening on interfaces:
Feb 1 16:48:11 vyos01 charon: 00[KNL] eth3
Feb 1 16:48:11 vyos01 charon: 00[KNL] 192.168.0.190
Feb 1 16:48:11 vyos01 charon: 00[KNL] fe80::250:56ff:feb7:2648
Feb 1 16:48:11 vyos01 charon: 00[KNL] eth2
Feb 1 16:48:11 vyos01 charon: 00[KNL] 192.168.10.190
Feb 1 16:48:11 vyos01 charon: 00[KNL] fe80::250:56ff:feb7:950
Feb 1 16:48:11 vyos01 charon: 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Feb 1 16:48:11 vyos01 charon: 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Feb 1 16:48:11 vyos01 charon: 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Feb 1 16:48:11 vyos01 charon: 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Feb 1 16:48:11 vyos01 charon: 00[CFG] loading crls from '/etc/ipsec.d/crls'
Feb 1 16:48:11 vyos01 charon: 00[CFG] loading secrets from '/etc/ipsec.secrets'
Feb 1 16:48:11 vyos01 charon: 00[CFG] loaded IKE secret for 192.168.0.190 [AWS_vti_01]
Feb 1 16:48:11 vyos01 charon: 00[CFG] loaded IKE secret for 192.168.0.190 [AWS_vti_02]
Feb 1 16:48:11 vyos01 charon: 00[CFG] loading secrets from '/etc/dmvpn.secrets'
Feb 1 16:48:11 vyos01 charon: 00[CFG] sql plugin: database URI not set
Feb 1 16:48:11 vyos01 charon: 00[LIB] plugin 'sql': failed to load - sql_plugin_create returned NULL
Feb 1 16:48:11 vyos01 charon: 00[CFG] loaded 0 RADIUS server configurations
Feb 1 16:48:11 vyos01 charon: 00[CFG] HA config misses local/remote address
Feb 1 16:48:11 vyos01 charon: 00[LIB] plugin 'ha': failed to load - ha_plugin_create returned NULL
Feb 1 16:48:11 vyos01 charon: 00[DMN] loaded plugins: test-vectors curl ldap aes des sha1 sha2 md5 random x509 revocation constraints pubkey pkcs1 pgp pem openssl fips-prf gmp agent pkcs11 xcbc hmac ctr ccm gcm attr kernel-netlink resolve socket-raw farp stroke updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc dhcp led addrblock
Feb 1 16:48:11 vyos01 charon: 00[JOB] spawning 16 worker threads
Feb 1 16:48:11 vyos01 pluto[17275]: Changing to directory '/etc/ipsec.d/crls'
Feb 1 16:48:11 vyos01 pluto[17275]: listening for IKE messages
Feb 1 16:48:11 vyos01 pluto[17275]: adding interface eth2/eth2 192.168.10.190:500
Feb 1 16:48:11 vyos01 pluto[17275]: adding interface eth3/eth3 192.168.0.190:500
Feb 1 16:48:11 vyos01 pluto[17275]: adding interface lo/lo 127.0.0.1:500
Feb 1 16:48:11 vyos01 pluto[17275]: adding interface lo/lo ::1:500
Feb 1 16:48:11 vyos01 pluto[17275]: loading secrets from "/etc/ipsec.secrets"
Feb 1 16:48:11 vyos01 pluto[17275]: loaded PSK secret for 192.168.0.190 [AWS_vti_01]
Feb 1 16:48:11 vyos01 pluto[17275]: loaded PSK secret for 192.168.0.190 [AWS_vti_02]
Feb 1 16:48:11 vyos01 pluto[17275]: loading secrets from "/etc/dmvpn.secrets"
Feb 1 16:48:11 vyos01 ipsec_starter[17274]: charon (17341) started after 20 ms
Feb 1 16:48:11 vyos01 charon: 07[CFG] received stroke: add connection '[AWS_vti_02]'
Feb 1 16:48:11 vyos01 charon: 07[CFG] added configuration '[AWS_vti_02]'
Feb 1 16:48:11 vyos01 charon: 07[CFG] received stroke: add connection '[AWS_vti_01]'
Feb 1 16:48:11 vyos01 pluto[17275]: added connection description "[AWS_vti_02]"
Feb 1 16:48:11 vyos01 charon: 07[CFG] added configuration '[AWS_vti_01]'
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: perl: warning: Setting locale failed.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: perl: warning: Please check that your locale settings:
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: \011LANGUAGE = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: \011LC_ALL = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: \011LANG = "en_US.UTF-8"
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: are supported and installed on your system.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]": route-client output: perl: warning: Falling back to the standard locale ("C").
Feb 1 16:48:11 vyos01 zebra[2449]: interface vti0 index 5 changed <UP,POINTOPOINT,RUNNING,NOARP>.
Feb 1 16:48:11 vyos01 bgpd[2459]: Zebra rcvd: interface vti0 up
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: initiating Main Mode
Feb 1 16:48:11 vyos01 pluto[17275]: added connection description "[AWS_vti_01]"
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: perl: warning: Setting locale failed.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: perl: warning: Please check that your locale settings:
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: \011LANGUAGE = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: \011LC_ALL = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: \011LANG = "en_US.UTF-8"
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: are supported and installed on your system.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]": route-client output: perl: warning: Falling back to the standard locale ("C").
Feb 1 16:48:11 vyos01 zebra[2449]: interface vti1 index 6 changed <UP,POINTOPOINT,RUNNING,NOARP>.
Feb 1 16:48:11 vyos01 bgpd[2459]: Zebra rcvd: interface vti1 up
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: received Vendor ID payload [Dead Peer Detection]
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: initiating Main Mode
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: received Vendor ID payload [Dead Peer Detection]
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: Peer ID is ID_IPV4_ADDR: '[AWS_vti_01]'
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #1: ISAKMP SA established
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP {using isakmp#1}
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: Peer ID is ID_IPV4_ADDR: '[AWS_vti_02]'
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #2: ISAKMP SA established
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP {using isakmp#2}
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: perl: warning: Setting locale failed.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: perl: warning: Please check that your locale settings:
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: \011LANGUAGE = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: \011LC_ALL = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: \011LANG = "en_US.UTF-8"
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: are supported and installed on your system.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: up-client output: perl: warning: Falling back to the standard locale ("C").
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: Dead Peer Detection (RFC 3706) enabled
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_02]" #3: sent QI2, IPsec SA established {ESP=>0x4ff64e1e <0xcbcc8420}
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: perl: warning: Setting locale failed.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: perl: warning: Please check that your locale settings:
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: \011LANGUAGE = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: \011LC_ALL = (unset),
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: \011LANG = "en_US.UTF-8"
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: are supported and installed on your system.
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: up-client output: perl: warning: Falling back to the standard locale ("C").
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: Dead Peer Detection (RFC 3706) enabled
Feb 1 16:48:11 vyos01 pluto[17275]: "[AWS_vti_01]" #4: sent QI2, IPsec SA established {ESP=>0xf52b5d00 <0xc442253d}
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 [FSM] TCP_connection_open (Connect->OpenSent)
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 open active, local address 169.254.40.170
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 sending OPEN, version 4, my as 65000, holdtime 32, id 192.168.0.190
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 send message type 1, length (incl. header) 53
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 went from Connect to OpenSent
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 1, length (excl. header) 34
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv OPEN, version 4, remote-as (in open) 7224, holdtime 30, id 169.254.40.169
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv OPEN w/ OPTION parameter len: 24
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has MultiProtocol Extensions capability (1), length 4
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has MP_EXT CAP for afi/safi: 1/1
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb 1 16:48:12 vyos01 bgpd[2459]: message index 128 [Route Refresh (Old)] found in capcode_str at position 6 (max is 8)
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has Route Refresh (Old) capability (128), length 0
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has Route Refresh capability (2), length 0
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb 1 16:48:12 vyos01 bgpd[2459]: message index 65 [4-octet AS number] found in capcode_str at position 4 (max is 8)
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 OPEN has 4-octet AS number capability (65), length 4
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm)
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 went from OpenSent to OpenConfirm
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established)
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 went from OpenConfirm to Established
Feb 1 16:48:12 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.169 Up
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:48:12 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:48:13 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb 1 16:48:13 vyos01 bgpd[2459]: 169.254.40.169 send UPDATE 10.0.0.0/26
Feb 1 16:48:13 vyos01 bgpd[2459]: 169.254.40.169 rcvd UPDATE w/ attr: nexthop 169.254.40.169, origin i, metric 100, path 7224
Feb 1 16:48:14 vyos01 kernel: [1316316.879526] e1000 0000:02:01.0 eth2: Reset adapter
Feb 1 16:48:14 vyos01 netplugd[2371]: eth2: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
Feb 1 16:48:14 vyos01 zebra[2449]: interface eth2 index 3 changed <UP,BROADCAST,MULTICAST>.
Feb 1 16:48:14 vyos01 bgpd[2459]: Zebra rcvd: interface eth2 down
Feb 1 16:48:14 vyos01 netplugd[17642]: /etc/netplug/netplug eth2 out -> pid 17642
Feb 1 16:48:14 vyos01 conntrack-tools[7886]: no dedicated links available!
Feb 1 16:48:14 vyos01 netplugd[2371]: eth2: state OUTING pid 17642 exited status 0
Feb 1 16:48:14 vyos01 zebra[2449]: interface eth2 index 3 changed <UP,BROADCAST,RUNNING,MULTICAST>.
Feb 1 16:48:14 vyos01 bgpd[2459]: Zebra rcvd: interface eth2 up
Feb 1 16:48:14 vyos01 netplugd[2371]: eth2: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
Feb 1 16:48:14 vyos01 netplugd[17645]: /etc/netplug/netplug eth2 in -> pid 17645
Feb 1 16:48:14 vyos01 netplugd[2371]: eth2: state INNING pid 17645 exited status 0
Feb 1 16:48:15 vyos01 heartbeat: [8225]: WARN: Late heartbeat: Node 192.168.10.198: interval 12000 ms
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 [FSM] TCP_connection_open (Connect->OpenSent)
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 open active, local address 169.254.40.62
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 sending OPEN, version 4, my as 65000, holdtime 32, id 192.168.0.190
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 send message type 1, length (incl. header) 53
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 went from Connect to OpenSent
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 1, length (excl. header) 34
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv OPEN, version 4, remote-as (in open) 7224, holdtime 30, id 169.254.40.61
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv OPEN w/ OPTION parameter len: 24
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has MultiProtocol Extensions capability (1), length 4
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has MP_EXT CAP for afi/safi: 1/1
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb 1 16:48:16 vyos01 bgpd[2459]: message index 128 [Route Refresh (Old)] found in capcode_str at position 6 (max is 8)
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has Route Refresh (Old) capability (128), length 0
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has Route Refresh capability (2), length 0
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Feb 1 16:48:16 vyos01 bgpd[2459]: message index 65 [4-octet AS number] found in capcode_str at position 4 (max is 8)
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 OPEN has 4-octet AS number capability (65), length 4
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm)
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 went from OpenSent to OpenConfirm
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established)
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 went from OpenConfirm to Established
Feb 1 16:48:16 vyos01 bgpd[2459]: %ADJCHANGE: neighbor 169.254.40.61 Up
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:48:16 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:48:16 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:48:17 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (routeadv timer expire)
Feb 1 16:48:17 vyos01 bgpd[2459]: 169.254.40.61 send UPDATE 10.0.0.0/26
Feb 1 16:48:17 vyos01 bgpd[2459]: 169.254.40.61 rcvd UPDATE w/ attr: nexthop 169.254.40.61, origin i, metric 200, path 7224
Feb 1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:48:22 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:48:22 vyos01 bgpd[2459]: Performing BGP general scanning
Feb 1 16:48:22 vyos01 bgpd[2459]: scanning IPv4 Unicast routing tables
Feb 1 16:48:22 vyos01 bgpd[2459]: scanning IPv6 Unicast routing tables
Feb 1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:48:26 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:48:31 vyos01 bgpd[2459]: Import timer expired.
Feb 1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:48:32 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:48:36 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (keepalive timer expire)
Feb 1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 sending KEEPALIVE
Feb 1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 send message type 4, length (incl. header) 19
Feb 1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 rcv message type 4, length (excl. header) 0
Feb 1 16:48:42 vyos01 bgpd[2459]: 169.254.40.169 KEEPALIVE rcvd
Feb 1 16:48:43 vyos01 bgpd[2459]: 169.254.40.169 [FSM] Timer (routeadv timer expire)
Feb 1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 [FSM] Timer (keepalive timer expire)
Feb 1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 sending KEEPALIVE
Feb 1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 send message type 4, length (incl. header) 19
Feb 1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 rcv message type 4, length (excl. header) 0
Feb 1 16:48:46 vyos01 bgpd[2459]: 169.254.40.61 KEEPALIVE rcvd
Feb 1 16:48:46 vyos01 bgpd[2459]: Import timer expired.
```
Our configuration:
```
cluster {
dead-interval 10000
group ClusterGroup1 {
auto-failback false
primary vyos01
secondary vyos02
}
interface eth0
interface eth1
keepalive-interval 2000
monitor-dead-interval 15000
}
protocols {
bgp LOCAL_AS {
maximum-paths {
ebgp 2
}
neighbor 169.254.40.5 {
ebgp-multihop 2
nexthop-self
remote-as REMOTE_AS
route-map {
}
soft-reconfiguration {
inbound
}
timers {
holdtime 32
keepalive 10
}
}
neighbor 169.254.41.21 {
ebgp-multihop 2
nexthop-self
remote-as REMOTE_AS
route-map {
}
soft-reconfiguration {
inbound
}
timers {
holdtime 32
keepalive 10
}
}
parameters {
log-neighbor-changes
}
}
vpn {
ipsec {
auto-update 45
esp-group AWS {
compression disable
lifetime 900
mode tunnel
pfs enable
proposal 1 {
encryption aes128
hash sha1
}
}
ike-group AWS {
dead-peer-detection {
action restart
interval 15
timeout 32
}
ikev2-reauth no
key-exchange ikev1
lifetime 1800
proposal 1 {
dh-group 2
encryption aes128
hash sha1
}
}
ipsec-interfaces {
interface eth0
}
logging {
log-modes all
}
nat-traversal disable
site-to-site {
peer [AWS_IP_VPN2] {
authentication {
mode pre-shared-secret
}
connection-type initiate
ike-group AWS
ikev2-reauth inherit
local-address 192.168.x.y
vti {
bind vti0
esp-group AWS
}
}
peer [AWS_IP_VPN1] {
authentication {
mode pre-shared-secret
}
connection-type initiate
ike-group AWS
ikev2-reauth inherit
local-address 192.168.x.z
vti {
bind vti1
esp-group AWS
}
}
}
}
}
```