I use VyOS 1.1.7 to connect the office to AWS and a data centre.
We noticed that the VyOS uses its local address of the vti tunnel when it tries to access the data centre:
$ ip -o -4 a 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 2: eth0 inet 192.168.2.254/24 brd 192.168.2.255 scope global eth0\ valid_lft forever preferred_lft forever ... 10: vti1 inet 169.254.33.118/30 scope global vti1\ valid_lft forever preferred_lft forever 11: vti2 inet 169.254.32.106/30 scope global vti2\ valid_lft forever preferred_lft forever $ ip route get 172.17.70.50 172.17.70.50 via 169.254.32.105 dev vti2 src 169.254.32.106 cache
I can't reach any server on network 172.17.70.0 from the VyOS server itself unless I force bind the address (e.g. with ssh -b...).
Other hosts behind the VyOS do manage to connect to network 172.17.70.0/24.
The problem with this is that the VyOS also doubles as a local DNS cache, so it has to be able to talk over the IPsec tunnel.
My question - is there a way for me to make the VyOS know that it shouldn't use the vti local address to try to connect anywhere?
Thanks but I can't use GRE or OVPN since the other side is AWS virtual gateway (more specifically, I use it as a Virtual CloudHub connecting offices and multiple VPC's).
I was curious about this so I added the 169.254 addresses of my vyos bgp link to my vpc route tables and security groups but I still could not get the traffic to pass. I'm guessing AWS is not allowing traffic from the BGP interface to route. Ideally you would want a way to source all traffic from a dummy interface. IMO
I'm not an expert but my understanding of the definition of 169.254/16 is that it's a "link local" address and shouldn't be used for "real traffic", e.g. AWS uses it to pass EC2 metadata through 169.254.169.254, or see https://tools.ietf.org/html/rfc3927 for how this network is defined.
So I think that the solution should more in the direction of somehow telling VyOS to prefer the local interface address over the 169.254 address.
Linux is going to source traffic for an arbitrary service to the outgoing interface of the selected route. I don't know what facilities might be available to plumb into vyos the notion of binding specific services to a specific interface, (I'd suggest a loopback/dummy interface) and route internally. But the problem is that you wouldn't want this for every service, ie vpn etc. and this is where it gets hairy.
I'm guessing that this will fall into the "Don't run network services on your router" mantra, since it will be so hairy.
Perhaps you can launch the service in /config/scripts/vyatta-postconfig-bootup.script where you have the control to bind it to your desired interface?
I seem to recall this when I did the same thing on VyOS to a VPC,
Have you got a BGP session running on that vyos, it seems that there are two ways to setup a local -to-remote gateway to the vpc, because the AWS end is exdpecting to negotiate a BGP session - your diag seems to imply not..
Have you seen this page ? http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VPN.html
When I setup IPSec between a DC with vyos and the AWS VPC/VPG i had to use BGP.
VTi config is really only used between two hosts where you don't want to automatically bind any routing mechanism, and enable BGP/OSPF to do the dynamic routing instead..
Have you tried doing a tcpdump on vti[n] to see if you're seeing any BGP negotiation data on ipsec startup ?
If you have got BPG setup, I would check the VPC/VPG end to see if it can see your local office/vyOS side lan IP
I have BGP tunnel up over the IPSec tunnel between the VyOS and the AWS Virtual GW.
It receives all the right routes and uses them.
About the AWS guide - yes I'm familiar with it. I've spent the last few months automating and perfecting configuration of VyOS to connect with Virtual Gateways (I use it mainly on EC2 instances which connect the multiple VPC's to each other via an AWS "CloudHub": http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPN_CloudHub.html)
I executed tcpdump with "-i any" and can see the test traffic from the VyOS itself being generated with the 169.254.x.y source address, and other traffic coming from the office vlan behind it successfully being routed to the DNS server with the right (192.168.x.y) source address, and receiving good responses.
This is definitely not a routing issue. The problem is that the VyOS generates traffic using the link-local address. As I stated in the original question, when I can force it to use the right address (of its own eth0 interface, which is on 192.168.x.y), e.g. using "ssh -b ...", it can communicate with any destination over the IPSec tunnel.
I would be happy with a way to tell its dnsmasq server to originate its queries from the eth0 address.
i think the only difference is that I did not use dns cache / dnsmasq, I wasn't familiar with the CloudHub setup although the connectivity requirements were quite different.
Have you tried using iptables to change the source IP to manipulate the source from the DNS masq server with the IP of the eth0 interface ?
That (using iptables to munge the DNS traffic) sounds like a great work-around. I like it because if it works then I can set it as part of the VyOS standard configuration.
I'll try it when I get to the office in a couple of hours.
Still if there is a way to tell dnsmasq to bind to specific interfaces then it would be a nice to have new feature to VyOS.
In linux you would type :
ip route change 172.17.70.0/24 via 169.254.32.105 dev vti2 src 184.108.40.206
(replace 220.127.116.11 with whatever IP you want it to use as source ...)
Unfortuately there doesn't seem to be a way to get VyOS to do that. The route configs are done by quagga, and I can't find any way to get quagga to set the "src" parameter. There is a patch from 2006 that adds that function to quagga :
But it doesn't seem to have been merged :(
Thanks for your research and findings.
My little experience with IPSec implementations on Linux (back from when I tried to do this on Ubuntu) is that it implements its own stack for routing etc.
It sounds like I might still be able to either "inject" an extra parameter to dnsmasq through a file under /etc/dnsmasq.d or use iptables to change the source address.
As far as I understand it, sending traffic with a source address of 169.254/16 to a destination other than 169.254/16 doesn't make sense does it?
The simple answer is to make your VPN tunnel a routable address.
Make the AWS tunnel address -
Make the VYoS tunnel address -
Then in AWS, route your 192.168.254.0/30 subnet to your VYoS box.
Thus, VYoS will default source the vti interface which is routable back to the VYoS box.
Alternatively, you could add routes for the 169.254.32 subnet. But that's not necessarily good practice, could be confusing down the line.
The 169.254/16 is a special link-local subnet which by definition can't be routed outside the immediate link (as far as I follow it). See https://tools.ietf.org/html/rfc3927 for authoritative definition.
My setup has changed a bit since I asked this question - the datacentre is obsolete (we are almost ready to shut it down) and what I'd like now is to cache queries to Route53, though this seems to be less of an issue since the response from our local AWS region seems to be fast enough.