How to force source address over IPsec site-to-site VPN?
Closed, ObsoletePublic
Actions

Asked by amos.shapira on Jul 20 2016, 8:13 AM.

Details

Hello,

I use VyOS 1.1.7 to connect the office to AWS and a data centre.
We noticed that the VyOS uses its local address of the vti tunnel when it tries to access the data centre:

$ ip -o -4 a
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 192.168.2.254/24 brd 192.168.2.255 scope global eth0\       valid_lft forever preferred_lft forever
...
10: vti1    inet 169.254.33.118/30 scope global vti1\       valid_lft forever preferred_lft forever
11: vti2    inet 169.254.32.106/30 scope global vti2\       valid_lft forever preferred_lft forever
$ ip route get 172.17.70.50
172.17.70.50 via 169.254.32.105 dev vti2  src 169.254.32.106
    cache

I can't reach any server on network 172.17.70.0 from the VyOS server itself unless I force bind the address (e.g. with ssh -b...).
Other hosts behind the VyOS do manage to connect to network 172.17.70.0/24.

The problem with this is that the VyOS also doubles as a local DNS cache, so it has to be able to talk over the IPsec tunnel.

My question - is there a way for me to make the VyOS know that it shouldn't use the vti local address to try to connect anywhere?

Event Timeline

I personally observed mixed reports regarding VTI in general, so just guessing that this should be polished still.
Expecting to see this on 1.2.x releases
Anyway, you can try GRE or even OVPN if it's possible

@EwaldvanGeffen @afics maybe you guys can give some advice on this matter

Thanks but I can't use GRE or OVPN since the other side is AWS virtual gateway (more specifically, I use it as a Virtual CloudHub connecting offices and multiple VPC's).

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPN_CloudHub.html

Got it, I'm pretty sure that solution exist, Just need to spend more time on this.

I was curious about this so I added the 169.254 addresses of my vyos bgp link to my vpc route tables and security groups but I still could not get the traffic to pass. I'm guessing AWS is not allowing traffic from the BGP interface to route. Ideally you would want a way to source all traffic from a dummy interface. IMO

I'm not an expert but my understanding of the definition of 169.254/16 is that it's a "link local" address and shouldn't be used for "real traffic", e.g. AWS uses it to pass EC2 metadata through 169.254.169.254, or see https://tools.ietf.org/html/rfc3927 for how this network is defined.
So I think that the solution should more in the direction of somehow telling VyOS to prefer the local interface address over the 169.254 address.

Linux is going to source traffic for an arbitrary service to the outgoing interface of the selected route. I don't know what facilities might be available to plumb into vyos the notion of binding specific services to a specific interface, (I'd suggest a loopback/dummy interface) and route internally. But the problem is that you wouldn't want this for every service, ie vpn etc. and this is where it gets hairy.

I'm guessing that this will fall into the "Don't run network services on your router" mantra, since it will be so hairy.

Perhaps you can launch the service in /config/scripts/vyatta-postconfig-bootup.script where you have the control to bind it to your desired interface?

@jeffbearer how about policy routing and telling Linux that traffic should only use local 169.254/16 source address if its destination is also on 169.254?

Here is a quick StackExchange answer which might be relevant: http://unix.stackexchange.com/a/188590/12394

Answers

dan.cave
Updated 2,809 Days Ago
Actions

Hi Amos

I seem to recall this when I did the same thing on VyOS to a VPC,

Have you got a BGP session running on that vyos, it seems that there are two ways to setup a local -to-remote gateway to the vpc, because the AWS end is exdpecting to negotiate a BGP session - your diag seems to imply not..

Have you seen this page ? http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VPN.html

When I setup IPSec between a DC with vyos and the AWS VPC/VPG i had to use BGP.

VTi config is really only used between two hosts where you don't want to automatically bind any routing mechanism, and enable BGP/OSPF to do the dynamic routing instead..

Have you tried doing a tcpdump on vti[n] to see if you're seeing any BGP negotiation data on ipsec startup ?

If you have got BPG setup, I would check the VPC/VPG end to see if it can see your local office/vyOS side lan IP

Event Timeline

I have BGP tunnel up over the IPSec tunnel between the VyOS and the AWS Virtual GW.

It receives all the right routes and uses them.

About the AWS guide - yes I'm familiar with it. I've spent the last few months automating and perfecting configuration of VyOS to connect with Virtual Gateways (I use it mainly on EC2 instances which connect the multiple VPC's to each other via an AWS "CloudHub": http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPN_CloudHub.html)

I executed tcpdump with "-i any" and can see the test traffic from the VyOS itself being generated with the 169.254.x.y source address, and other traffic coming from the office vlan behind it successfully being routed to the DNS server with the right (192.168.x.y) source address, and receiving good responses.

This is definitely not a routing issue. The problem is that the VyOS generates traffic using the link-local address. As I stated in the original question, when I can force it to use the right address (of its own eth0 interface, which is on 192.168.x.y), e.g. using "ssh -b ...", it can communicate with any destination over the IPSec tunnel.

I would be happy with a way to tell its dnsmasq server to originate its queries from the eth0 address.

i think the only difference is that I did not use dns cache / dnsmasq, I wasn't familiar with the CloudHub setup although the connectivity requirements were quite different.

Have you tried using iptables to change the source IP to manipulate the source from the DNS masq server with the IP of the eth0 interface ?

That (using iptables to munge the DNS traffic) sounds like a great work-around. I like it because if it works then I can set it as part of the VyOS standard configuration.

I'll try it when I get to the office in a couple of hours.

Still if there is a way to tell dnsmasq to bind to specific interfaces then it would be a nice to have new feature to VyOS.

246tnt
Updated 2,808 Days Ago
Actions

In linux you would type :

ip route change 172.17.70.0/24 via 169.254.32.105 dev vti2 src 1.2.3.4

(replace 1.2.3.4 with whatever IP you want it to use as source ...)

Unfortuately there doesn't seem to be a way to get VyOS to do that. The route configs are done by quagga, and I can't find any way to get quagga to set the "src" parameter. There is a patch from 2006 that adds that function to quagga :

https://lists.quagga.net/pipermail/quagga-dev/attachments/20060629/9fac0e19/sVIPAZ3-0001.obj

But it doesn't seem to have been merged :(

xrpixer
Updated 2,787 Days Ago
Actions

The simple answer is to make your VPN tunnel a routable address.
Make the AWS tunnel address -
192.168.254.1
Make the VYoS tunnel address -
192.168.254.2

Then in AWS, route your 192.168.254.0/30 subnet to your VYoS box.
Thus, VYoS will default source the vti interface which is routable back to the VYoS box.

Alternatively, you could add routes for the 169.254.32 subnet. But that's not necessarily good practice, could be confusing down the line.

New Answer

Answer

This question has been marked as closed, but you can still leave a new answer.

How to force source address over IPsec site-to-site VPN?Closed, ObsoletePublicActions

Details

Event Timeline

Answers

dan.caveUpdated 2,809 Days AgoActions

Event Timeline

246tntUpdated 2,808 Days AgoActions

Event Timeline

xrpixerUpdated 2,787 Days AgoActions

Event Timeline

New Answer

Answer

How to force source address over IPsec site-to-site VPN?
Closed, ObsoletePublic
Actions

dan.cave
Updated 2,809 Days Ago
Actions

246tnt
Updated 2,808 Days Ago
Actions

xrpixer
Updated 2,787 Days Ago
Actions