DNS forwarding service or nat forwarding bug
Closed, WontfixPublicBUG
Actions

Assigned To

Authored By

	Smiley
	Mar 7 2018, 5:10 PM

Description

Hi,

I have something strange on some VyOS instances, in AWS (v1.1.7 but same problem occurs after upgrading them to v1.1.8, and on another instance initialy installed in v1.1.8)
When I set dns forwarding service, for unknown reason, the vyos sometimes correctly forward the incoming request to the dns server as expected, but sometimes it doesn't ! (perhaps it works one time over 3 attemps...). When I tshark on the vyos, we can see the incoming packet, and some times it is forwarded as expected, but sometimes nothing happens, nothing is forwarded, so the client send two or three other request and then ended by a "connection timed out" error message)

What is curious is that if I replace this service dns forwarding by NAT rules (destination + source), to do the same job, I get the same issue: some requests are corretly NATed, but some others are not.
So I tried to reproduce the problem, however on a clean VyoS with only this nat rule or dns forwarding service set, everything is working correctly: every requests are forwarded as expected.
So, I'm not sure but it seems that the common point of all the problematic vyos instances is that they all handle vpn ipsec tunnels (so I can't easily reproduce the problem to check if it's always the case or not). No firewall rules are implemented on these instances...

Does someone get any idea ?
Many thanks,
Best regards

Details

Difficulty level: Unknown (require assessment)
Version: 1.1.8
Why the issue appeared?: Will be filled on close

Event Timeline

Smiley created this task.Mar 7 2018, 5:10 PM

By default the DNS forwarder will cache recent responses. Have you disabled DNS caching on the forwarding service with the following configuration?

set service dns forwarding cache-size '0'

Yes, I thought about that too, but with or without setting the dns
cache-size to 0, I have the same result !

We'll need some more information.

Have you allowed both UDP and TCP in the local firewall policy for port 53?
Is there an example domain that only works sometimes?
Can you expand your capture to show the DNS request type and provide a paste of the failure?

There are no firewall rules set, and no firewall rulset set to the interface on the affected VyOS instances
The problems seems to occur whatever the name resolution request is
Yes, see below

Well, in fact I've found the problem, regarding dns service forwarding at least: this is because with this kind of configuration :

vyos@VyOS-AMI# show service dns
forwarding {

cache-size 0
dhcp eth0
listen-on eth0

}

The dns forwarding works well from local network computers. However, for computers remotely connected thru a vpn tunnel, it's necessary to add : listen-on vti0 and listen-on vti1 (for example). With this, the requests are all well forwarded.
However, what is strange then is that it is not the case for all requests (they normally should all be forwarded, or none, but why only some of them ?)

Here is the requests from a test computer: (3 dns requests, but only the third got an answer):
(DNS request from 172.21.1.75 to vyos IP 172.31.0.52, which should forward to 172.31.0.2)

[user@ip-172-21-1-75 ~]$ nslookup www.google.com 172.31.0.52
;; connection timed out; trying next origin
;; connection timed out; no servers could be reached

[user@ip-172-21-1-75 ~]$ nslookup www.google.com 172.31.0.52
;; connection timed out; trying next origin
Server: 172.31.0.52
Address: 172.31.0.52#53

server can't find www.google.com: NXDOMAIN

[user@ip-172-21-1-75 ~]$ nslookup www.google.com 172.31.0.52
Server: 172.31.0.52
Address: 172.31.0.52#53

Non-authoritative answer:
Name: www.google.com
Address: 209.85.202.104
Name: www.google.com
Address: 209.85.202.105
Name: www.google.com
Address: 209.85.202.106
Name: www.google.com
Address: 209.85.202.147
Name: www.google.com
Address: 209.85.202.99
Name: www.google.com
Address: 209.85.202.103

And here is the capture from vyos :

vyos@VyOS-AMI# tshark -i any -f "((udp port 53) or (tcp port 53)) and (host 172.31.0.221 or host 172.31.0.2 or host 172.31.0.52)"
Capturing on Pseudo-device that captures on all interfaces

0.000000  172.21.1.75 -> 172.31.0.52  DNS Standard query A www.google.com
5.000036  172.21.1.75 -> 172.31.0.52  DNS Standard query A www.google.com

10.001531 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
15.000621 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com.eu-west-1.compute.internal
26.494492 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
31.494560 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
36.494685 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
41.494923 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com.eu-west-1.compute.internal
41.495026 172.31.0.52 -> 172.31.0.2 DNS Standard query A www.google.com.eu-west-1.compute.internal
41.496165 172.31.0.2 -> 172.31.0.52 DNS Standard query response, No such name
41.496203 172.31.0.52 -> 172.21.1.75 DNS Standard query response, No such name
44.900376 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
44.900485 172.31.0.52 -> 172.31.0.2 DNS Standard query A www.google.com
44.901888 172.31.0.2 -> 172.31.0.52 DNS Standard query response A 209.85.202.104 A 209.85.202.105 A 209.85.202.106 A 209.85.202.147 A 209.85.202.99 A 209.85.202.103
44.901937 172.31.0.52 -> 172.21.1.75 DNS Standard query response A 209.85.202.104 A 209.85.202.105 A 209.85.202.106 A 209.85.202.147 A 209.85.202.99 A 209.85.202.103

(And I guess that it's the same reason for NAT rule : the inbound-interface should not only be eth0...)

(By the way, it would be interesting to be able to add more than only one inbound-interface to a NAT rule...)

At first glance it looks like the name servers you are using are not reliable, and the lack of response is because the forwarder is also not getting a response.

Note that your sever only responded once the .internal domain was appended. Is your DNS server configure to allow recursion from the IP that VyOS is sourcing requests?

Can you try configuring the forwarder with multiple DNS servers? Here is an example using Google Public DNS:

set service dns forwarding name-server '8.8.8.8'
set service dns forwarding name-server '8.8.4.4'

Also if you have some internal domains you would like to resolve you can direct requests for specific domains to an internal server:

set service dns forwarding domain eu-west-1.compute.internal server '172.31.0.52'

P.S. This is really starting to get more into the territory of support than bug reporting, have you considered purchasing support?

https://sentrium.io/vyos-commercial-support/

Well, as I previously said, I finally know why it doesn't worked as expected for me, since lines like "listen-on vti0 and listen-on vti1" were missing, for requests incoming from tunneled networks.
However, it seems to be strange that requests are sometimes still forwarded, as we can expect that none are forwarded, or all are forwarded, but why sometimes only some request are forwarded ? This seems to be a bug, however this ticket can be closed since for my needs it's ok...

To answer you nevertheless, I added the two lines "set service dns forwarding name-server" for 8.8.8.8 and 8.8.4.4 in addition to the previous configuration, and here is the tshark result regarding 7 consecutive identical dns requests (first is KO, 2 next are OK, then KO, then OK and OK: as you can see, some are forwarded, but not all (until I add the "listen-on vti0" line, thus)
(For information, forwarded DNS server is an AWS integrated dns server)

Best regards,

On the vyos forwarding dns requests:
(DNS requests are sent at 0 (KO because not forwarded !), 22 (OK), 29 (OK), 44 (KO because first dns request is not forwarded !), 74 (OK), 83 (OK), and 92 (OK) )

vyos@VyOS-AMI# tshark -i any -f "((udp port 53) or (tcp port 53)) and (host 172.21.1.75 or host 172.31.0.2 or host 172.31.0.52)"
Capturing on Pseudo-device that captures on all interfaces

0.000000  172.21.1.75 -> 172.31.0.52  DNS Standard query A www.google.com
5.000003  172.21.1.75 -> 172.31.0.52  DNS Standard query A www.google.com

10.000045 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
15.000393 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com.eu-west-1.compute.internal

22.357439 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
22.357551 172.31.0.52 -> 172.31.0.2 DNS Standard query A www.google.com
22.357562 172.31.0.52 -> 8.8.4.4 DNS Standard query A www.google.com
22.357566 172.31.0.52 -> 8.8.8.8 DNS Standard query A www.google.com
22.358590 8.8.4.4 -> 172.31.0.52 DNS Standard query response A 216.58.198.68
22.358626 172.31.0.52 -> 172.21.1.75 DNS Standard query response A 216.58.198.68
22.358725 8.8.8.8 -> 172.31.0.52 DNS Standard query response A 216.58.198.68
22.360034 172.31.0.2 -> 172.31.0.52 DNS Standard query response A 74.125.28.103 A 74.125.28.104 A 74.125.28.105 A 74.125.28.106 A 74.125.28.147 A 74.125.28.99

29.389707 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
29.389806 172.31.0.52 -> 8.8.4.4 DNS Standard query A www.google.com
29.390847 8.8.4.4 -> 172.31.0.52 DNS Standard query response A 209.85.203.104 A 209.85.203.99 A 209.85.203.105 A 209.85.203.103 A 209.85.203.147 A 209.85.203.106
29.390884 172.31.0.52 -> 172.21.1.75 DNS Standard query response A 209.85.203.104 A 209.85.203.99 A 209.85.203.105 A 209.85.203.103 A 209.85.203.147 A 209.85.203.106

44.781944 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
49.782004 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
54.782161 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
59.782714 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com.eu-west-1.compute.internal
59.782798 172.31.0.52 -> 172.31.0.2 DNS Standard query A www.google.com.eu-west-1.compute.internal
59.782807 172.31.0.52 -> 8.8.4.4 DNS Standard query A www.google.com.eu-west-1.compute.internal
59.782811 172.31.0.52 -> 8.8.8.8 DNS Standard query A www.google.com.eu-west-1.compute.internal
59.783724 172.31.0.2 -> 172.31.0.52 DNS Standard query response, No such name
59.783756 172.31.0.52 -> 172.21.1.75 DNS Standard query response, No such name
59.783829 8.8.4.4 -> 172.31.0.52 DNS Standard query response, No such name
59.801593 8.8.8.8 -> 172.31.0.52 DNS Standard query response, No such name

74.088908 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
74.089004 172.31.0.52 -> 172.31.0.2 DNS Standard query A www.google.com
74.089322 172.31.0.2 -> 172.31.0.52 DNS Standard query response A 74.125.28.99 A 74.125.28.103 A 74.125.28.104 A 74.125.28.105 A 74.125.28.106 A 74.125.28.147
74.089352 172.31.0.52 -> 172.21.1.75 DNS Standard query response A 74.125.28.99 A 74.125.28.103 A 74.125.28.104 A 74.125.28.105 A 74.125.28.106 A 74.125.28.147

83.054100 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
83.054203 172.31.0.52 -> 172.31.0.2 DNS Standard query A www.google.com
83.054214 172.31.0.52 -> 8.8.4.4 DNS Standard query A www.google.com
83.054218 172.31.0.52 -> 8.8.8.8 DNS Standard query A www.google.com
83.055215 8.8.4.4 -> 172.31.0.52 DNS Standard query response A 216.58.198.68
83.055247 172.31.0.52 -> 172.21.1.75 DNS Standard query response A 216.58.198.68
83.055284 8.8.8.8 -> 172.31.0.52 DNS Standard query response A 216.58.198.68
83.055370 172.31.0.2 -> 172.31.0.52 DNS Standard query response A 74.125.197.103 A 74.125.197.104 A 74.125.197.105 A 74.125.197.106 A 74.125.197.147 A 74.125.197.99

92.758808 172.21.1.75 -> 172.31.0.52 DNS Standard query A www.google.com
92.758915 172.31.0.52 -> 8.8.4.4 DNS Standard query A www.google.com
92.759969 8.8.4.4 -> 172.31.0.52 DNS Standard query response A 209.85.203.104 A 209.85.203.99 A 209.85.203.105 A 209.85.203.103 A 209.85.203.147 A 209.85.203.106
92.760003 172.31.0.52 -> 172.21.1.75 DNS Standard query response A 209.85.203.104 A 209.85.203.99 A 209.85.203.105 A 209.85.203.103 A 209.85.203.147 A 209.85.203.106

On the client sending dns requests:
[ec2-user@ip-172-21-1-75 ~]$ nslookup www.google.com 172.31.0.52
;; connection timed out; trying next origin
;; connection timed out; no servers could be reached