Certain domains randomly stop resolving for about 5 minutes. No pattern in which domains. Happens 3-4 times a day. Rights itself with no configuration changes and no actions I've found make any difference in speeding that up. (Short of rebooting the router - which is a short term fix).
google.com
$ dig google.com ; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 39675 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;google.com. IN A ;; AUTHORITY SECTION: . 2647 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2019041000 1800 900 604800 86400 ;; Query time: 2 msec ;; SERVER: 192.168.18.1#53(192.168.18.1) ;; WHEN: Wed Apr 10 21:59:38 AEST 2019 ;; MSG SIZE rcvd: 111
21:59
$ host google.com Host google.com not found: 3(NXDOMAIN)
$ nslookup google.com Server: 192.168.18.1 Address: 192.168.18.1#53 ** server can't find google.com: NXDOMAIN
On router itself, resolution is fine.
vyos@graham-vyos:~$ host google.com google.com has address 216.58.196.142 google.com has IPv6 address 2404:6800:4006:805::200e google.com mail is handled by 10 aspmx.l.google.com. google.com mail is handled by 40 alt3.aspmx.l.google.com. google.com mail is handled by 50 alt4.aspmx.l.google.com. google.com mail is handled by 20 alt1.aspmx.l.google.com. google.com mail is handled by 30 alt2.aspmx.l.google.com.
vyos@graham-vyos:~$ sudo rec_control ping pong
after running host google.com on the router - makes no difference to resolving google.com inside my lan.
router lan interface tcpdump port 53
around this timestamp I hit refresh repeatedly on google.com in chrome on linux (hostname blue-canoe, graham is the router running vyos).
21:59:16.745527 IP blue-canoe.40818 > graham.domain: 14122+ A? www.google.com. (32) 21:59:16.745597 IP graham.domain > blue-canoe.40818: 14122 NXDomain 0/1/0 (104) 21:59:16.747666 IP blue-canoe.54593 > graham.domain: 9066+ A? www.google.com.xxxxx.info. (47) 21:59:16.747730 IP graham.domain > blue-canoe.54593: 9066 NXDomain 0/1/0 (96) 21:59:16.764154 IP blue-canoe.45325 > graham.domain: 62526+ A? www.google.com. (32) 21:59:16.764229 IP graham.domain > blue-canoe.45325: 62526 NXDomain 0/1/0 (104) 21:59:16.765527 IP blue-canoe.44157 > graham.domain: 30752+ A? www.google.com.xxxxx.info. (47) 21:59:16.765591 IP graham.domain > blue-canoe.44157: 30752 NXDomain 0/1/0 (96) 21:59:16.767154 IP blue-canoe.35299 > graham.domain: 12206+ A? www.google.com. (32) 21:59:16.767227 IP graham.domain > blue-canoe.35299: 12206 NXDomain 0/1/0 (104) 21:59:16.768541 IP blue-canoe.37555 > graham.domain: 20245+ A? www.google.com.xxxxx.info. (47) 21:59:16.768604 IP graham.domain > blue-canoe.37555: 20245 NXDomain 0/1/0 (96) 21:59:16.770041 IP blue-canoe.39109 > graham.domain: 64731+ A? www.google.com. (32) 21:59:16.770112 IP graham.domain > blue-canoe.39109: 64731 NXDomain 0/1/0 (104) 21:59:16.771904 IP blue-canoe.42471 > graham.domain: 11000+ A? www.google.com.xxxxxxx.info. (47) 21:59:16.771966 IP graham.domain > blue-canoe.42471: 11000 NXDomain 0/1/0 (96) 21:59:16.803780 IP blue-canoe.40262 > graham.domain: 49111+ A? www.google.com. (32) 21:59:16.803853 IP graham.domain > blue-canoe.40262: 49111 NXDomain 0/1/0 (104) 21:59:16.806279 IP blue-canoe.49720 > graham.domain: 57719+ A? www.google.com.xxxxxxx.info. (47) 21:59:16.806341 IP graham.domain > blue-canoe.49720: 57719 NXDomain 0/1/0 (96) 21:59:16.835405 IP blue-canoe.41244 > graham.domain: 30213+ A? www.google.com. (32) 21:59:16.835477 IP graham.domain > blue-canoe.41244: 30213 NXDomain 0/1/0 (104) 21:59:16.836906 IP blue-canoe.53697 > graham.domain: 24971+ A? www.google.com.xxxxxxx.info. (47) 21:59:16.836975 IP graham.domain > blue-canoe.53697: 24971 NXDomain 0/1/0 (96) 21:59:17.197047 IP mediaroom-tv.13159 > graham.domain: 27754+ A? socialize.au1.gigya.com. (41) 21:59:17.197212 IP graham.domain > mediaroom-tv.13159: 27754 NXDomain 0/1/0 (113) 21:59:17.197668 IP mediaroom-tv.32640 > graham.domain: 2787+ A? socialize.au1.gigya.com.xxxxxxx.info. (56) 21:59:17.197846 IP graham.domain > mediaroom-tv.32640: 2787 NXDomain 0/1/0 (105) 21:59:17.432176 IP mediaroom-tv.31617 > graham.domain: 21253+ A? bcp.crwdcntrl.net. (35) 21:59:17.444639 IP graham.domain > mediaroom-tv.31617: 21253 NXDomain 2/1/0 CNAME td.crwdcntrl.net., CNAME nginx-bcp-stackA-1013960178.ap-southeast-2.elb.amazonaws.com. (195) 21:59:17.445439 IP mediaroom-tv.33475 > graham.domain: 22858+ A? bcp.crwdcntrl.net.xxxxxxx.info. (50) 21:59:17.445741 IP graham.domain > mediaroom-tv.33475: 22858 NXDomain 0/1/0 (99) 21:59:17.548682 IP blue-canoe.38113 > graham.domain: 32303+ A? www.google.com. (32) 21:59:17.548771 IP graham.domain > blue-canoe.38113: 32303 NXDomain 0/1/0 (104) 21:59:17.551055 IP blue-canoe.55358 > graham.domain: 25861+ A? www.google.com.xxxxxxx.info. (47) 21:59:17.551124 IP graham.domain > blue-canoe.55358: 25861 NXDomain 0/1/0 (96) 21:59:17.554305 IP blue-canoe.42845 > graham.domain: 2210+ A? www.google.com. (32) 21:59:17.554377 IP graham.domain > blue-canoe.42845: 2210 NXDomain 0/1/0 (104) 21:59:17.556555 IP blue-canoe.41995 > graham.domain: 6374+ A? www.google.com.xxxxxxx.info. (47) 21:59:17.556619 IP graham.domain > blue-canoe.41995: 6374 NXDomain 0/1/0 (96) 21:59:18.165826 IP blue-canoe.46368 > graham.domain: 56695+ A? www.google.com. (32) 21:59:18.165930 IP graham.domain > blue-canoe.46368: 56695 NXDomain 0/1/0 (104) 21:59:18.168701 IP blue-canoe.50915 > graham.domain: 33099+ A? www.google.com.xxxxxxx.info. (47) 21:59:18.168776 IP graham.domain > blue-canoe.50915: 33099 NXDomain 0/1/0 (96) 21:59:18.171836 IP blue-canoe.40443 > graham.domain: 28363+ A? www.google.com. (32) 21:59:18.171964 IP graham.domain > blue-canoe.40443: 28363 NXDomain 0/1/0 (104) 21:59:18.174825 IP blue-canoe.44161 > graham.domain: 18174+ A? www.google.com.xxxxxxx.info. (47) 21:59:18.174906 IP graham.domain > blue-canoe.44161: 18174 NXDomain 0/1/0 (96) 21:59:18.232833 IP blue-canoe.37112 > graham.domain: 42181+ A? www.google.com. (32) 21:59:18.232918 IP graham.domain > blue-canoe.37112: 42181 NXDomain 0/1/0 (104) 21:59:18.234203 IP blue-canoe.54379 > graham.domain: 45708+ A? www.google.com.xxxxxxx.info. (47) 21:59:18.234284 IP graham.domain > blue-canoe.54379: 45708 NXDomain 0/1/0 (96) 21:59:18.237202 IP blue-canoe.33771 > graham.domain: 46809+ A? www.google.com. (32) 21:59:18.237265 IP graham.domain > blue-canoe.33771: 46809 NXDomain 0/1/0 (104) 21:59:18.239327 IP blue-canoe.57118 > graham.domain: 29551+ A? www.google.com.xxxxxxx.info. (47) 21:59:18.239401 IP graham.domain > blue-canoe.57118: 29551 NXDomain 0/1/0 (96) 21:59:18.556466 IP blue-canoe.34221 > graham.domain: 20864+ A? www.google.com. (32) 21:59:18.556543 IP graham.domain > blue-canoe.34221: 20864 NXDomain 0/1/0 (104) 21:59:18.558089 IP blue-canoe.54049 > graham.domain: 19530+ A? www.google.com.xxxxxxx.info. (47) 21:59:18.558152 IP graham.domain > blue-canoe.54049: 19530 NXDomain 0/1/0 (96) 21:59:18.560237 IP blue-canoe.42765 > graham.domain: 16723+ A? www.google.com. (32) 21:59:18.560305 IP graham.domain > blue-canoe.42765: 16723 NXDomain 0/1/0 (104) 21:59:18.561713 IP blue-canoe.41813 > graham.domain: 21407+ A? www.google.com.xxxxxxx.info. (47) 21:59:18.561789 IP graham.domain > blue-canoe.41813: 21407 NXDomain 0/1/0 (96)
tcpdump of port 53 on router WAN interface
21:58:33.031563 IP one.one.one.one.domain > xxx-xxx-xxx-xxx.myrepublic.net.42790: 26354 0/1/0 (92) 21:59:16.698347 IP xxx-xxx-xxx-xxx.myrepublic.net.48441 > 103-217-165-53.myrepublic.net.domain: 6265+ [1au] A? asia.adform.net. (44) 21:59:16.708928 IP 103-217-165-53.myrepublic.net.domain > xxx-xxx-xxx-xxx.myrepublic.net.48441: 6265 6/0/1 CNAME track-apac.adformnet.akadns.net., A 185.84.60.25, A 185.84.60.29, A 185.84.60.23, A 185.84.60.27, A 185.84.60.12 (166) 21:59:17.432390 IP xxx-xxx-xxx-xxx.myrepublic.net.61432 > 103-217-165-53.myrepublic.net.domain: 2427+ [1au] A? bcp.crwdcntrl.net. (46) 21:59:17.444421 IP 103-217-165-53.myrepublic.net.domain > xxx-xxx-xxx-xxx.myrepublic.net.61432: 2427 4/0/1 CNAME td.crwdcntrl.net., CNAME nginx-bcp-stackA-1013960178.ap-southeast-2.elb.amazonaws.com., A 3.104.4.113, A 13.210.233.18 (169) 21:59:18.574420 IP xxx-xxx-xxx-xxx.myrepublic.net.13604 > 103-217-165-53.myrepublic.net.domain: 23454+ [1au] A? config.swm.digital. (47) 21:59:18.585203 IP 103-217-165-53.myrepublic.net.domain > xxx-xxx-xxx-xxx.myrepublic.net.13604: 23454 4/0/1 A 13.35.146.96, A 13.35.146.95, A 13.35.146.31, A 13.35.146.20 (111)
22:06 it's back with no configuration changes at all in any way.
It's almost like it has its wires crossed about a cached value and is sure it has checked and the domain doesn't resolve. Then it does check again and resolves it forevermore until the next time about 6 hours later.
I have tried setting the cache size to 0 and also to 128 with no discernible difference.
I switched off dnssec in case that helped, no luck. I log the firewall packets dropped, nothing interesting. I ensure port 53 is not firewalled, no difference. I've followed other superstition too with no luck.
vyos@graham-vyos:~$ cat /etc/powerdns/recursor.conf
### Autogenerated by dns_forwarding.py ### # Non-configurable defaults daemon=yes threads=1 allow-from=0.0.0.0/0, ::/0 log-common-errors=yes non-local-bind=yes query-local-address=0.0.0.0 query-local-address6=:: # cache-size max-cache-entries=128 # negative TTL for NXDOMAIN max-negative-ttl=3600 # ignore-hosts-file export-etc-hosts=yes # listen-on local-address=192.168.18.1 # domain ... server ... # dnssec dnssec=off # name-server forward-zones-recurse=.=103.217.165.53;45.248.197.53
Just quietly I might be almost ready to go back to running straight debian as a router, using dnsmasq, IPTables commands etc because I can't fix or even diagnose this. Which seems a shame, the vyos configuration ui seems nice to use but this is actually quite annoying for users of the local network when suddenly mid flow things just stop working.
Cheers vyos team. I really do like your work...