DNS failing

[jgarbora]

package's version:
openvpn-systemd-resolved: 1.3.0-3
openvpn: 2.4.7-1ubuntu2

root@xps-13:~# openvpn --version
OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep  5 2019
library versions: OpenSSL 1.1.1c  28 May 2019, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_maintainer_mode=no enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

config file

client
dev tun
proto udp
remote vpn.xxx.com 1194
resolv-retry infinite
nobind
;user nobody
;group nobody
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
comp-lzo
;pull dhcp-options

script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

starting

root@xps-13:~# openvpn xxxVPN.ovpn 
Wed Jan  1 12:35:11 2020 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep  5 2019
Wed Jan  1 12:35:11 2020 library versions: OpenSSL 1.1.1c  28 May 2019, LZO 2.10
Wed Jan  1 12:35:11 2020 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Wed Jan  1 12:35:11 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Wed Jan  1 12:35:11 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]18.228.104.124:1194
Wed Jan  1 12:35:11 2020 UDP link local: (not bound)
Wed Jan  1 12:35:11 2020 UDP link remote: [AF_INET]x.x.x.x:1194
Wed Jan  1 12:35:13 2020 [server] Peer Connection Initiated with [AF_INET]18.228.104.124:1194
Wed Jan  1 12:35:14 2020 TUN/TAP device tun0 opened
Wed Jan  1 12:35:14 2020 /sbin/ip link set dev tun0 up mtu 1500
Wed Jan  1 12:35:14 2020 /sbin/ip addr add dev tun0 local 10.99.0.42 peer 10.99.0.41
Wed Jan  1 12:35:14 2020 /etc/openvpn/update-resolv-conf tun0 1500 1553 10.99.0.42 10.99.0.41 init
dhcp-option DNS 10.104.1.130
Wed Jan  1 12:35:19 2020 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Wed Jan  1 12:35:19 2020 Initialization Sequence Completed

resolv.conf after connecting

root@xps-13:~$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 10.104.1.130
nameserver 127.0.0.53
search home tendawifi.com

then some how DNS is resolved

root@xps-13:~# nslookup kibana-teahupoo.aws.xxx.com
Server:     10.104.1.130
Address:    10.104.1.130#53

Non-authoritative answer:
kibana-teahupoo.aws.xxx.com canonical name = kibana-prod.aws.xxx.com.
Name:   kibana-prod.aws.xxx.com
Address: 10.103.4.184

but not for ping

root@xps-13:~# ping kibana-teahupoo.aws.xxx.com
ping: kibana-teahupoo.aws.xxx.com: Name or service not known

or browser

This site can’t be reached kibana-teahupoo.aws.xxx.com’s server IP address could not be found.
DNS_PROBE_FINISHED_NXDOMAIN

systemd-resolve --status

root@xps-13:~$ systemd-resolve --status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 10.104.1.130
         DNS Servers: 10.104.1.130
          DNS Domain: tendawifi.com
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (tun0)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (wlp2s0)
      Current Scopes: DNS
DefaultRoute setting: yes
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 192.168.5.1
         DNS Servers: 192.168.5.1
          DNS Domain: ~.
                      tendawifi.com

Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

root@xps-13:/etc/openvpn# systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-02 09:58:48 CET; 25min ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 12478 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 3.1M
   CGroup: /system.slice/systemd-resolved.service
           └─12478 /lib/systemd/systemd-resolved

ene 02 10:17:07 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:17:07 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:17:15 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:18:41 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:18:41 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:18:41 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:19:02 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:19:02 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:19:02 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
ene 02 10:19:12 xps-13 systemd-resolved[12478]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

any clue ?
same config is running fine on a debian 10
this is a Ubuntu 19.10

[Hordeking]

I'm having similar issues on Mint 19.3. update-systemd-resolved seems like it's updating dns, but then when I go to ping anything on that lan, ping isn't able to resolve it (but it can ping the machine by ip)

[piotr-dobrogost]

nslookup goes directly to /etc/resolv.conf to find out DNS servers to use and (as can be seen in the output) it uses the first one there (10.104.1.130) which is the DNS server available through VPN link thus it works.
ping on the other hand uses libc resolver thus honours /etc/nsswitch.conf configuration which probably directs the query in some moment to systemd-resolved (either directly by means of systemd-specific resolve directive or indirectly through a systemd-resolved stub-resolver by means of the standard dns directive).
As there is ~. routing domain specified on wlp2s0 interface systemd-resolved routes all queries to the DNS server for this link (192.168.5.1) which is a different DNS server that in the case of nslookup which might explain different result in domain lookup result.
The root of the problem is lack of VPN provided DNS server (10.104.1.130) on the list of DNS servers of VPN link (tun0) which is strange.
However one thing is not clear to me; as kibana-teahupoo.aws.xxx.com seems to be a public domain I would expect it to be resolved by the DNS server on wlp2s0 link (192.168.5.1) just fine. Is this DNS server somehow restricted? Do you expect kibana-teahupoo.aws.xxx.com to be resolved specifically by the DNS server of VPN link?

If you want to know more on split DNS (different DNS servers for different domains) I would recommend recent article https://fedoramagazine.org/systemd-resolved-introduction-to-split-dns/

[tomeon]

@jgarbora @Hordeking -- does this remain an issue?

[Hordeking]

@tomeon I will have to take a look and see. I will post something when I try it again.

[tomeon]

@jgarbora -- couple of things:

1. Suggest you use systemd-resolved's stub resolver; see here for instructions on setting it up.
2. Do you have access to the OpenVPN server configuration file(s)? If so, do you see the server pushing any dhcp-options? I don't see anything in your $ openvpn xxxVPN.ovpn snippet to suggest that the server is specifying any DNS servers, DNS search or routing domains, etc. Note that, if the server doesn't push appropriate settings, you can always add, say, dhcp-option DNS <VPN-DNS-resolver-IP> to your client configuration. See here for available options.
3. Is kibana-teahupoo.aws.xxx.com the actual domain you are resolving, or is xxx a placeholder for something else? It sure seems like it's the latter:

$ dig lolol.wat.xxx.com

; <<>> DiG 9.18.16 <<>> lolol.wat.xxx.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57195
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;lolol.wat.xxx.com.             IN      A

;; ANSWER SECTION:
lolol.wat.xxx.com.      1800    IN      CNAME   www.xxx.com.
www.xxx.com.            1749    IN      A       141.0.173.173

;; Query time: 311 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Sep 08 09:28:03 EDT 2023
;; MSG SIZE  rcvd: 80