RamNode logo
    Sign UpLogin
    Back to Cloud VPS Documentation
    Networking

    Troubleshooting DNS Resolution Failures

    When your server cannot resolve hostnames — diagnose the resolver stack, read key configuration files, use standard testing tools, and apply targeted fixes.

    Covers: Debian/Ubuntu and RHEL/CentOS/AlmaLinux/Rocky Linux | Tools: dig, nslookup, systemd-resolved, resolvectl, NetworkManager

    1. Introduction

    DNS resolution failures are among the most disruptive issues a server administrator can encounter. When a server cannot resolve hostnames, application deployments fail, package managers stop working, outbound API calls break, and mail delivery halts — even when the network itself is otherwise healthy.

    This guide covers the diagnostic process end to end: understanding the resolver stack, reading key configuration files, using standard testing tools, and applying targeted fixes.

    2. How Linux DNS Resolution Works

    When a process calls getaddrinfo(), the kernel consults /etc/nsswitch.conf to determine the lookup order. For most systems:

    Typical nsswitch.conf hosts line
    hosts: files dns myhostname

    This means the system checks /etc/hosts first, then queries the DNS server(s) in /etc/resolv.conf. Modern distributions layer additional components:

    ComponentRole
    systemd-resolvedStub resolver daemon (127.0.0.53). Manages caching, DNSSEC, and per-link DNS settings.
    NetworkManagerNetwork daemon that can push DNS servers to systemd-resolved or write directly to /etc/resolv.conf.
    /etc/resolv.confClassic resolver config. May be a real file or a symlink managed by systemd-resolved.
    DHCP clientReceives DNS addresses from the DHCP lease. May overwrite /etc/resolv.conf on reboot.

    The Symlink Problem

    On systemd-based systems /etc/resolv.conf is frequently a symlink. Its target determines behavior:

    Symlink TargetBehavior
    /run/systemd/resolve/stub-resolv.confPoints to 127.0.0.53. Recommended for most setups.
    /run/systemd/resolve/resolv.confDirect upstream DNS list. Bypasses stub; loses caching and DNSSEC.
    (regular file)Static config. DHCP client may overwrite it on network events.
    /run/resolvconf/resolv.confManaged by the resolvconf package (older Ubuntu/Debian).

    CAUTION: If /etc/resolv.conf is a dangling symlink (its target does not exist), all DNS will fail silently. This commonly happens after OS upgrades or when systemd-resolved is disabled without restoring the file.

    3. Symptoms and Initial Triage

    These symptoms point specifically to DNS rather than broader network connectivity issues:

    • Commands like apt update, dnf install, curl, or git clone fail with "Name or service not known"
    • ping 8.8.8.8 succeeds but ping google.com fails
    • Application logs show connection errors only for hostname-based targets
    • getent hosts <hostname> returns nothing
    Quick connectivity check
    # Confirm IP-level connectivity is fine
ping -c 3 8.8.8.8

# Confirm DNS specifically fails
ping -c 3 google.com

# Low-level resolver test (bypasses caching)
getent hosts google.com

    4. Checking /etc/resolv.conf

    This is always the first file to inspect. A working resolv.conf must contain at least one reachable nameserver entry.

    Inspect resolv.conf
    cat /etc/resolv.conf

# Check whether it is a real file or a symlink
ls -la /etc/resolv.conf

    Common Problems Found in resolv.conf

    ProblemHow to Identify
    Empty file or no nameserver linescat shows no 'nameserver' entries
    Dangling symlinkls -la shows → but the target path does not exist
    Nameserver 127.0.0.53 but systemd-resolved not runningsystemctl status systemd-resolved shows inactive
    DHCP overwrote servers with wrong IPnameserver is 192.168.x.x and that host is unreachable
    File is immutablechattr +i was set; writes silently fail
    Check for immutable flag
    lsattr /etc/resolv.conf
# If output shows '----i-----------' the file is immutable
# Remove immutable flag to allow edits:
chattr -i /etc/resolv.conf

    5. Testing with dig and nslookup

    5.1 Installing the Tools

    Debian / Ubuntu
    apt install -y dnsutils    # provides dig and nslookup
    RHEL / CentOS / AlmaLinux
    dnf install -y bind-utils    # provides dig, nslookup, host

# RHEL 7 / CentOS 7:
yum install -y bind-utils

    5.2 Basic dig Usage

    dig commands
    # Basic lookup using the system default resolver
dig google.com

# Lookup using a specific upstream server (bypasses resolv.conf)
dig @8.8.8.8 google.com

# Short output: just the answer
dig +short google.com

# Reverse DNS lookup
dig -x 8.8.8.8

# Query a specific record type
dig google.com MX
dig google.com AAAA

# Trace the full resolution path from root servers
dig +trace google.com

    Key fields to examine in dig output:

    FieldWhat to Look For
    status:Should be NOERROR. SERVFAIL = upstream failing; NXDOMAIN = name doesn't exist; REFUSED = server rejected query.
    ;; SERVER:Shows which resolver answered. 127.0.0.53 = stub resolver handled it.
    ;; Query time:Values >500ms suggest a slow or cold DNS server.
    ANSWER SECTIONMust be present and non-empty for successful resolution.

    5.3 Basic nslookup Usage

    nslookup commands
    # Basic query
nslookup google.com

# Query a specific DNS server
nslookup google.com 8.8.8.8

# Interactive mode
nslookup
> server 1.1.1.1
> google.com
> set type=MX
> google.com
> exit

    5.4 Diagnostic Decision Tree

    Test ResultLikely Cause
    ping 8.8.8.8 failsNetwork issue, not DNS — check routing and firewall first
    dig @8.8.8.8 fails, ping worksUDP port 53 is blocked by a firewall or security group
    dig @8.8.8.8 works, default failsLocal resolver broken — check resolv.conf and systemd-resolved
    dig works but applications failApplication using its own resolver or /etc/nsswitch.conf misconfigured
    dig works only for some domainsDNS server has no forwarder configured, or split-horizon issue

    6. systemd-resolved Configuration

    systemd-resolved is the default stub resolver on Ubuntu 18.04+, Debian 10+, RHEL 8+, and most modern systemd-based distributions.

    6.1 Checking Status

    systemd-resolved status commands
    # Service status
systemctl status systemd-resolved

# Resolver status: DNS servers, search domains, DNSSEC state
resolvectl status

# Per-link DNS settings (useful on multi-homed servers)
resolvectl status eth0

# Show current DNS server and domain configuration
resolvectl dns
resolvectl domain

# Flush the DNS cache
resolvectl flush-caches

# Query a name through systemd-resolved directly
resolvectl query google.com

    6.2 Main Configuration File

    The primary configuration file is /etc/systemd/resolved.conf. Drop-in files in /etc/systemd/resolved.conf.d/*.conf override or extend the base config.

    /etc/systemd/resolved.conf — hardened VPS config
    [Resolve]
DNS=8.8.8.8 8.8.4.4 1.1.1.1
FallbackDNS=9.9.9.9 149.112.112.112
Domains=~.
DNSSEC=allow-downgrade
DNSOverTLS=opportunistic
Cache=yes
DNSStubListener=yes

    After editing, apply the changes:

    Apply resolved config
    systemctl restart systemd-resolved

    6.3 Ensuring the Correct resolv.conf Symlink

    Set the recommended symlink
    # Check current state
ls -la /etc/resolv.conf

# Set the recommended stub symlink
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

# Alternative: direct upstream list (no stub, no caching)
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

# Verify
cat /etc/resolv.conf

    7. Distribution-Specific Notes

    7.1 Debian / Ubuntu

    Ubuntu 18.04 and Later

    systemd-resolved is enabled by default. The symlink should point to /run/systemd/resolve/stub-resolv.conf.

    Ubuntu — restore resolved
    # Restore symlink
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

# Confirm resolved is running
systemctl enable --now systemd-resolved

# Check NetworkManager is handing off DNS to resolved
cat /etc/NetworkManager/conf.d/dns.conf
# Should contain:
# [main]
# dns=systemd-resolved

    Debian with resolvconf Package

    resolvconf management
    # Check which package is active
dpkg -l resolvconf 2>/dev/null | grep ^ii
dpkg -l openresolv 2>/dev/null | grep ^ii

# Regenerate resolv.conf
resolvconf -u

# Statically add a DNS server via resolvconf
echo 'nameserver 8.8.8.8' | resolvconf -a lo.dns

    NetworkManager DNS Override

    Prevent NM from touching resolv.conf
    # Check NetworkManager DNS plugin
grep -r 'dns=' /etc/NetworkManager/

# Delegate DNS to systemd-resolved
mkdir -p /etc/NetworkManager/conf.d
cat > /etc/NetworkManager/conf.d/dns.conf << 'EOF'
[main]
dns=systemd-resolved
EOF

systemctl reload NetworkManager

    7.2 RHEL / CentOS / AlmaLinux / Rocky Linux

    RHEL 8/9 and Derivatives

    NetworkManager is the primary DNS manager. By default it writes directly to /etc/resolv.conf.

    RHEL 8/9 — NetworkManager DNS
    # Check which DNS backend NetworkManager uses
NetworkManager --print-config | grep dns

# Show DNS servers per connection profile
nmcli con show
nmcli con show 'System eth0' | grep DNS

# Set DNS servers on a connection profile
nmcli con mod 'System eth0' ipv4.dns '8.8.8.8 8.8.4.4'
nmcli con mod 'System eth0' ipv4.ignore-auto-dns yes
nmcli con up 'System eth0'

    RHEL 7 / CentOS 7

    RHEL 7 — interface config
    # View the interface config file
cat /etc/sysconfig/network-scripts/ifcfg-eth0

# Add DNS to the interface file (persists across reboots):
# DNS1=8.8.8.8
# DNS2=8.8.4.4
# PEERDNS=no  (prevents DHCP from overwriting your DNS settings)

# After editing, restart network:
nmcli con reload
systemctl restart network    # RHEL 7 only

    firewalld and DNS Port 53

    firewalld DNS rules
    # Test if DNS port is reachable
nc -zuv 8.8.8.8 53

# Check firewalld DNS rules
firewall-cmd --list-all

# Allow outbound DNS (if needed in a restrictive policy)
firewall-cmd --permanent --add-service=dns
firewall-cmd --reload

    8. Common Causes and Fixes

    8.1 Corrupted or Empty resolv.conf After Reboot

    The most frequently reported DNS failure on VPS instances. Occurs when the file is regenerated empty, or when a symlink target is not yet available during early boot.

    Fix corrupted resolv.conf
    # Inspect
cat /etc/resolv.conf        # empty or missing nameserver lines?
ls -la /etc/resolv.conf     # real file or symlink?

# Fix 1: Restore symlink (systemd-resolved present)
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

# Fix 2: Write a static file (no systemd-resolved)
cat > /etc/resolv.conf << 'EOF'
nameserver 8.8.8.8
nameserver 8.8.4.4
options edns0
EOF

# Fix 3: Protect from being overwritten (use carefully)
chattr +i /etc/resolv.conf

    8.2 Missing or Wrong DNS Servers

    Test each nameserver
    # Test each nameserver individually
dig @$(grep nameserver /etc/resolv.conf | awk '{print $2}' | head -1) google.com

# Time how long a query takes (should be <100ms for cached)
time dig @8.8.8.8 google.com +short

# Check if port 53 is reachable
nc -zuv $(grep nameserver /etc/resolv.conf | awk '{print $2}' | head -1) 53

    8.3 DHCP Overwriting DNS Settings

    Prevent DHCP DNS overwrite
    # ---- Debian / Ubuntu with NetworkManager ----
nmcli con mod 'System eth0' ipv4.ignore-auto-dns yes
nmcli con mod 'System eth0' ipv4.dns '8.8.8.8 8.8.4.4'
nmcli con up 'System eth0'

# ---- RHEL 7 via ifcfg file ----
# Edit /etc/sysconfig/network-scripts/ifcfg-eth0
# Set: PEERDNS=no
# Set: DNS1=8.8.8.8
# Set: DNS2=8.8.4.4

# ---- Debian / Ubuntu via DHCP client config ----
echo 'supersede domain-name-servers 8.8.8.8, 8.8.4.4;' >> /etc/dhcp/dhclient.conf

    8.4 systemd-resolved Disabled or Crashed

    If /etc/resolv.conf points to 127.0.0.53 but systemd-resolved is not running, all lookups will fail immediately.

    Fix systemd-resolved
    # Check status
systemctl status systemd-resolved

# Start and enable
systemctl enable --now systemd-resolved

# View recent logs for crash details
journalctl -u systemd-resolved -n 50 --no-pager

# Validate the stub is listening
ss -ulnp | grep ':53'
# Expected: udp  UNCONN  0 0 127.0.0.53%lo:53 ...

    8.5 SELinux or AppArmor Blocking DNS

    Check security framework blocks
    # Check for DNS-related AVC denials (SELinux)
ausearch -c 'systemd-resolve' --raw | audit2why
grep 'dns\|resolv\|named' /var/log/audit/audit.log | tail -20

# Temporarily put SELinux in permissive mode to test
setenforce 0
# Test DNS, then re-enable:
setenforce 1

# ---- AppArmor (Debian/Ubuntu) ----
aa-status
grep -i 'dns\|resolv' /var/log/syslog | tail -20

    8.6 Split-Horizon / VPN DNS Leakage

    Per-domain DNS routing
    # Check current per-link DNS assignments
resolvectl status

# Configure per-domain DNS routing
# Send *.internal.example.com to the internal DNS server:
resolvectl dns tun0 10.0.0.1
resolvectl domain tun0 ~internal.example.com

# For NetworkManager-managed VPN connections:
nmcli con mod 'VPN Profile' ipv4.dns '10.0.0.1'
nmcli con mod 'VPN Profile' ipv4.dns-search 'internal.example.com'

    9. Advanced Diagnostics

    9.1 Tracing What Resolver an Application Uses

    Some applications (Java, Go binaries, certain containers) implement their own DNS resolver and do not use the system NSS stack. Use strace to trace what they're doing.

    9.2 nsswitch.conf Verification

    Check NSS lookup order
    cat /etc/nsswitch.conf | grep hosts

# Correct (most systems):
# hosts: files dns myhostname

# Broken (dns missing):
# hosts: files

# Test a lookup explicitly:
getent ahosts google.com

    9.3 Checking DNS Cache State

    DNS cache management
    # systemd-resolved cache statistics
resolvectl statistics

# Flush all caches
resolvectl flush-caches

# RHEL/CentOS with nscd (Name Service Cache Daemon)
systemctl status nscd
nscd -g              # show stats
nscd -i hosts        # invalidate host cache only

    9.4 Packet-Level Verification

    When all else fails, capture DNS traffic to confirm whether queries are leaving the machine and receiving responses:

    Capture DNS traffic with tcpdump
    tcpdump -i any -n port 53 -w /tmp/dns.pcap

# In another terminal, trigger a lookup:
dig google.com

# Read the capture
tcpdump -r /tmp/dns.pcap -v

# Quick inline capture (no file, human-readable)
tcpdump -i any -n 'udp port 53 or tcp port 53' -v &
dig google.com
kill %1

    10. Quick-Reference Checklist

    Use this checklist in order when DNS resolution fails on any server:

    StepCommand / Action
    1Confirm it's DNS: ping 8.8.8.8 (works) vs ping google.com (fails)
    2Inspect resolv.conf: cat /etc/resolv.conf
    3Check for dangling symlink: ls -la /etc/resolv.conf
    4Check for immutable flag: lsattr /etc/resolv.conf
    5Test upstream DNS directly: dig @8.8.8.8 google.com +short
    6Test local resolver: dig google.com +short
    7Check systemd-resolved status: systemctl status systemd-resolved
    8Check resolved config: resolvectl status
    9Check NetworkManager: nmcli con show <profile> | grep DNS
    10Flush DNS cache: resolvectl flush-caches
    11Check firewall port 53: nc -zuv 8.8.8.8 53
    12Check NSS order: cat /etc/nsswitch.conf | grep hosts
    13Review service logs: journalctl -u systemd-resolved -n 50
    14Capture DNS packets: tcpdump -i any -n port 53

    11. Preventing DNS Failures

    • Use static DNS in the interface profile: Set DNS1= and PEERDNS=no on RHEL, or ipv4.ignore-auto-dns yes on NetworkManager connections, to prevent DHCP lease renewals from overwriting your nameservers.
    • Pin the resolv.conf symlink: On systemd systems ensure /etc/resolv.conf always points to /run/systemd/resolve/stub-resolv.conf. Add this to a startup script or cloud-init configuration.
    • Define FallbackDNS in resolved.conf: Even if your primary DNS is internal, always set FallbackDNS=8.8.8.8 1.1.1.1 so resolution continues if the primary is unreachable.
    • Monitor with a health check: A simple cron job running dig google.com > /dev/null can catch DNS failures before applications notice them.
    • Test DNS in your provisioning pipeline: Add dig +short google.com to post-deploy smoke tests.
    • Document your DNS architecture: For servers on private networks or VPNs, document which resolver handles which domains and keep this in version control.

    Appendix: Key Files and Locations

    File / PathPurpose
    /etc/resolv.confActive resolver config (may be a symlink)
    /run/systemd/resolve/stub-resolv.confStub resolver config (127.0.0.53) — symlink target
    /run/systemd/resolve/resolv.confDirect upstream DNS list from systemd-resolved
    /etc/systemd/resolved.confMain systemd-resolved configuration
    /etc/systemd/resolved.conf.d/Drop-in config directory for resolved
    /etc/nsswitch.confName service switch — controls lookup order
    /etc/hostsStatic hostname-to-IP mappings
    /etc/NetworkManager/conf.d/NetworkManager drop-in configuration directory
    /etc/sysconfig/network-scripts/ifcfg-*RHEL 7 interface config files (DNS1=, PEERDNS=)
    /etc/dhcp/dhclient.confDHCP client config (Debian/Ubuntu — use supersede)
    /var/log/syslog or /var/log/messagesGeneral system log including DNS events

    Need more help?: Open a support ticket at my.ramnode.com or consult the RamNode knowledge base for additional guides on networking and server administration.