A name resolution issue with systemd-resolved we found in the wild

Moss allows you to connect any Ubuntu 18.04 or 16.04 server, whether it’s hosted with Amazon, DigitalOcean, Google Cloud, Vultr, or any other provider. While we offer native integrations with major cloud platforms, you can use Moss with virtually any VPS, cloud instance, or even bare metal servers.

Recently, one of our customers encountered difficulties connecting their Ubuntu 18.04 instance to Moss. To investigate, I set up an account with their hosting provider and discovered that their default server image had some questionable configuration choices. Combined with known bugs in Ubuntu 18.04’s DNS resolution system, this created an interesting debugging challenge worth documenting. This post explores systemd-resolved and how misconfigurations can create hard-to-diagnose problems.

Understanding systemd-resolved

The systemd project provides fundamental building blocks for modern Linux systems. Most notably, it includes an init system capable of parallel service startup, which significantly reduces boot times. If you’re running Ubuntu 16.04 (xenial) or 18.04 (bionic), systemd is already managing your system as the PID 1 process.

While systemd offers many improvements, it has faced criticism from parts of the open source community. One common complaint is that systemd reimplements existing subsystems that were working fine, sometimes introducing subtle incompatibilities.

DNS resolution is one such subsystem. Instead of using traditional approaches, systemd includes its own implementation: systemd-resolved. Ubuntu adopted systemd-resolved starting with version 16.10, and it’s now the default in the current LTS release (18.04).

systemd-resolved acts as a local DNS proxy, offering features like DNS caching and DNSSEC validation. Applications can interact with it through three different interfaces:

  1. D-Bus API – Used primarily by desktop applications and systemd services through inter-process communication
  2. glibc API – Traditional functions like getaddrinfo(), though this requires additional configuration and doesn’t support all systemd-resolved features
  3. Local DNS stub listener – Runs on 127.0.0.53, used by applications that handle DNS queries directly

This creates complexity: different applications on the same server might resolve DNS queries differently depending on which interface they use and how the system is configured.

Traditional Linux DNS Configuration

Historically, Linux systems configured DNS through /etc/resolv.conf, a simple file listing nameservers queried in order. Administrators would manually edit this file and move on.

As computing environments became more dynamic—particularly with mobile devices and cloud infrastructure—static configuration files became inadequate. Tools like resolvconf emerged to dynamically update /etc/resolv.conf based on network changes, DHCP responses, and other events.

With systemd-resolved, things get more complicated. The system can either provide /etc/resolv.conf itself or consume it, depending on the compatibility mode selected. You can configure it to use 127.0.0.53 as the resolver, allow applications to bypass systemd-resolved entirely, or let other packages manage the file completely.

The Problem We Encountered

Our customer was experiencing this error when trying to import GPG keys:

root@server:~# gpg --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 14AA40EC0831756756D7F66C4F4EA0AAE5267A6C
gpg: keyserver receive failed: Invalid argument

The GPG tool uses dirmngr (part of GNU Privacy Guard) to handle certificates and key servers. Checking the logs revealed a DNS resolution failure:

root@server:~# cat /var/log/syslog | grep dirmngr
Jun  5 09:13:44 ubuntu dirmngr[3005]: resolving 'keyserver.ubuntu.com' failed: Invalid argument
Jun  5 09:13:44 ubuntu dirmngr[3005]: can't connect to 'keyserver.ubuntu.com': host not found

Strangely, using the host command to resolve the same hostname worked perfectly:

root@server:~# host keyserver.ubuntu.com
keyserver.ubuntu.com has address 91.189.89.49
keyserver.ubuntu.com has address 91.189.90.55

This inconsistency suggested different DNS resolution paths were being used. Let’s examine the nameserver configuration:

root@server:~# cat /etc/resolv.conf 
nameserver 127.0.0.53
nameserver 1.0.0.1
nameserver 1.1.1.1

The file lists:

  • 127.0.0.53 – systemd-resolved’s stub listener
  • 1.0.0.1 and 1.1.1.1 – Cloudflare’s public DNS servers

Since 127.0.0.53 appears first, it should handle DNS queries initially. Let’s verify systemd-resolved’s configuration:

root@server:~# cat /etc/systemd/resolved.conf
[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

The stub listener is enabled (last line). Is the service actually running?

root@server:~# systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded
   Active: active (running)

root@server:~# netstat -nlutp | grep 127.0.0.53
tcp  0  0  127.0.0.53:53  0.0.0.0:*  LISTEN  1753/systemd-resolv
udp  0  0  127.0.0.53:53  0.0.0.0:*          1753/systemd-resolv

The service is running and listening on the correct ports. So why the difference between dirmngr and host?

The key is that these tools use different DNS resolution methods:

  • dirmngr uses glibc’s getaddrinfo() function
  • host (part of BIND) makes DNS queries directly

To determine whether systemd-resolved handles glibc calls, we need to check /etc/nsswitch.conf:

root@server:~# cat /etc/nsswitch.conf
hosts:  files dns

The absence of “resolve” in the hosts line means systemd-resolved isn’t handling glibc-based lookups. Therefore, dirmngr queries reach 127.0.0.53 via /etc/resolv.conf, while host bypasses systemd-resolved entirely.

Analyzing Network Traffic

To understand what’s happening, let’s capture DNS traffic with tcpdump while running the failing GPG command:

The packet capture reveals:

  1. Application queries for IPv4 addresses (A records) – systemd-resolved forwards to three nameservers in parallel (Cloudflare and Google DNS)
  2. Application queries for IPv6 addresses (AAAA records) – same parallel forwarding
  3. Application queries for Type 0 (RRSIG records) – systemd-resolved returns a Format Error
  4. Queries timeout after 5 seconds, process repeats

RRSIG records contain DNSSEC digital signatures. Under certain conditions, systemd-resolved doesn’t support queries for these records. We can verify this:

root@server:~# host -t RRSIG keyserver.ubuntu.com 127.0.0.53
Host keyserver.ubuntu.com not found: 1(FORMERR)

root@server:~# host -t RRSIG keyserver.ubuntu.com 1.1.1.1
keyserver.ubuntu.com has no RRSIG record

The Format Error from systemd-resolved explains the GPG failure. But why do all queries timeout when A and AAAA queries succeeded?

Checking systemd-resolved’s status reveals the issue:

root@server:~# systemd-resolve --status
Global
         DNS Servers: 1.0.0.1
                      1.1.1.1

Link 2 (eth0)
         DNS Servers: 8.8.8.8
                      8.8.4.4

Link 3 (eth1)
         DNS Servers: 8.8.8.8
                      8.8.4.4

We have three sets of DNS servers:

  • Cloudflare (1.0.0.1, 1.1.1.1) as global servers from /etc/resolv.conf
  • Google DNS (8.8.8.8, 8.8.4.4) for each network interface from DHCP

This explains the three parallel queries. But where does this configuration come from?

root@server:~# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 29 May  7 12:36 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

The file is managed by resolvconf, yet includes 127.0.0.53. This creates conflicting configurations:

  • systemd-resolved gets DNS servers from DHCP (Google DNS)
  • resolvconf manages /etc/resolv.conf and adds different servers (Cloudflare)
  • Both systems are active simultaneously

When systemd-resolved receives unsupported queries (like RRSIG) while waiting for responses from upstream servers, the entire resolution process fails catastrophically.

The Root Cause

This server’s configuration is fundamentally broken:

  1. Uses systemd-resolved with per-interface DNS from DHCP
  2. Also uses resolvconf to manage /etc/resolv.conf
  3. Lists different DNS servers in both systems
  4. Triggers a systemd-resolved bug with RRSIG queries

The provider should choose one approach—either use systemd-resolved exclusively or disable it completely—not mix both systems with conflicting configurations.

Conclusion

DNS is more complex than it appears, and systemd-resolved adds another layer of complexity in exchange for additional features. According to systemd-resolved’s author:

“resolved is not supposed to be a DNS server, it’s supposed to be exactly good enough so that libc-like DNS clients can resolve their stuff”

Yet distributions like Ubuntu Server enable it by default without careful consideration of how it interacts with existing DNS infrastructure. This creates subtle, hard-to-debug issues like the one documented here.

If you’re managing Ubuntu servers, understand your DNS resolution path and ensure you’re using a single, consistent approach rather than mixing incompatible systems.

Tags: dns, systemd, systemd-resolved, ubuntu, ubuntu-18.04, bug-hunting, sysadmin