Can't resolve Consul DNS records from within Docker container for some minutes after initialization

Consul
1

I’m having an issue resolving Consul dns records from within my containers.

My consul server is located at 192.168.1.103. This host has systemd-resolved listening on 192.168.1.103:53 along with 127.0.0.1:8600 configured as the consul search domain as per Forward DNS for Consul Service Discovery | Consul - HashiCorp Learn.

From that host I can query service DNS entries without any issues, but my containers are having issues. For the first 6-7 minutes after the container initializes no records can be found, then at some point it starts working.

Here’s a sample job I can replicate the issue with.

job "dnsutils" {
  datacenters = ["dc1"]
  group "dnsutils" {
    network {
      mode = "bridge"
      dns {
        servers  = ["192.168.1.103"]
        searches = ["."]
        options  = []
      }
    }
    service {
      name = "dnsutils"
    }
    task "dnsutils" {
      driver = "docker"
      config {
        image = "tutum/dnsutils:latest"
        entrypoint = ["tail", "-f", "/dev/null"]
        dns_servers = ["192.168.1.103"]
      }
      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

Just after the container starts:

nomad alloc exec 0d4b2be4-3a0e-68e0-a8f4-b57fe915cde3 dig dnsutils.service.consul

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> dnsutils.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14150
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;dnsutils.service.consul.       IN      A

;; AUTHORITY SECTION:
consul.                 0       IN      SOA     ns.consul. hostmaster.consul. 1655084579 3600 600 86400 0

;; Query time: 0 msec
;; SERVER: 192.168.1.103#53(192.168.1.103)
;; WHEN: Mon Jun 13 01:42:59 UTC 2022
;; MSG SIZE  rcvd: 102

After waiting a few minutes:

nomad alloc exec 0d4b2be4-3a0e-68e0-a8f4-b57fe915cde3 dig dnsutils.service.consul

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> dnsutils.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34690
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;dnsutils.service.consul.       IN      A

;; ANSWER SECTION:
dnsutils.service.consul. 0      IN      A       192.168.1.103

;; Query time: 0 msec
;; SERVER: 192.168.1.103#53(192.168.1.103)
;; WHEN: Mon Jun 13 01:49:24 UTC 2022
;; MSG SIZE  rcvd: 68

Is this maybe a cache thing?