DNS works on host, not in container

Nomad
,
1

Bonjour, :wave:

Sorry, I know there is lot of topics about this… I tried many ways, without good results.

Ubuntu 22.04
Nomad 1.7.2 (ACL & TLS)
Consul 1.17.1 (ACL & TLS)

On my host, my /etc/systemd/resolved.conf.d/consul.conf

DNS=127.0.0.1:8600
DNSSEC=no
DNSOverTLS=no
Domains=~consul
Cache=no

I add to anonymous token on Consul, authorizations.

So, this works: dig consul.service.consul

; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> consul.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32091
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;consul.service.consul.		IN	A

;; ANSWER SECTION:
consul.service.consul.	0	IN	A	127.0.0.1

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Jan 15 10:13:34 CET 2024
;; MSG SIZE  rcvd: 66

:partying_face:

But… In container launch by Nomad.

/ # ping consul.service.consul
ping: bad address 'consul.service.consul'

/ # dig consul.service.consul

; <<>> DiG 9.18.19 <<>> consul.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 6320
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul.service.consul.         IN      A

;; AUTHORITY SECTION:
.                       1062    IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2024011500 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 192.168.64.1#53(192.168.64.1) (UDP)
;; WHEN: Mon Jan 15 09:20:50 UTC 2024
;; MSG SIZE  rcvd: 125

But, with parameters add to dig.

/ # dig @172.26.64.1 -p 8600 consul.service.consul

; <<>> DiG 9.18.19 <<>> @172.26.64.1 -p 8600 consul.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6983
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;consul.service.consul.         IN      A

;; ANSWER SECTION:
consul.service.consul.  0       IN      A       127.0.0.1

;; Query time: 0 msec
;; SERVER: 172.26.64.1#8600(172.26.64.1) (UDP)
;; WHEN: Mon Jan 15 09:22:12 UTC 2024
;; MSG SIZE  rcvd: 66

:ok_hand:

I tried to add to my nomad job tester:

  		dns {
    			servers = ["172.26.64.1:8600"]
  		}

or

  		dns {
    			servers = ["172.26.64.1"]
  		}

No more works. service.consul resolution does not work :sob:

I tried to follow this answer on Github: Question - DNS Resolution from Consul inside of Mesh Network · Issue #8343 · hashicorp/nomad · GitHub

But already at begining, when I tried: dig @172.26.64.1 -p 53 nomad-clients.service.dc1.consul ANY

dig @172.26.64.1 -p 53 nomad-clients.service.dc1.consul ANY
;; Connection to 172.26.64.1#53(172.26.64.1) for nomad-clients.service.dc1.consul failed: connection refused.
;; Connection to 172.26.64.1#53(172.26.64.1) for nomad-clients.service.dc1.consul failed: connection refused.
;; Connection to 172.26.64.1#53(172.26.64.1) for nomad-clients.service.dc1.consul failed: connection refused.

I don’t understand… :ring_buoy: :disappointed_relieved:

Thanks!

2

Hi @fred-gb,

You should add the DNSStubListenerExtra=<docker bridge IP> in your consul.conf (under resolved.conf.d) and restart systemd-resolved. Once this is done, use the dns {} block in the job (as you already did), and everything should work fine.

ref: Update deployment to use the latest Ubuntu LTS Image by Ranjandas · Pull Request #23 · hashicorp/learn-nomad-cluster-setup · GitHub

Please let me know if this works for you.

1 Like
3

Thanks @Ranjandas ! :slightly_smiling_face:

For those who read this topic, this is my working configuration.

/etc/systemd/resolved.conf.d/consul.conf

[Resolve]
DNS=127.0.0.1:8600
DNSSEC=no
DNSOverTLS=no
Domains=~consul
Cache=no
DNSStubListenerExtra=172.17.0.1

And in my job:

  group "mosquitto-server" {

    network {
  	  dns {
    	servers = ["172.17.0.1"]
  	  }
      mode = "bridge"

        port "mqtt" {
        to = 1883
        static = 1883
      }
    }

Does exist a plugin docker configuration in Nomad to put this DNS by default? I don’t find or don’t understand in doc.

Thanks again