# Consul will not resolve DNS names on ubuntu 18.04

**URL:** <https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129>\
**Category:** Consul\
**Tags:** dns\
**Created:** [July 25, 2019, 5:19pm UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129 "2019-07-25T17:19:13Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![spuder](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/spuder/32/1638_2.png) [@spuder](https://discuss.hashicorp.com/u/spuder)\
**Post date:** [July 25, 2019, 5:19pm UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/1 "2019-07-25T17:19:13Z")

</div>

I’m trying to setup a consul + nomad cluster on ubuntu 18.04. Attempting to use the `systemd-resolvd` setup method [as documented here](https://learn.hashicorp.com/consul/security-networking/forwarding#systemd-resolved-setup)

/etc/systemd/resolved.conf

```auto
[Resolve]
DNS=127.0.0.1
Domains=~consul

```

```auto
service systemd-resolved restart

```

iptables are configured

```auto
iptables -t nat -A OUTPUT -d localhost -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600

```

```auto
iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- anywhere localhost.localdomain tcp dpt:domain redir ports 8600
REDIRECT udp -- anywhere localhost.localdomain udp dpt:domain redir ports 8600

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

```

I’ve ensured that consul port 8600 is listening on 127.0.0.1 and not the public ip

```auto
netstat -plnt | grep consul
tcp 0 0 10.47.80.7:8301 0.0.0.0:* LISTEN 21245/consul
tcp 0 0 127.0.0.1:8500 0.0.0.0:* LISTEN 21245/consul
tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN 21245/consul

```

Yet I’m still not able to resolve .consul domains automatically

```auto
host foobar.service.consul
Host foobar.service.consul not found: 3(NXDOMAIN)

```

```auto
ping foobar.service.consul
ping: foobar.service.consul: Name or service not known

```

```auto
dig foobar.service.consul
# no results

```

Whereas these requests do work

```auto
dig @127.0.0.1 foobar.service.consul
# works

```

```auto
dig @127.0.0.1 -p 8600 foobar.service.consul
# works

```

Consul is configured as minimally as possible

```auto
{
  "acl": {
    "default_policy": "deny",
    "enable_token_persistence": true,
    "enabled": true
  },
  "bind_addr": "10.x.x.x",
  "data_dir": "/opt/consul",
  "datacenter": "foobar",
  "log_level": "INFO",
  "primary_datacenter": "",
  "retry_join": [
    "10.x.x.x",
    "10.x.x.x",
    "10.x.x.x"
  ],
  "server": false,
  "ui": false
}

```

What am I missing? How can I get consul to work on ubuntu 18.04?

---

<div class="post-metadata">

**Author:** ![fortman](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/fortman/32/1026_2.png) [@fortman](https://discuss.hashicorp.com/u/fortman)\
**Post date:** [July 25, 2019, 7:44pm UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/2 "2019-07-25T19:44:41Z")

</div>

We’ve had nothing but trouble with systemd-resolved and ripped that sucker out

---

<div class="post-metadata">

**Author:** ![spuder](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/spuder/32/1638_2.png) [@spuder](https://discuss.hashicorp.com/u/spuder)\
**Post date:** [July 25, 2019, 7:51pm UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/3 "2019-07-25T19:51:28Z")

</div>

How do you get consul working on 18.04? I’ve tried ‘bind’, ‘unbound’ and ‘dnsmasq’ in addition to ‘systemd-resolvd’ with no luck.

---

<div class="post-metadata">

**Author:** ![vasilij-icabbi](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/vasilij-icabbi/32/1191_2.png) [@vasilij-icabbi](https://discuss.hashicorp.com/u/vasilij-icabbi)\
**Post date:** [July 28, 2019, 10:13pm UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/4 "2019-07-28T22:13:49Z")

</div>

@spuder the answer to your question is here:

> [@DNS Forwarding using systemd-resolved on AWS Ubuntu Minimal 18.04](https://discuss.hashicorp.com/t/dns-forwarding-using-systemd-resolved-on-aws-ubuntu-minimal-18-04/1656):
>
> Hello, As discussed here [https://github.com/hashicorp/consul/issues/5875](https://github.com/hashicorp/consul/issues/5875) I open a topic in this discussion board, sorry for a delay. Documentation [here](https://learn.hashicorp.com/consul/security-networking/forwarding#systemd-resolved-setup) did not work for me for several reasons that will be outlined below. I am using AWS as a cloud platform and Ubuntu Minimal 18.04 AMI. Consul Agent is 1.5.1 and runs as a server. Internal communication and client interfaces are bind to local IPv4 (10.0.0.0 will be used as example) using both -bind and -client options. There are no custom DNS c…

Long story short, bind your client to 0.0.0.0

---

<div class="post-metadata">

**Author:** ![queglay](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/queglay/32/1858_2.png) [@queglay](https://discuss.hashicorp.com/u/queglay)\
**Post date:** [January 14, 2021, 11:39am UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/5 "2021-01-14T11:39:39Z")

</div>

I’m pretty new to all this, but shared similar issues, that I partly resolved.

The only reason why I use ubuntu 18 is because of the AWS vault deployment example, and I also use open vpn ami’s which are ubuntu 18. I’m open to alternatives though…

I tried to use [https://github.com/hashicorp/terraform-aws-consul.git](https://github.com/hashicorp/terraform-aws-consul.git)

In ubuntu 18, I found that /etc/resolv.conf was not linked to /run/systemd/resolve/resolv.conf, and although I could be wrong, this seems incorrect

I assume this is why it attempts to use 127.0.0.53 to resolve dns, which is definitely a part of the problem

So if I do this,

```auto
"sudo unlink /etc/resolv.conf",
"sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf"

```

Then this now works where previously it didn’t:

```auto
dig vault.service.consul

```

but… as soon as I do that,  
i’ll see another error when I use sudo:

```auto
sudo: unable to resolve host ip-172-31-19-129: Resource temporarily unavailable

```

And that is because the hostname (ip-172-31-19-129) doesn’t exist in /etc/hosts.  
So if ubuntu doesn’t set the hostname in /etc/hosts, how does it resolve it normally? Considering this information, I wonder if im on the right path and there is just some other missing piece…

I would love to find a golden answer to this problem. It seems google produces so many different solutions to the topic…

---

<div class="post-metadata">

**Author:** ![Wolfsrudel](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wolfsrudel/32/28465_2.png) [@Wolfsrudel](https://discuss.hashicorp.com/u/Wolfsrudel)\
**Post date:** [January 14, 2021, 11:46am UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/6 "2021-01-14T11:46:02Z")

</div>

I think you’ll have to set the `recursors` to use your “normal” dns for every request consul isn’t responsible for:

> [`recursors`](https://www.consul.io/docs/agent/options.html#recursors) This flag provides addresses of upstream DNS servers that are used to recursively resolve queries if they are not inside the service domain for Consul. For example, a node can use Consul directly as a DNS server, and if the record is outside of the “consul.” domain, the query will be resolved upstream. As of Consul 1.0.1 recursors can be provided as IP addresses or as go-sockaddr templates. IP addresses are resolved in order, and duplicates are ignored.

> **[Agents - Configuration File Reference | Consul | HashiCorp Developer](https://developer.hashicorp.com/consul/docs/agent/config/config-files)**
>
> Use agent configuration files to assign attributes to agents and configure multiple agents at once. Learn about agent configuration file parameters and formatting with this reference page and sample code.

```auto
# upstream DNS servers
recursors = ["<your-regular-dns-here>"]

```

Maybe this was a different issue on my configuration, but it’s the first that comes to my mind. 🙂

---

<div class="post-metadata">

**Author:** ![queglay](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/queglay/32/1858_2.png) [@queglay](https://discuss.hashicorp.com/u/queglay)\
**Post date:** [January 14, 2021, 12:10pm UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/7 "2021-01-14T12:10:44Z")

</div>

Thanks @Wolfsrudel !

I will try that out.

There may be another problem additionally though - normally shouldn’t the hostname exist in /etc/hosts? And since preserve\_hostname is disabled, I don’t understand why ubuntu is not adding the hostname in there on startup (or if it needs to). Is there anyway I can find out what startup script actually operates on preserve\_hostname and how it is supposed to update /etc/hosts if the hostname changes all the time? or am I heading down the wrong path with this thinking?

---

<div class="post-metadata">

**Author:** ![queglay](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/queglay/32/1858_2.png) [@queglay](https://discuss.hashicorp.com/u/queglay)\
**Post date:** [January 14, 2021, 12:14pm UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/8 "2021-01-14T12:14:42Z")

</div>

For posterity - this was also educational - [linux - Why does /etc/resolv.conf point at 127.0.0.53? - Unix & Linux Stack Exchange](https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-point-at-127-0-0-53)

---

<div class="post-metadata">

**Author:** ![queglay](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/queglay/32/1858_2.png) [@queglay](https://discuss.hashicorp.com/u/queglay)\
**Post date:** [January 16, 2021, 5:45am UTC](https://discuss.hashicorp.com/t/consul-will-not-resolve-dns-names-on-ubuntu-18-04/2129/9 "2021-01-16T05:45:35Z")

</div>

I have solved this. I was making false assumptions and there were a bunch of things I was doing wrong that sent me down the wrong track.

Firstly, the dhcp options, definitely helped.

```auto
resource "aws_vpc_dhcp_options" "main" {
  count = var.create_vpc ? 1 : 0
  domain_name = "service.consul"
  domain_name_servers = ["127.0.0.1", "AmazonProvidedDNS"]
  ntp_servers = ["127.0.0.1"]
  netbios_name_servers = ["127.0.0.1"]
  netbios_node_type = 2
  tags = merge(var.common_tags, local.extra_tags, map("Name", format("dhcpoptions_%s", local.name)))
}

```

I also learnt that I made a big mistake by sym linking this:

```auto
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

```

…before I had problems because I didn’t have the correct dhcp options, and so doing the above step got me around that, and got consul working, but it broke everything else.  
It is best to leave the default stub resolver.

I also found that the default settings that the systemd-resolved helper applies were incorrect.

> **[hashicorp/terraform-aws-consul](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved)**
>
> master/modules/setup-systemd-resolved

```auto
Domains=~consul

```

…was not consistently working, especially after booting the ami on another instance, and also for an open vpn ubuntu 18 AMI I had. I read somewhere that an FQDN in this config needed to end with a ‘.’, I can’t be 100% sure if it was that or just the more specific name, but in changing just this line, dig commands for that domain suddenly switched from 127.0.0.53 to the correct 127.0.0.1 and worked …

```auto
Domains=~service.consul.

```

so I run this now before I install the systemd-resolved script…

```auto
sudo sed -i "s/#Domains=/Domains=~service.consul./g" /etc/systemd/resolved.conf

```

Doing these steps, the recursors setting was also not required.

This is an example snippet from my current packer template that is working for ubuntu 18.

```auto
  provisioner "shell" { # Generate certificates with vault.
    inline = [
      "set -x; sudo sed -i \"s/#Domains=/Domains=~service.consul./g\" /etc/systemd/resolved.conf",
      "set -x; /tmp/terraform-aws-consul/modules/setup-systemd-resolved/setup-systemd-resolved",
      "set -x; sudo systemctl daemon-reload",
      "set -x; sudo systemctl restart systemd-resolved",
      "set -x; sudo cat /etc/systemd/resolved.conf",
      "set -x; sudo /opt/consul/bin/run-consul --client --cluster-tag-key \"${var.consul_cluster_tag_key}\" --cluster-tag-value \"${var.consul_cluster_tag_value}\"", # this is normally done with user data but dont for convenience here
      "set -x; consul members list",
      "set -x; dig $(hostname) | awk '/^;; ANSWER SECTION:$/ { getline ; print $5 ; exit }'", # check localhost resolve's
      "set -x; dig @127.0.0.1 vault.service.consul | awk '/^;; ANSWER SECTION:$/ { getline ; print $5 ; exit }'", # check consul will resolve vault
      "set -x; dig @localhost vault.service.consul | awk '/^;; ANSWER SECTION:$/ { getline ; print $5 ; exit }'", # check localhost will resolve vault
      "set -x; dig vault.service.consul | awk '/^;; ANSWER SECTION:$/ { getline ; print $5 ; exit }'", # check default lookup will resolve vault
      ]
  }

```

If I can see at the end of this segment that the 4 dig commands resolve IP addresses then it was succesful. to be doubly sure, this check (with dig) should also be done on the next boot.