Consul 1.5.3/1.5.2 conflict DNS resolve

Consul
1

Just to say in advance this situation does not occur in 1.5.1 and lower

OS: Ubuntu Minimal 18.04 (ami-0340dafef51c18ba1)
Consul: 1.5.2, 1.5.3

For DNS, I am using systemd-resolved, that is configured with following bash script:

#!/usr/bin/env bash

ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

mkdir /etc/systemd/resolved.conf.d
touch /etc/systemd/resolved.conf.d/forward-consul-domains.conf

IPV4=$(ec2metadata --local-ipv4)

printf "[Resolve]\nDNS=${IPV4}\nDomains=~consul\n" > /etc/systemd/resolved.conf.d/forward-consul-domains.conf

iptables -t nat -A OUTPUT -d "${IPV4}" -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d "${IPV4}" -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600

systemctl daemon-reload
systemctl restart systemd-resolved

Consul Server configuration:

autopilot = {
  cleanup_dead_servers = true
  last_contact_threshold = "5s"
  max_trailing_logs = 250
  server_stabilization_time = "10s"
}

connect {
  enabled = true
}

verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
auto_encrypt = {
  allow_tls = true
}
ca_file = "/opt/consul/tls/certs/consul-ca.pem"
cert_file = "/opt/consul/tls/certs/consul-server.pem"
key_file = "/opt/consul/tls/private/consul-server-key.pem"
ports = {
  http = -1
  https = 8501
}

-bind option points to EC2 Private IPV4
-client option points to 0.0.0.0

Problem occurs when agents starts, after that node stops resolving any DNS name, which causes AWS API error. It looks like as soon as agent starts all DNS queries are routed to Consul, but Consul is not bootstrapped yet.

/etc/resolv.conf:

nameserver 10.28.144.24
nameserver 10.28.0.2
search eu-west-1.compute.internal

Logs:

Aug 08 17:57:44 ip-10-28-144-24 bash[1833]: bootstrap_expect > 0: expecting 3 servers
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]: ==> Starting Consul agent...
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:            Version: 'v1.5.3'
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:            Node ID: '71282438-1ccd-6030-3308-7c685a8dfa86'
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:          Node name: 'Consul-Server-10-28-144-24'
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:         Datacenter: 'equilibrium' (Segment: '<all>')
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:             Server: true (Bootstrap: false)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:        Client Addr: [0.0.0.0] (HTTP: -1, HTTPS: 8501, gRPC: -1, DNS: 8600)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:       Cluster Addr: 10.28.144.24 (LAN: 8301, WAN: 8302)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:            Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true, Auto-Encrypt-TLS: true
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]: ==> Log data will now stream in as it occurs:
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO]  raft: Initial configuration (index=0): []
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO]  raft: Node at 10.28.144.24:8300 [Follower] entering Follower state (Leader: "")
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] serf: EventMemberJoin: Consul-Server-10-28-144-24.equilibrium 10.28.144.24
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] serf: EventMemberJoin: Consul-Server-10-28-144-24 10.28.144.24
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] consul: Handled member-join event for server "Consul-Server-10-28-144-24.equilibrium" in area "wan"
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] consul: Adding LAN server Consul-Server-10-28-144-24 (Addr: tcp/10.28.144.24:8300) (DC: equilibrium)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Started HTTPS server on [::]:8501 (tcp)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: started state syncer
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]: ==> Consul agent running!
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s mdns os packet scaleway softlayer triton vsphere
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Joining LAN cluster...
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] discover-aws: Region is eu-west-1
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] discover-aws: Filter instances with Group:Consul=icabbi-ic3-playground-equilibrium
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [WARN] dns: QName invalid: eu-west-1.ec2.amazonaws.com
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [WARN] dns: QName invalid: ec2.eu-west-1.amazonaws.com
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [WARN] dns: QName invalid: eu-west-1.ec2.amazonaws.com
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [WARN] dns: QName invalid: ec2.eu-west-1.amazonaws.com
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] discover-aws: Instance i-0c8af0c1709e54864 has private ip 10.28.144.10
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] discover-aws: Instance i-0155772eed90225d9 has private ip 10.28.144.24
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] discover-aws: Instance i-06a8d129d2f261b5e has private ip 10.28.80.23
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] discover-aws: Instance i-0e7afed4781bfca44 has private ip 10.28.16.11
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] discover-aws: Instance i-052c146fb6a6e0498 has private ip 10.28.16.29
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Discovered LAN servers: 10.28.144.10 10.28.144.24 10.28.80.23 10.28.16.11 10.28.16.29
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: (LAN) joining: [10.28.144.10 10.28.144.24 10.28.80.23 10.28.16.11 10.28.16.29]
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] serf: EventMemberJoin: Consul-Server-10-28-16-11 10.28.16.11
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] serf: EventMemberJoin: Consul-Server-10-28-80-23 10.28.80.23
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] serf: EventMemberJoin: Consul-Server-10-28-16-29 10.28.16.29
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] consul: Adding LAN server Consul-Server-10-28-16-11 (Addr: tcp/10.28.16.11:8300) (DC: equilibrium)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: (LAN) joined: 4
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Join LAN completed. Synced with 4 initial agents
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [WARN]  raft: Failed to get previous log: 21 log not found (last: 0)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] consul: Existing Raft peers reported by Consul-Server-10-28-16-11, disabling bootstrap mode
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] consul: Adding LAN server Consul-Server-10-28-80-23 (Addr: tcp/10.28.80.23:8300) (DC: equilibrium)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] consul: Adding LAN server Consul-Server-10-28-16-29 (Addr: tcp/10.28.16.29:8300) (DC: equilibrium)
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] serf: EventMemberUpdate: Consul-Server-10-28-144-24
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] agent: Synced node info
Aug 08 17:57:44 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:44 [INFO] serf: EventMemberUpdate: Consul-Server-10-28-144-24.equilibrium
Aug 08 17:57:46 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:46 [INFO] serf: EventMemberJoin: Consul-Server-10-28-144-10 10.28.144.10
Aug 08 17:57:46 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:46 [INFO] consul: Adding LAN server Consul-Server-10-28-144-10 (Addr: tcp/10.28.144.10:8300) (DC: equilibrium)
Aug 08 17:57:46 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:46 [INFO] serf: EventMemberUpdate: Consul-Server-10-28-144-10
Aug 08 17:57:46 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:46 [WARN] dns: QName invalid: autoscaling.eu-west-1.amazonaws.com
Aug 08 17:57:47 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:47 [WARN] dns: QName invalid: autoscaling.eu-west-1.amazonaws.com
Aug 08 17:57:48 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:48 [WARN] dns: QName invalid: autoscaling.eu-west-1.amazonaws.com
Aug 08 17:57:48 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:48 [WARN] dns: QName invalid: autoscaling.eu-west-1.amazonaws.com
Aug 08 17:57:48 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:48 [WARN] dns: QName invalid: autoscaling.eu-west-1.amazonaws.com
Aug 08 17:57:48 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:48 [WARN] dns: QName invalid: autoscaling.eu-west-1.amazonaws.com
Aug 08 17:57:52 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:52 [WARN] dns: QName invalid: logs.eu-west-1.amazonaws.com
Aug 08 17:57:56 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:56 [WARN] dns: QName invalid: checkpoint-api.hashicorp.com
Aug 08 17:57:56 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:56 [WARN] dns: QName invalid: checkpoint-api.hashicorp.com
Aug 08 17:57:56 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:56 [WARN] dns: QName invalid: tochigi-2301.herokussl.com
Aug 08 17:57:56 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:56 [WARN] dns: QName invalid: elb046917-1045816452.us-east-1.elb.amazonaws.com
Aug 08 17:57:56 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:56 [WARN] dns: QName invalid: tochigi-2301.herokussl.com
Aug 08 17:57:56 ip-10-28-144-24 bash[1833]:     2019/08/08 17:57:56 [WARN] dns: QName invalid: elb046917-1045816452.us-east-1.elb.amazonaws.com
2

Just double checked, this behaviour starts from version 1.5.2

3

As did not get a response, duplicates issue as bug in Github: https://github.com/hashicorp/consul/issues/6315