Reverse lookup with systemd-resolved and Consul as secondary DNS Server

Not getting any attention at https://github.com/hashicorp/consul/issues/6462
Maybe someone can take a look here, and point me to a configuration issue on my system?

Overview of the Issue

We are running Consul server on Ubuntu 18.04 in AWS. Systemd-resolved setup was followed from this guid: https://learn.hashicorp.com/consul/security-networking/forwarding#systemd-resolved-setup

We don’t have any issues resolving AWS domain or Consul domain, the issue is only related to reverse lookup. We are occasionally seeing instance resolves it’s FQDN with Consul domain.

Example of the same dig command running within few seconds interval:

ip-172-31-28-9:~$ dig -x 172.31.28.9

; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> -x 172.31.28.9
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43381
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;9.28.31.172.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
9.28.31.172.in-addr.arpa. 0	IN	PTR	ip-172-31-28-9.us-east-1.compute.internal.

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Fri Sep 06 18:45:48 UTC 2019
;; MSG SIZE  rcvd: 108

ip-172-31-28-9:~$ dig -x 172.31.28.9

; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> -x 172.31.28.9
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63509
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;9.28.31.172.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
9.28.31.172.in-addr.arpa. 0	IN	PTR	ip-172-31-28-9.node.dc1.consul.

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Fri Sep 06 18:45:49 UTC 2019
;; MSG SIZE  rcvd: 97

Same thing happens when trying to get FQDN in python using socket.getfqdn():

ip-172-31-28-9:~$ cat test.py
import socket
fullname = socket.getfqdn()
print(fullname)
ip-172-31-28-9:~$ python test.py
ip-172-31-28-9.node.dc1.consul
ip-172-31-28-9:~$ python test.py
ip-172-31-28-9.us-east-1.compute.internal

Config

/etc/resolv.conf

nameserver 127.0.0.53
search us-east-1.compute.internal

/etc/systemd/resolved.conf.d/10-consul.conf

[Resolve]
DNS=127.0.0.1
Domains=~consul

/etc/consul.d/agent/config.json

{
  "disable_update_check": true,
  "disable_remote_exec": true,
  "domain": "consul",
  "data_dir": "/var/lib/consul",
  "enable_syslog": true,
  "leave_on_terminate": true,
  "recursors": ["172.31.0.2"]
}

systemd-resolve --status

Global
         DNS Servers: 127.0.0.1
          DNS Domain: ~consul
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 172.31.0.2
          DNS Domain: us-east-1.compute.internal

/etc/iptables/rules.v4

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600
-A OUTPUT -d 127.0.0.1/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
COMMIT
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING -p tcp -m multiport ! --dports 80,443,3006,8080 -j NOTRACK
-A OUTPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -p udp -m udp --sport 8600 -j ACCEPT
-A OUTPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -p udp -m udp --dport 53 -j ACCEPT
-A OUTPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -p tcp -m tcp --sport 8600 -j ACCEPT
-A OUTPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -p tcp -m tcp --dport 53 -j ACCEPT
-A OUTPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -j NOTRACK
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m limit --limit 5/min -m tcp --dport 22 -j LOG --log-prefix "iptables-dropped: "
COMMIT

Reproduction Steps

OS: Ubuntu 18.04
Infrastructure: AWS EC2 instance with default DNS

Systemd-resolved and Iptables setup from the guid: https://learn.hashicorp.com/consul/security-networking/forwarding#systemd-resolved-setup

Run dig -x your_ip_address +short in a loop (when running it 100 times we were getting ~10 names resolved with Consul domain)

OR

Use python socket.getfqdn() to get FQDN

Consul info for both Client and Server

Client info
output from client 'consul info' command here
Server info
agent:
	check_monitors = 0
	check_ttls = 0
	checks = 2
	services = 1
build:
	prerelease =
	revision = 944cc710
	version = 1.6.0
consul:
	acl = disabled
	bootstrap = true
	known_datacenters = 1
	leader = true
	leader_addr = 10.42.10.40:8300
	server = true
raft:
	applied_index = 5732
	commit_index = 5732
	fsm_pending = 0
	last_contact = 0
	last_log_index = 5732
	last_log_term = 8
	last_snapshot_index = 0
	last_snapshot_term = 0
	latest_configuration = [{Suffrage:Voter ID:e5552921-5a4d-3180-8e8d-8275b5151833 Address:10.42.10.40:8300}]
	latest_configuration_index = 1
	num_peers = 0
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 8
runtime:
	arch = amd64
	cpu_count = 4
	goroutines = 80
	max_procs = 4
	os = linux
	version = go1.12.8
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 1
	event_time = 2
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 8
	members = 1
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 5
	members = 1
	query_queue = 0
	query_time = 1

Operating system and Environment details

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=“Ubuntu 18.04.1 LTS”

systemd --version
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

AWS instance with default DNS.