Consul with bind9 as forwarder throws noanswer_response

Consul
1

We have configured a consul based health check for our proxySQL service. Here bind9 has a forward zone for consul servers. it works fine , but it fails once in 24 hours randomly. how to debug this issue?

named.conf =======options {
#listen-on port 53 { 127.0.0.1; };
listen-on port 53 { ; };
#listen-on-v6 port 53 { ::1; };
directory “/var/named”;
dump-file “/var/named/data/cache_dump.db”;
statistics-file “/var/named/data/named_stats.txt”;
zone-statistics yes ;
memstatistics-file “/var/named/data/named_mem_stats.txt”;
allow-query { any; };
forwarders {8.8.8.8; 8.8.4.4;};
recursion yes;
querylog yes;
dnssec-enable no;
dnssec-validation no;

    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.iscdlv.key";
    managed-keys-directory "/var/named/dynamic";

};
include “/etc/named/consul.conf”;

consul.conf=========zone “healthcheck” IN {
type forward;
forward only;
forwarders {
<Consul server IP 1> port 8600;
<Consul server IP 2> port 8600;
<Consul server IP 3> port 8600;
<Consul server IP 4> port 8600;
<Consul server IP 5> port 8600;
<Consul server IP 6> port 8600; };

};

Bind Version===========BIND 9.11.4-P2-RedHat-9.11.4-9.P2.el7 (Extended Support Version) id:7107deb

Below are the bind9 logs :

NORMAL Response

03-Apr-2020 19:38:36.116 client: debug 10: client @0x7fc46c0c61d0 X.X.X.X#60850 (proxysql-slave.service.healthcheck): ns_client_attach: ref = 1
03-Apr-2020 19:38:36.116 client: debug 10: client @0x7fc45c5655a0 X.X.X.X#60850 (proxysql-slave.service.healthcheck): ns_client_attach: ref = 1
03-Apr-2020 19:38:36.116 security: debug 3: client @0x7fc46c0c61d0 X.X.X.X#60850 (proxysql-slave.service.healthcheck): query (cache) ‘proxysql-slave.service.healthcheck/A/IN’ approved
03-Apr-2020 19:38:36.116 security: debug 3: client @0x7fc45c5655a0 X.X.X.X#60850 (proxysql-slave.service.healthcheck): query (cache) ‘proxysql-slave.service.healthcheck/AAAA/IN’ approved
03-Apr-2020 19:38:36.116 client: debug 3: client @0x7fc46c0c61d0 X.X.X.X#60850 (proxysql-slave.service.healthcheck): replace
03-Apr-2020 19:38:36.116 general: debug 3: clientmgr @0x7fc478a4a458: get client
03-Apr-2020 19:38:36.116 general: debug 3: clientmgr @0x7fc478a4a458: recycle
03-Apr-2020 19:38:36.116 client: debug 3: client @0x7fc45c5655a0 X.X.X.X#60850 (proxysql-slave.service.healthcheck): replace
03-Apr-2020 19:38:36.117 general: debug 3: clientmgr @0x7fc478a4a458: get client
03-Apr-2020 19:38:36.117 general: debug 3: clientmgr @0x7fc478a4a458: recycle
03-Apr-2020 19:38:36.117 resolver: debug 1: fetch: proxysql-slave.service.healthcheck/A
03-Apr-2020 19:38:36.117 resolver: debug 1: fetch: proxysql-slave.service.healthcheck/AAAA
03-Apr-2020 19:38:36.117 client: debug 3: client @0x7fc46c0e2e30 (no-peer): udprecv
03-Apr-2020 19:38:36.117 client: debug 3: client @0x7fc45c0e7bb0 (no-peer): udprecv
03-Apr-2020 19:38:36.117 resolver: debug 10: log_ns_ttl: fctx 0x7fc46d835bd0: fctx_create: proxysql-slave.service.healthcheck (in ‘.’?): 1 53109
03-Apr-2020 19:38:36.117 resolver: debug 10: log_ns_ttl: fctx 0x7fc4643726f0: fctx_create: proxysql-slave.service.healthcheck (in ‘.’?): 1 53109
03-Apr-2020 19:38:36.118 resolver: debug 10: received packet from 10.140.33.172#8600
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38011
;; flags: qr aa rd cd; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 3
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;proxysql-slave.service.healthcheck. IN AAAA

;; AUTHORITY SECTION:
;healthcheck. 0 IN SOA ns.healthcheck. hostmaster.healthcheck. (
; 1585922916 ; serial
; 3600 ; refresh (1 hour)
; 600 ; retry (10 minutes)
; 86400 ; expire (1 day)
; 0 ; minimum (0 seconds)
; )

;; ADDITIONAL SECTION:
;proxysql-slave.service.healthcheck. 1 IN TXT “consul-network-segment=”
;proxysql-slave.service.healthcheck. 1 IN TXT “consul-network-segment=”

03-Apr-2020 19:38:36.118 resolver: debug 10: log_ns_ttl: fctx 0x7fc4643726f0: noanswer_response: proxysql-slave.service.healthcheck (in ‘healthcheck’?): 1 53109
03-Apr-2020 19:38:36.118 client: debug 3: client @0x7fc45c5655a0 X.X.X.X#60850 (proxysql-slave.service.healthcheck): send
03-Apr-2020 19:38:36.118 resolver: debug 10: received packet from 10.140.33.172#8600
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10343
;; flags: qr aa rd cd; QUESTION: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;proxysql-slave.service.healthcheck. IN A
;; ANSWER SECTION:
;proxysql-slave.service.healthcheck. 1 IN A X.X.X.X
;proxysql-slave.service.healthcheck. 1 IN A X.X.X.X

;; ADDITIONAL SECTION:
;proxysql-slave.service.healthcheck. 1 IN TXT “consul-network-segment=”
;proxysql-slave.service.healthcheck. 1 IN TXT “consul-network-segment=”

03-Apr-2020 19:38:36.118 resolver: debug 10: log_ns_ttl: fctx 0x7fc46d835bd0: answer_response: proxysql-slave.service.healthcheck (in ‘healthcheck’?): 1 53109

=============================================
At the time of incident

03-Apr-2020 19:39:01.203 client: debug 10: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): ns_client_attach: ref = 1
03-Apr-2020 19:39:01.203 security: debug 3: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): query (cache) ‘proxysql-slave.service.healthcheck/A/IN’ approved
03-Apr-2020 19:39:01.203 client: debug 3: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): replace
03-Apr-2020 19:39:01.203 general: debug 3: clientmgr @0x7fc478a4a458: get client
03-Apr-2020 19:39:01.203 general: debug 3: clientmgr @0x7fc478a4a458: recycle
03-Apr-2020 19:39:01.203 resolver: debug 1: fetch: proxysql-slave.service.healthcheck/A
03-Apr-2020 19:39:01.203 resolver: debug 10: log_ns_ttl: fctx 0x7fc4682bbcb0: fctx_create: proxysql-slave.service.healthcheck (in ‘.’?): 1 53084
03-Apr-2020 19:39:01.203 client: debug 3: client @0x7fc45c0e7bb0 (no-peer): udprecv
03-Apr-2020 19:39:01.229 resolver: debug 10: received packet from 10.120.31.118#8600
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14713
;; flags: qr aa rd cd; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;proxysql-slave.service.healthcheck. IN A

;; AUTHORITY SECTION:
;healthcheck. 0 IN SOA ns.healthcheck. hostmaster.healthcheck. (
; 1585922941 ; serial
; 3600 ; refresh (1 hour)
; 600 ; retry (10 minutes)
; 86400 ; expire (1 day)
; 0 ; minimum (0 seconds)
; )

03-Apr-2020 19:39:01.229 resolver: debug 10: log_ns_ttl: fctx 0x7fc4682bbcb0: noanswer_response: proxysql-slave.service.healthcheck (in ‘healthcheck’?): 1 53084
03-Apr-2020 19:39:01.229 resolver: debug 10: log_ns_ttl: fctx 0x7fc4682bbcb0: noanswer_response: proxysql-slave.service.healthcheck (in ‘healthcheck’?): 1 53084
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): send
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): sendto
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): senddone
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): next
03-Apr-2020 19:39:01.229 client: debug 10: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): ns_client_detach: ref = 0
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc46c0e2e30 X.X.X.X#59837 (proxysql-slave.service.healthcheck): endrequest
03-Apr-2020 19:39:01.229 query-errors: debug 4: fetch completed at …/…/…/lib/dns/resolver.c:8711 for proxysql-slave.service.healthcheck/A in 0.025326: success/success [domain:healthcheck,referral:0,restart:1,qrysent:1,timeout:0,lame:0,quota:0,neterr:0,badresp:0,adberr:0,findfail:0,valfail:0]
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023: UDP request
03-Apr-2020 19:39:01.229 client: debug 5: client @0x7fc45c0e7bb0 X.X.X.X#33023: using view ‘_default’
03-Apr-2020 19:39:01.229 security: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023: request is not signed
03-Apr-2020 19:39:01.229 security: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023: recursion available
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023: query
03-Apr-2020 19:39:01.229 client: debug 10: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): ns_client_attach: ref = 1
03-Apr-2020 19:39:01.229 security: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): query (cache) ‘proxysql-slave.service.healthcheck/A/IN’ approved
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): send
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): sendto
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): senddone
03-Apr-2020 19:39:01.229 client: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): next
03-Apr-2020 19:39:01.230 client: debug 10: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): ns_client_detach: ref = 0
03-Apr-2020 19:39:01.230 client: debug 3: client @0x7fc45c0e7bb0 X.X.X.X#33023 (proxysql-slave.service.healthcheck): endrequest