Getting "dns: Skipping invalid node for NS records"

Consul
#1

Hi there,

We are getting these warning messages

dns: Skipping invalid node for "servernode1" NS records
dns: Skipping invalid node for "servernode2" NS records
dns: Skipping invalid node for "servernode3" NS records
dns: Skipping invalid node for "servernode4" NS records
dns: Skipping invalid node for "servernode5" NS records

on nodes servernode3, servernode4 and servernode5. The first node and second node are working fine.

These messages suddenly come without any changes on the servers.

I tried to investigate a lot but I can’t find any useful information about this.

I am not sure what’s wrong with the node3,4,5 but this issue causes a metric spiked.

Screen Shot 2022-01-18 at 6.51.33 PM
Screen Shot 2022-01-18 at 6.51.33 PM1920×859 120 KB

Please advise how to resolve this issue.

Thanks

#2

Hi @wphromma,

The error you are seeing should only be logged if the node name contains characters other than A-Z, a-z, 0-9 or -. (See consul/agent/dns.go.)

What version of Consul are you using? Can you also provide the output of consul operator raft list peers so that I can see the names of all servers in your environment?

#3

Hi @blake Thanks for the response.

We are running Consul in version 1.6.1.

This is the output of consul operator raft list peers 

sg-consul-6024.agprod1.agoda.local  fbb78719-a534-af71-aebf-a4a0232e6a55  10.104.206.71:8300   follower  true   3
sg-consul-6020.agprod1.agoda.local  8a3fea0f-fad6-686d-276f-265874e981be  10.104.206.72:8300   leader    true   3
sg-consul-6021.agprod1.agoda.local  738edfec-ba08-d094-4ce7-a22a1e6146a7  10.104.205.35:8300   follower  true   3
sg-consul-6022.agprod1.agoda.local  e52238dd-80ad-b61f-028a-9f20e075711c  10.104.206.73:8300   follower  true   3
sg-consul-6023.agprod1.agoda.local  d31caefc-4350-5d88-ca2e-c3851a8bd73d  10.104.209.205:8300  follower  true   3

The issue happens only on sg-consul-6020, sg-consul-6021, and sg-consul-6024. And from the raft list peers, the output is the same for all nodes.

The side effects we see are that those all issue nodes will call too many RPC calls.