Prepared_query doesn't work for services on kubernetes pods

Consul
1

We have a consul server cluster on AWS VMs. We also have 2 EKS cluster: one in us-east and one in us-west.

We then install two consul-k8s charts, one to each of the EKS clusters. The consul-k8s nodes are connected as clients to our consul server cluster.

When we deploy a service to our EKS clusters (a db-service with the consul dns of db-service-release.query.consul in our case), we are able to query it through consul. So the connection and service registration definitely work.

However, the service/pod IPs are returned to us in a shuffled manner (sometimes the us-east pod IPs are returned first, sometimes the us-west ones are returned first). We have never run into this issue when querying for services running on actual VMs. For services running on VMs, the closest IPs to the querying server are always returned first.

Here is an example output when we query for services on vms

localhost:~# nslookup database.query.consul
Server:		[consul_server_1]
Address:	[consul_server_1]#53

Name:	database.query.consul
Address: [database_US-WEST_server_1]
Name:	database.query.consul
Address: [database_US-WEST_server_2]
Name:	database.query.consul
Address: [database_US-EAST_server_1]

vault-consul-server/fbfb81fa-39b4-4bcb-aff1-3750d3c1766a:~# nslookup db-service.query.consul
Server:		[consul_server_1]
Address:	[consul_server_1]#53

Name:	database.query.consul
Address: [database_US-WEST_server_1]
Name:	database.query.consul
Address: [database_US-WEST_server_2]
Name:	database.query.consul
Address: [database_US-EAST_server_1]

Here is an example output when we query for services running on kubernetes pods

vault-consul-server/fbfb81fa-39b4-4bcb-aff1-3750d3c1766a:~# nslookup db-service-release.query.consul
Server:		[consul_server_1]
Address:	[consul_server_1]#53

Name:	database-kube.query.consul
Address: [database-kube_US-WEST_server_1]
Name:	database-kube.query.consul
Address: [database-kube_US-EAST_server_1]
Name:	database-kube.query.consul
Address: [database-kube_US-WEST_server_2]

vault-consul-server/fbfb81fa-39b4-4bcb-aff1-3750d3c1766a:~# nslookup db-service-release.query.consul
Server:		10.128.1.11
Address:	10.128.1.11#53

Name:	database-kube.query.consul
Address: [database-kube_US-EAST_server_1]
Name:	database-kube.query.consul
Address: [database-kube_US-WEST_server_2]
Name:	database-kube.query.consul
Address: [database-kube_US-WEST_server_1]

We dig in a little further a notice that every single one of our services running on kubernetes pods are registering to consul server with an address of “127.0.0.1” (the ‘ServiceAddress’ correctly shows the pod IP, but the ‘Address’ just shows ‘127.0.0.1’). We believe this is the reason consul hasn’t been able to properly perform rtt test against the services on kubernetes pods.

Below is an example consul catalog output of our db-service-release service:

[
  {
    "ID": "",
    "Node": "k8s-sync",
    "Address": "127.0.0.1",
    "Datacenter": "dc1",
    "TaggedAddresses": null,
    "NodeMeta": {
      "external-source": "kubernetes"
    },
    "ServiceKind": "",
    "ServiceID": "database-kube-2b5c960fe63a",
    "ServiceName": "database-kube",
    "ServiceTags": [
    ],
    "ServiceAddress": "[database-kube_US-WEST_server_1]",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
...
]

The full catalog service output file is attached.
database-kube-output.txt (4.8 KB)

Is there a way to adjust the ‘Address’ value of the services on kubernetes pods to reflect the pod or the node IPs? Or is there any other way to enable consul to properly execute prepared_query against services on kubernetes pods and the return the closest IPs first?