I have Consul installed on K8s, after upgrading from Consul App version 1.13.9 to 1.14.0 and enabling Consul Dataplanes, I have noticed issues with cataloged services loosing connections to the Primary DC after a few minutes servers are started up mostly with the error below:
$ consul catalog services --datacenter xyz
Error listing services: Unexpected response code: 500 (rpc error getting client: failed to get conn: read tcp 10.10.0.1:67541->10.10.4.1:6443: read: connection reset by peer)
Usually a restart of the mesh-gateway deployment sometimes restores the connection and consul catalog services get listed, other times it doesn’t work after a restart of the mesh gateway pods.
This was first noticed right after upgrading to Consul 1.14.0 and enabling consul-dataplanes:1.0.0. Now I have upgraded all the way up to Consul 1.14.11 and consul-dataplane 1.1.11, yet seeing the same issues with Consul catalog services.
BTW: consul-clients have been disabled too but it made no difference as this happens when consul-clients enabled.
Also worthy of mention too is that running $consul members -wan does successfully list all consul wan members.
Side note: I have tried this on 3 DC’s and 5DC’s and its same catalog issues.