I'm facing an issue when deregister agent service a Consul cluster using the Go SDK

Consul
1

I’m using the Go SDK to connect to a Consul cluster, and the connection address is obtained through Kubernetes service, which includes the addresses of the three servers in the Consul cluster. Registering services works fine, but when it comes to deregistering services, I often receive errors indicating that the service ID cannot be found. After investigation, I suspect that the issue is caused by unstable connections to different servers. For example, during service registration, I may be connected to server-0, but during service deregistration, I may be connected to server-1. How can I solve this issue?

2

It seems you are using Consul in an unexpected/unsupported way.

Normally, you are not supposed to register services directly with the agent API of the server nodes. Rather, you register services with the agent API of a local Consul (non-server) agent running on the same host as the service.

The local agent takes on the responsibility for evaluating health checks for that service instance, and transmits results to the Consul servers when health status changes.

What you’ve done is to tell arbitrary members of the Consul server cluster to additionally take on local agent responsibility for particular service instances.

Hopefully this is of use in understanding how to change your infrastructure.

3

I don’t quite understand the concept of a local agent. My applications are running in pods. Do I need to start another Consul agent as a sidecar?

4

I lack practical experience of using Consul within Kubernetes personally. However I have read enough to be aware that you have to do something beyond just talking directly to the Consul servers - potential options include:

  • A Consul agent sidecar as you suggest, although this option is not generally used at scale, as one Consul agent per pod is quite wasteful of resources.

  • A Consul agent running as a DaemonSet pod - i.e. one per underlying Kubernetes host. This is the approach that was formerly used by HashiCorp’s Consul Helm chart, IIRC.

  • Use Consul Dataplane. I don’t really know what Consul Dataplane is - I haven’t had a personal need to study it yet - but I know it was invented to modify the Consul architecture to fit better into the Kubernetes ecosystem.