Dynamically de-register K8s services after Agent/ServiceSync is removed?

Consul
, ,
1

Hi there,

we have a single Consul server (Cluster #1) and a Consul Agent + K8s Service Sync deployments in two separate Kubernetes clusters (Cluster #2 and #3). We have a basic Consul installation without mesh or connect functionality.

Our syncCatalog block for Cluster #2 looks as follows (Cluster #3 is identical, except for the k8sTag) :

syncCatalog:
  enabled: true
  default: true
  toK8S: false
  toConsul: true
  k8sTag: cluster2

The result of this is that all services in Cluster #2 and #3 are synced under (Helm default) synthetic node k8s-sync. This is fine, as long as the agents/service sync is running. If we remove agents/service sync from Cluster #2, the services will not get removed and from UI (or API) it is not possible to tell if it’s a K8s service actively synced by Consul K8s sync or just a stale service from previous deployments.

My main questions are as follows. They are related to each other, so hopefully answering one, would shed some light on others:

  1. How do we deal with the deregistration of services which were registered by agent/service sync which is no longer present (e.g. agent config removed via helm delete)?
  2. Is it possible to set a health check on a synthetic node if we override the consulNodeName value from our Helm deployment?
  3. What is the purpose of Kubernetes nodes which are running the agent? Is it somehow possible to register services to individual nodes instead of any synthetic constructs, so we can piggyback on the health-check functionality provided by the agent? (e.g. nodes removed would also remove the services running in those nodes).
  4. What’s the purpose of using node tags from the PoV of the resource organization? What is the best practice? Tag per cluster? Or synthetic node per cluster?

It would be interesting to hear what are configuration patterns should be used to have a relatively simple Server Cluster + Agent Cluster deployment, that does not require any intervention from the user to handle accidentally deleted configuration, etc.

Thanks a lot!

1 Like