Hello,
we have external services that we bootstrap during the initial deployment. AFAIK, while registering external services consul picks up random agent pod and registers the service using that pod present in the node as agent pods run as a daemonset.
Problem Statement
when the said agent pod restarts/ gets replaced by a new one/underlying node gets removed from k8s cluster. we lose those external services.
there are many applications running inside our k8s cluster that relies on these external services, so the moment that node goes away (which registered those external services into consul catalog during initial deployment) we are seeing an outage in the application.
** Steps to reproduce ** : Posted it in Consul catalog services disappears when underlying node goes away where client agent pod was running as daemonset · Issue #10057 · hashicorp/consul · GitHub
consul version 1.8
Currently, we are not using Consul ESM. Is this answer to my issue? please remember, these are external AWS service URLs and NOT external nodes.
As per the document, Consul ESM is a daemon that runs alongside Consul in order to health check external nodes and updates their status.
But here, the question is what happens if an agent node through which we have directly registered the external service goes away to the pod get restarted. what happens then
we want to make sure the external services persists in consul catalog even if consul agent node goes down or pods crashes or restarts happens.