Zombie services not automatically deregistered

I am using Consul 1.16.0 on Azure Kubernetes service, without Nomad or helm charts. When my microservices restart, there is good chance that the old instance is not deregistered properly and remains as a zombie (ex. in the image below, all the services above are zombies while the last one is working). What I’m confused about is the inconsistency - when a pod restarts, the old instance is sometimes able to deregister from Consul, while other times it becomes a zombie.

This is not a problem with consul failing to detect that the service is in critical condition, since I can see the critical services in my consul logs:

2023-09-18T20:01:27.384Z [WARN]  agent: Check is now critical: check=service:sr-ms-core-38d7a764f727f988124fa9052bc53a3d
2023-09-18T20:01:27.384Z [WARN]  agent: Check is now critical: check=service:sr-ms-core-a44f3691468908a1a11661ecefe63234
2023-09-18T20:01:27.384Z [WARN]  agent: Check is now critical: check=service:sr-ms-core-cb65441139767511a9d527d0a2fb07d3

Currently, a manual workaround is to kubectl exec into the consul pods and run consul services deregister -id <service-id>, but this is very tedious and inefficient, so I’m looking for a better solution or workaround.

I was wondering if there is anything I’m missing in the Consul config in my manifests:

        - name: consul
          image: docker.io/bitnami/consul:1.16.0
          imagePullPolicy: "IfNotPresent"
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1001
            - name: http
              containerPort: 8500
            - name: rpc
              containerPort: 8400
            - name: serflan-tcp
              protocol: "TCP"
              containerPort: 8301
            - name: serflan-udp
              containerPort: 8301
              protocol: "UDP"
            - name: rpc-server
              containerPort: 8300
            - name: dns-tcp
              containerPort: 8600
            - name: dns-udp
              containerPort: 8600
              protocol: "UDP"
              cpu: "100m"
              memory: "512Mi"
            - name: BITNAMI_DEBUG
              value: "false"
            - name: CONSUL_NODE_NAME
                  fieldPath: metadata.name
            - name: CONSUL_RETRY_JOIN
              value: "consul-headless.default.svc.cluster.local"
              value: "true"
            - name: CONSUL_BOOTSTRAP_EXPECT
              value: "3"
            - name: CONSUL_RAFT_MULTIPLIER
              value: "1"
            - name: CONSUL_DOMAIN
              value: "consul"
            - name: CONSUL_DATACENTER
              value: "dc1"
            - name: CONSUL_UI
              value: "true"
            - name: CONSUL_HTTP_PORT_NUMBER
              value: "8500"
            - name: CONSUL_DNS_PORT_NUMBER
              value: "8600"
            - name: CONSUL_RPC_PORT_NUMBER
              value: "8400"
            - name: CONSUL_SERF_LAN_PORT_NUMBER
              value: "8301"
                - consul
                - operator
                - raft
                - list-peers
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
                - consul
                - members
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
                  - consul
                  - leave
            - name: data
              mountPath: /bitnami/consul

I have also heard about the deregister_critical_service_after config option, but I’m not sure how to apply it in my Kubernetes manifests. (However, I would prefer a more elegant solution if it exists, because this option requires the critical service to exist for a certain period of time before being deregistered.)

Hi @ML72,

Welcome to the HashiCorp Forums!

Could you please share why you are not using the Consul-K8S Helm Chart for setting up the cluster?

Considering that you are not using the Official Helm Chart, providing more details on your service registration and deregistration workflow will help better understand the issue.

If you want Consul to deregister the critical services, as you rightly identified, deregister_critical_service_after will be the right option to use.

Hello @Ranjandas !

I am not using the Consul-K8S Helm Chart because my microservices are spring boot applications generated by JHipster, and I used the JHipster kubernetes subgenerator to initialize my kubernetes manifests. By default, consul configuration was provided without Helm, and I simply went along with it.

Regarding my service registration and deregistration workflow, my microservices provide health check endpoints that consul calls to determine the status of services. The actual registration/deregistration process is the default provided by Consul, as I haven’t made any configuration changes for that.

If I were to use deregister_critical_service_after, how would I add it to my manifests?

Hi @ML72,

Unfortunately, I am not familiar with either of them. But I did a quick search and looks like you will have to set spring.cloud.consul.discovery.health-check-critical-timeout in the Spring Cloud configuration.

spring.cloud.consul.discovery.health-check-critical-timeout: Timeout to deregister services critical for longer than timeout (e.g. 30m).
ref: https://docs.spring.io/spring-cloud-consul/docs/current/reference/html/appendix.html

I hope you know how to configure the above in your Spring Cloud setup and that this helps.

1 Like

Thank you, I tried that and it seems to work equivalently to deregister_critical_service_after!

1 Like

For me, I am unable to deregister the service using the command as well. Able to see the service in the catalogue but unable to delete it

Error deregistering service “”: Unexpected response code: 404 (Unknown service ID “test-service”. Ensure that the service ID is passed, not the service name.)

It seems to be service in orphaned and unable to remove it

How to handle this situation?

Finally was able to remove the zombie services following are the steps I followed

  1. Login to the consul agent pod that runs in the same node as that of the consul server
  2. Execute the following command curl -H "Authorization: Bearer <TOKEN>" --request PUT http://<consul-server-ip>:8500/v1/agent/service/deregister/serviceId

Was able to figure it out with the help of this script Unable to deregister a service · Issue #1188 · hashicorp/consul · GitHub