Unable to deregister "zombie" services that were registered by Nomad

I have these 3 services that I would describe as zombies. They don’t show up in Nomad. I can’t figure out how to get rid of them. When I am viewing consul monitor and try consul service deregister -id=_nomad-task-5afe9a87-2c66-c60c-5a9b-b9f83130c33b-group-nginx-nginx-loadbalancer-httpr the monitor reports that the service gets de-registered. But then it immediately pops up again. Any ideas?

2 Likes

I finally resolved this.

consul services deregister -id=<id>
service nomad restart
service consul restart

I ran into a zombie service today. I also eventually had to restart the consul agent on a node. Seems a bit heavy-handed and manual, but it did the job. Fortunately I did not need to restart nomad. Seems like there is a probably bug here, but I do not have enough data to submit a report.

sudo systemctl restart consul-agent

Hello, running into the exact same issue but restarting both consul and nomad agent doesn’t remove the zombie service.

1 Like