Hi,
I have a consul v1.19.1 and nomad v1.10.4 cluster. I deleted a job from nomad which then appeared to successfully deregister the service from consul and I’m unable to see said job in the output from nomad status or consul catalog services. However, I keep seeing metrics for the deleted service appear in prometheus scrapes and do not understand why. As far as I can tell I have no references in existing running jobs to this deleted service.
sum by (service) (consul_health_service_query{service="ocsp-db"})
With the following magical incantation, I do see some data in consul’s raft.db file. This is probably a red herring, but who know.
$ sudo cat /var/consul/raft/raft.db | tr -dc '[:print:]\n' | grep -i ocsp-db
...
PeerNameServiceIDK_nomad-task-075dcc1a-33ea-f9eb-9f58-623c711209d8-redis-server-ocsp-db-redisServiceNameocsp-dbS
...