How does consul know about previous agent / nodes?

Consul
1

hey folks,

i recently installed consul using its helm chart on a GKE cluster. Plan was to federate it with another cluster after installing it there. Everything worked fine.

For some reason, I had to uninstall the helm chart from the first cluster. Removed the secrets, pvc , sa and proxydefault etc that was leftover manually

Then I reinstall it on the same cluster and I see the following in logs of the consul-server-0

2021-10-13T15:06:58.509Z [INFO]  agent.server.serf.lan: serf: Attempting re-join to previously known node: consul-server-2: 242.5.0.30:8301
2021-10-13T15:07:08.509Z [INFO]  agent.server.serf.lan: serf: Attempting re-join to previously known node: my-sandbox-node-pool-rdsk: 242.5.0.134:8301
2021-10-13T15:07:18.509Z [INFO]  agent.server.serf.lan: serf: Attempting re-join to previously known node: consul-server-1: 242.5.2.154:8301
2021-10-13T15:07:28.510Z [WARN]  agent.server.serf.lan: serf: Failed to re-join any previously known node
2021-10-13T15:08:08.508Z [WARN]  agent: grpc: addrConn.createTransport failed to connect to {alpha-242.5.0.32:8300 0 consul-server-2.alpha <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <nil>->242.5.0.32:8300: operation was canceled". Reconnecting...

So I am curious as to how does it know / remember the previous connections that it had made? And how do I fully clean it?

2

Hi - sorry for the trouble here.
In theory you should just need to delete the PVCs, serviceaccounts and secrets after doing a helm delete, maybe one of your commands failed?

Just an FYI - we have an upcoming new CLI interface which has an uninstall command that does a full cleanup!

I personally use a little bash script for my work:

  helm del consul
  kubectl delete pvc -l "release=consul"
  kubectl delete `kubectl get all | grep consul | awk '{print $1}'`
3

Thanks. I think it had something to do with “lease” CRD? Deleted them and I was able to reinstall it again. Again, not sure if it was exactly that but that did the trick.