Service can't register in consul after unknown failure in kubernetes: Error while renaming Node ID

Greetings,

We are running our service in kubernetes pod with sidecard containing consul agent, also main consul cluster as statefulset.
When any casual problem occurs on service side - k8s reloading pods and service registers in consul without problems.

But we faced some floating problem - something happens (maybe network issues?), then service can’t reregister in consul and got errors:

rpc error making call: failed inserting node: Error while renaming Node ID: "4833aa15-8428-1d1a-46d8-9dba157dbc60": Node name xxx.xxx.xxx is reserved by node c5dc5b48-f105-79f0-7910-3de6629fddd0 with name xxx.xxx.xx

Also service returning 200 - OK healthcheck, but consul forbids registering.

Service:

  • Spring Boot - 2.2.4
  • Spring Cloud - Hoxton

Sidecar consul agent cli:
- consul
- agent
- -retry-join={{.Values.global.consul.server}}
- -datacenter={{ template “alv.consul.datacenter” . }}
- -data-dir=/consul/data
And ‘{“leave_on_terminate”: true}’

Duplicate Node IDs after upgrading to 1.2.3 · Issue #4741 · hashicorp/consul · GitHub but we are encountering same problem on fresh consul version - 1.9.0