Consul k8s create a lot of tokens

Consul
1

Hello guys,

I have my consul servers 1.10.3 with ACL enabled.
My consul k8s talk to the consul servers via consul-k8s-auth-method to retrieve token when a new pod is started in order to be used into envoy configuration.

I noticed that everytime a new pod is created, a new token is also created.
It turns out that I was have 12XX token created at some point :smile: .
So on my 1st test I decided to create the appropriate ACL policy and bind the freshly created token with it.
Unfortunately, new tokens where created no matter what.
I finally noticed that token are created with the option "AuthMethod": "consul-k8s-auth-method", and that’s why any pods were not fetching the right token event if AuthMethod wasn’t referenced here Access Control List (ACL) - Overview | Consul | HashiCorp Developer.

So I tried to create a the token with this option but I’ve been rejected!

AuthMethod field is disallowed outside of Login

Following the documentation here https://www.consul.io/docs/security/acl/auth-methods:

Applications SHOULD use the consul logout subcommand or the logout API endpoint to destroy their token when it is no longer required.

From my perspective, in the log of the pod, yes I saw many times:

my-dummy-pod-xxxx consul-connect-inject-init 2021-11-03T14:36:33.045Z [INFO]  Consul login complete
my-dummy-pod-xxxx consul-connect-inject-init 2021-11-03T14:36:33.146Z [INFO]  Registered service has been detected: service=my-dummy-pod
my-dummy-pod-xxxx consul-connect-inject-init 2021-11-03T14:36:33.146Z [INFO]  Registered service has been detected: service=my-dummy-pod-sidecar-proxy
my-dummy-pod-xxxx consul-connect-inject-init 2021-11-03T14:36:33.146Z [INFO]  Connect initialization completed
my-dummy-pod-xxxx consul-connect-inject-init     Successfully applied traffic redirection rules

but for sure I never saw any:

my-dummy-pod-xxxx consul-connect-inject-init 2021-11-03T14:36:33.045Z [INFO]  Consul logout complete

and the the token being deleted on consul servers.

So my questions are:

  • if there any fixes coming to clean unused tokens?
  • is there a way that all my pods for 1 service are using the same token?

From my design, I have for the service A:

  • ACL policy
  • ACL token

Service B will never use token from service A.

The funny thing is, the ACL I created is used by the token created by consul-k8s* components.

Is it a best practice to have 1 token for 1 service or it’s better to have 1 token per pod.
If we have 100 pods for a service, we will have 100 token so I’m not sure if it’s a best practice.

Waiting for your answers.