I have an EKS cluster setup with 2 node groups, 5 nodes total. I was able to get vault running fine and the vault-agent-init side car was able to mount secrets and my pods started up correctly with the vault secrets mounted. However, while doing an unrelated test the involved deleting the node ec2 instances one by one, everything came back fine except for vault. vault-agent-init now errors with permission denied to http://vault.vault.svc:8200/v1/auth/kubernetes/login and the vault server logs have:
login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token
I checked the service account token and it is 11 days old so has not be recreated. I can try to recreate the service account to see if that fixes it, but I would have expected this to recover automatically. Is there something additional that needs to be setup to enable automatic recovery?
Thanks.
Some additional details: I have the vault setup to auto-unseal with AWS KMS
Vault server started successfully after the restarts:
==> Vault server started! Log data will stream in below:
2021-09-20T21:57:37.481Z [INFO] core: stored unseal keys supported, attempting fetch
2021-09-20T21:57:37.527Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2021-09-20T21:57:37.527Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2021-09-20T21:57:37.528Z [INFO] core: post-unseal setup starting
2021-09-20T21:57:37.529Z [INFO] core: loaded wrapping token key
2021-09-20T21:57:37.529Z [INFO] core: successfully setup plugin catalog: plugin-directory=""
2021-09-20T21:57:37.531Z [INFO] core: successfully mounted backend: type=system path=sys/
2021-09-20T21:57:37.531Z [INFO] core: successfully mounted backend: type=identity path=identity/
2021-09-20T21:57:37.532Z [INFO] core: successfully mounted backend: type=kv path=kv/
2021-09-20T21:57:37.532Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2021-09-20T21:57:37.537Z [INFO] core: successfully enabled credential backend: type=token path=token/
2021-09-20T21:57:37.537Z [INFO] core: successfully enabled credential backend: type=kubernetes path=kubernetes/
2021-09-20T21:57:37.537Z [INFO] rollback: starting rollback manager
2021-09-20T21:57:37.537Z [INFO] core: restoring leases
2021-09-20T21:57:37.544Z [INFO] identity: entities restored
2021-09-20T21:57:37.544Z [INFO] identity: groups restored
2021-09-20T21:57:37.557Z [INFO] expiration: lease restore complete
2021-09-20T21:57:37.590Z [INFO] core: usage gauge collection is disabled
2021-09-20T21:57:37.590Z [INFO] core: post-unseal setup complete
2021-09-20T21:57:37.590Z [INFO] core: vault is unsealed
2021-09-20T21:57:37.590Z [INFO] core: unsealed with stored key