I have the consul template container running within my pod that talks to vault via Kubernetes auth method to retrieve my secrets which is then used by my application container running within the same pod.
It works very well when we deploy the pod for the first time…It can talk to vault pull credential successfully… all is well…However after few hours the pod gets restarted over and over again. On looking at consul template container logs I see this error
permission denied (retry attempt 9 after “1m0s”)
2019/09/20 19:01:31.382180 [WARN] (view) vault.read(secret/spacemgr/config): vault.read(secret/spacemgr/config): Error making API request.
URL: GET https:///v1/secret/spacemgr/config
Code: 403. Errors:
Why would vault return permission denied. I am able to connect successfully and retrieve all my creds successfully for initial few hours but all of a sudden after several hours it starts to get this error and the pods gets restarted over and over.
Any idea about this error and resolution is appreciated. Thanks
Without seeing the full config, I think the token which is used to access Vault is being revoked after a period of time. Are you using Vault Agent Auth to renew the token?
Thanks Nic for getting back on this issue.
Yes I am using vault-agent-auth which loads as initContainer for the very first time I deploy the pod…Consul-template then uses the token received from Vault to grab all my secrets.
May be Do I need to keep the vault-agent-auth container run as main container instead of initContainer ?
Wow I am sorry for the late reply, I totally missed this.
The way vault-agent-auth works is that it has the capability to renew the Vault token when it expires. Running the Vault-Agent as a long running sidecar process will ensure that Consul Template will always have an up to date token available to it.
If you run Vault-Agent as an init container, once the token expires Consul Template will no longer be able to retrieve secrets from Vault.
No Worries Nic, I moved the vault agent container to run as sidecar and now the token get renewed automatically. No more restarts now and working like a charm now. Thanks for your pointer which took me to resolution.