Vault injector, kube 1.21, external vault : permission denyed

Vault
1

I have an external Vault (1.9.2), a kubernetes 1.21 with the vault injector(0.18.0).
Followed the documentation:

But I’m getting

==> Vault agent configuration:
Cgo: disabled
Log Level: trace
Version: Vault v1.9.0
2022-01-07T21:51:40.215Z [INFO] template.server: starting template server
2022-01-07T21:51:40.215Z [INFO] (runner) creating new runner (dry: false, once: false)
2022-01-07T21:51:40.216Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"TRACE","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"{{ with secret \"secret/devwebapp/config\" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/credentials.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://external-vault:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,
"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
2022-01-07T21:51:40.217Z [INFO] sink.server: starting sink server
2022-01-07T21:51:40.217Z [INFO] (runner) creating watcher
2022-01-07T21:51:40.261Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://external-vault:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=1s

Not managing to understand where the problem could be.
Another problem is that the same pod deployment sometimes the first time do not activate the injector and get created without it, do not know if related to the same problem.

2

On Vault server log is:

auth.kubernetes.auth_kubernetes_1ca3db30: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token
3

Sounds like the service account that was setup with the auth is not valid anymore or the token was replaced. So the auth is no longer valid. Check the script that’s doing the deployment.

4

That is strange since the service account has been created following the documentation and nothing occurred in the meantime.
I’ve read that there is a problem with kubernetes 1.21 could it be that?

5

I’m a user light user of Kubernetes so I couldn’t say with any authority, but from my own experience 1.21 has so many issues, it’s barely usable. I switched over to vmware tanzu.

6

Hi @luca.gioppo !

I just have the same problem. With GKE 1.23.
Seems to be a problem with the new way how serviceaccount token works and a short-lived time.

So I try to work with the short-lived option to be more secure but no success.

Does it work on your side now ?