Kubernetes, vault-agent-init sidecar keeps failing with auth error

I have been following the tutorial for injecting secrets into pods in kubernetes. I was able to follow through until the last step where a deployment for a pod is defined with the service account internal-app. When I create the deployment and apply with kubectl, it injects the init container (vault-agent-init) and sidecar (vault-agent) correctly, but the init container keeps failing to authenticate. The logs from the container show:

==> Vault agent started! Log data will stream in below:                                                              
 ==> Vault agent configuration:                                                                                       
                      Cgo: disabled                                                                                   
                Log Level: info                                                                                       
                  Version: Vault v1.6.1                                                                               
              Version Sha: 6d2db3f033e02e70202bef9ec896360062b88b03                                                   
 2021-01-11T19:56:09.783Z [INFO]  sink.file: creating file sink                                                       
 2021-01-11T19:56:09.783Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----      
 2021-01-11T19:56:09.784Z [INFO]  template.server: starting template server                                           
 2021-01-11T19:56:09.784Z [INFO]  template.server: no templates found                                                 
 2021-01-11T19:56:09.784Z [INFO]  sink.server: starting sink server                                                   
 2021-01-11T19:56:09.784Z [INFO]  auth.handler: starting auth handler                                                 
 2021-01-11T19:56:09.784Z [INFO]  auth.handler: authenticating                                                        
 2021-01-11T19:56:09.785Z [ERROR] auth.handler: error authenticating: error="Put "http://vault.default.svc:8200/v1/auth/kubernetes/login": dial tcp 10.43.169.48:8200: connect: connection refused" backoff=1.495047283

I don’t know why the authentication is failing. I made sure that the kubernetes auth is enabled and internal-app policy and role are correct. What should I look into, to diagnose what’s failing here? I appreciate any help.

I was able to isolate the problem to vault-agent-init not being able to connect to vault.default.svc. That felt really weird, so I dug a little deeper. I used nc to check whether a tcp connection can be made in various pods.

# In vault-agent-init

$ nc -zv vault.default.svc 8200
# returns immediately with nothing printed to console. So, I wanted to see if I can connect to other services.

$ nc -zv pgsqldb.default.svc 5432
pgsqldb.default.svc (10.111.78.172:5432) open
# prints above and then returns. So, I can access to other services, but not to vault.default.svc

I thought I should try the connections from the vault-0 pod, which is connected to the service.

# In vault-0 pod.
$ nc -zv localhost 8200
localhost (127.0.0.1:8200) open
# This is a good sign. It can connect to port 8200 on loopback. I tried if it would connect over the pod's address. 

$ nc -zv $POD_IP 8200
# returns immediately with no output. This makes me think that vault is listening on loopback only, which does not get traffic from the service. 

Any ideas?

I’m following the same tutorial and getting the same error in the logs, but at the “Inject Secrets into the Pod” section after running kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets.yml)"
My updated orgchart is stuck displaying
NAME READY STATUS
orgchart-798cbc6c76-qn7n9 0/2 Init:0/1

There is a problem with the instructions in the tutorial, which causes the issue that I and @mb69 had. The helm installation needs to change from:

helm install vault hashicorp/vault --set "server.dev.enabled=true"

to

helm install vault hashicorp/vault --set "server.dev.enabled=true" --set 'server.extraArgs="-dev-listen-address=0.0.0.0:8200"'

It turns out that the dev server listens on the loopback (127.0.0.1) interface by default, which causes the port not being accessible from outside the container. For more information, see https://github.com/hashicorp/vault/blob/9459932e0694e59f5b47692a87de6bc6a95fb1c3/command/server.go#L232

1 Like