Kubernetes, vault-agent-init sidecar keeps failing with auth error

Vault
,
1

I have been following the tutorial for injecting secrets into pods in kubernetes. I was able to follow through until the last step where a deployment for a pod is defined with the service account internal-app. When I create the deployment and apply with kubectl, it injects the init container (vault-agent-init) and sidecar (vault-agent) correctly, but the init container keeps failing to authenticate. The logs from the container show:

==> Vault agent started! Log data will stream in below:                                                              
 ==> Vault agent configuration:                                                                                       
                      Cgo: disabled                                                                                   
                Log Level: info                                                                                       
                  Version: Vault v1.6.1                                                                               
              Version Sha: 6d2db3f033e02e70202bef9ec896360062b88b03                                                   
 2021-01-11T19:56:09.783Z [INFO]  sink.file: creating file sink                                                       
 2021-01-11T19:56:09.783Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----      
 2021-01-11T19:56:09.784Z [INFO]  template.server: starting template server                                           
 2021-01-11T19:56:09.784Z [INFO]  template.server: no templates found                                                 
 2021-01-11T19:56:09.784Z [INFO]  sink.server: starting sink server                                                   
 2021-01-11T19:56:09.784Z [INFO]  auth.handler: starting auth handler                                                 
 2021-01-11T19:56:09.784Z [INFO]  auth.handler: authenticating                                                        
 2021-01-11T19:56:09.785Z [ERROR] auth.handler: error authenticating: error="Put "http://vault.default.svc:8200/v1/auth/kubernetes/login": dial tcp 10.43.169.48:8200: connect: connection refused" backoff=1.495047283

I don’t know why the authentication is failing. I made sure that the kubernetes auth is enabled and internal-app policy and role are correct. What should I look into, to diagnose what’s failing here? I appreciate any help.

2

I was able to isolate the problem to vault-agent-init not being able to connect to vault.default.svc. That felt really weird, so I dug a little deeper. I used nc to check whether a tcp connection can be made in various pods.

# In vault-agent-init

$ nc -zv vault.default.svc 8200
# returns immediately with nothing printed to console. So, I wanted to see if I can connect to other services.

$ nc -zv pgsqldb.default.svc 5432
pgsqldb.default.svc (10.111.78.172:5432) open
# prints above and then returns. So, I can access to other services, but not to vault.default.svc

I thought I should try the connections from the vault-0 pod, which is connected to the service.

# In vault-0 pod.
$ nc -zv localhost 8200
localhost (127.0.0.1:8200) open
# This is a good sign. It can connect to port 8200 on loopback. I tried if it would connect over the pod's address. 

$ nc -zv $POD_IP 8200
# returns immediately with no output. This makes me think that vault is listening on loopback only, which does not get traffic from the service.

Any ideas?

3

I’m following the same tutorial and getting the same error in the logs, but at the “Inject Secrets into the Pod” section after running kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets.yml)"
My updated orgchart is stuck displaying
NAME READY STATUS
orgchart-798cbc6c76-qn7n9 0/2 Init:0/1

1 Like
4

There is a problem with the instructions in the tutorial, which causes the issue that I and @mb69 had. The helm installation needs to change from:

helm install vault hashicorp/vault --set "server.dev.enabled=true"

to

helm install vault hashicorp/vault --set "server.dev.enabled=true" --set 'server.extraArgs="-dev-listen-address=0.0.0.0:8200"'

It turns out that the dev server listens on the loopback (127.0.0.1) interface by default, which causes the port not being accessible from outside the container. For more information, see https://github.com/hashicorp/vault/blob/9459932e0694e59f5b47692a87de6bc6a95fb1c3/command/server.go#L232

2 Likes
5

As of vault-helm 0.9.1, the dev listen address is automatically set to [::]:8200 when server.dev.enabled=true, so the extraArgs here should no longer be necessary.