Bad certificate in vault-agent-injector

Vault
1

Hello guys !

I’m trying to have a functional Vault in Kubernetes using the most recent helm chart, with the Raft protocol and the TLS.

I believe I managed to get it about 90% completion, but there is something with the TLS that doesn’t work. So to generate the certificate, I used the jetstack/cert-manager (pretty common in Kubernetes), which create a k8s secret (vault-tls). In the helm chart, I refer the “secretName” property as vault-tls. I also specific the 3 TLS “files” in the config listener{} block and the retry_join{} blocks.

Something like this :
tls_cert_file = “/vault/userconfig/vault-tls/tls.crt”
tls_key_file = “/vault/userconfig/vault-tls/tls.key”
tls_client_ca_file = “/vault/userconfig/vault-tls/ca.crt”

The 3 container cluster itself seems to boot properly, the container keeps running and the logs seems ok (I see this message in the logs: core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1] but I don’t think it really impacts the cluster health).

In the agent-injector section of the yml, under the certs property I have something like
secretName: vault-tls
caBundle: --obfuscated base64 string–
certName: tls.crt
keyName: tls.key

However, when I check the logs of the agent-injector container, I can see a lot of “bad certificate” warning.

2020-06-28T17:50:02.572Z [INFO] handler: Starting handler…
Listening on “:8080”…
Updated certificate bundle received. Updating certs…
2020/06/28 17:50:11 http: TLS handshake error from 10.8.2.253:46456: remote error: tls: bad certificate
2020/06/28 17:50:11 http: TLS handshake error from 10.8.2.253:46458: remote error: tls: bad certificate
2020/06/28 17:50:16 http: TLS handshake error from 10.8.2.253:46470: remote error: tls: bad certificate

Seems I have 1 warning line spawning everytime I try to boot an application container with the annotations to fetch a secret. What exactly could cause this ? Is this a TLS issue between the container and the agent injector, or a tls issue between the agent injector and the cluster ?
When I describe the application container, the sidecar container does not show up, so I don’t think the agent injector do its job at all. I try adding the “vault.hashicorp.com/tls-skip-verify” = “true” annotation to the application container just for a test, and it still doesn’t work I have the same issue.

I believe my issue might be around the “caBundle” propery in the agent-injector section, I tried with the filename (ca.crt from the secret). I also tried with the base64, same error.

Any help would be really appreciated.
I provide my helm chart values.yml in attachment

values.yml.txt (15.7 KB)

Thanks for any advice !

1 Like
2

So I managed to fix some TLS stuff

I added this in the server block
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt
VAULT_ADDR: “https://$(HOSTNAME).vault-v1-internal.vault:8200”

I also disabled the injector caBundle, certName and keyName property

I also realized my leader_ca_cert_file in the retry_join was missing the T in cerT because the actual documentation here is missing it too https://www.vaultproject.io/docs/configuration/storage/raft

So know I have a working cluster with TLS, and Pods can have their secrets injected by the vault agent for as long as I keep “vault.hashicorp.com/tls-skip-verify” = “true” in their annotations. If I remove it I face the bug 2020-06-29T13:03:18.369Z [ERROR] auth.handler: error authenticating: error=“Put https://vault-v1.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority” backoff=2.16560675.

I tried fixing it with the annotaton “vault.hashicorp.com/ca-cert” = “/run/secrets/kubernetes.io/serviceaccount/ca.crt” but it’s not working.

I am ensure at this point if the container should use the ca cert of the certificate generated with cert-manager or the one of the kubernetes services account. I feel it’s missing a bit of guidance here. I’ll try to inject the CA of the cert-manager certificate in the container to see if I can fix it, but if the secret “vault-tls” is not in the same namespace I am ensure how I will make this work. As far as I know cert-manager generate a secret in the same namespace where the certificate resource is created, and vault cannot operate with multiple certificate.

So that’s where I am now, I’ll keep you posted.
If you know have some guidance for me that would be nice !
I’ll post my new values.yml in attachment
values.yml.txt (16.5 KB)

Thanks
–jm

1 Like
3

So yeah I did not managed to make this annotation work :
http://vault.hashicorp.com/ca-cert” = “/run/secrets/kubernetes.io/serviceaccount/ca.crt”

However if I copy my vault-tls secret (containing the cert-manager certificate artifacts) in the namespaces that I need it, like:

kubectl get secret vault-tls --namespace=${var.namespace} -ojson | jq ‘del(.metadata.namespace)’ | kubectl apply --namespace=default -f -

I can then use the following annotations to make this work

“vault.hashicorp.com/tls-secret” = “vault-tls”
“vault.hashicorp.com/ca-cert” = “/vault/tls/ca.crt”

2 Likes
4

Hey @JnMik
I’m getting the same error as you are (or were).
We’re using AWS EKS with an external Vault server. I have deployed vault-agent-injector with defaults so it uses it’s self-generated certs. As is with you - if I use the annotation
vault.hashicorp.com/tls-skip-verify: “true”
the secrets are injected into the application pod by the vault agent. If I remove it, I get the same error as yours.

I tried using the annotation
vault.hashicorp.com/ca-cert: “/run/secrets/kubernetes.io/serviceaccount/ca.crt”
but it’s not working for me too.

For the secret I’m not sure if it has to be a “tls” secret or can it be “generic” secret?
I created a generic secret (in the same namespace) from the certificate at location “/run/secrets/kubernetes.io/serviceaccount/ca.crt” (the same one I used to configure kubernetes auth method in Vault) and tried a k8s deployment using annotations:
“vault.hashicorp.com/tls-secret”: “vault-tls”
“vault.hashicorp.com/ca-cert”: = “/vault/tls/ca.crt”

But this doesn’t work and I get an error like:
2020-07-20T11:19:27.779Z [ERROR] auth.handler: error authenticating: error=“Put https://<my_vault_addr>/v1/auth/kubernetes/login: x509: certificate signed by unknown authority” backoff=1.354056995
2020-07-20T11:19:29.133Z [INFO] auth.handler: authenticating

I can’t create a kubernetes tls secret as I don’t know what to use as the key (as it is required to use a key along with cert while creating a tls secret).

Would you happen to know what I could be doing wrong here? or any idea’s for me to try?

Thanks in advance!

5

Ok, so I seem to have fixed it. I was just using the wrong cert - I mistook it for the cert on the kube api server rather than what my vault server was using. The official documentation around this is not too clear.

Though I’m still having to maintain the secret (can be “generic” too) with certificate in all the namespaces I want to deploy my apps to. Wish there was an easier way to do this.
Also, if it may be helpful for anyone reading this - I’m also having to create the service account (which will be used by pods to authenticate to vault via token) in different namespaces.

Happy to take suggestions/improvements.