Bad certificate in vault-agent-injector

Hello guys !

I’m trying to have a functional Vault in Kubernetes using the most recent helm chart, with the Raft protocol and the TLS.

I believe I managed to get it about 90% completion, but there is something with the TLS that doesn’t work. So to generate the certificate, I used the jetstack/cert-manager (pretty common in Kubernetes), which create a k8s secret (vault-tls). In the helm chart, I refer the “secretName” property as vault-tls. I also specific the 3 TLS “files” in the config listener{} block and the retry_join{} blocks.

Something like this :
tls_cert_file = “/vault/userconfig/vault-tls/tls.crt”
tls_key_file = “/vault/userconfig/vault-tls/tls.key”
tls_client_ca_file = “/vault/userconfig/vault-tls/ca.crt”

The 3 container cluster itself seems to boot properly, the container keeps running and the logs seems ok (I see this message in the logs: core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1] but I don’t think it really impacts the cluster health).

In the agent-injector section of the yml, under the certs property I have something like
secretName: vault-tls
caBundle: --obfuscated base64 string–
certName: tls.crt
keyName: tls.key

However, when I check the logs of the agent-injector container, I can see a lot of “bad certificate” warning.

2020-06-28T17:50:02.572Z [INFO] handler: Starting handler…
Listening on “:8080”…
Updated certificate bundle received. Updating certs…
2020/06/28 17:50:11 http: TLS handshake error from 10.8.2.253:46456: remote error: tls: bad certificate
2020/06/28 17:50:11 http: TLS handshake error from 10.8.2.253:46458: remote error: tls: bad certificate
2020/06/28 17:50:16 http: TLS handshake error from 10.8.2.253:46470: remote error: tls: bad certificate

Seems I have 1 warning line spawning everytime I try to boot an application container with the annotations to fetch a secret. What exactly could cause this ? Is this a TLS issue between the container and the agent injector, or a tls issue between the agent injector and the cluster ?
When I describe the application container, the sidecar container does not show up, so I don’t think the agent injector do its job at all. I try adding the “vault.hashicorp.com/tls-skip-verify” = “true” annotation to the application container just for a test, and it still doesn’t work I have the same issue.

I believe my issue might be around the “caBundle” propery in the agent-injector section, I tried with the filename (ca.crt from the secret). I also tried with the base64, same error.

Any help would be really appreciated.
I provide my helm chart values.yml in attachment

values.yml.txt (15.7 KB)

Thanks for any advice !

So I managed to fix some TLS stuff

I added this in the server block
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt
VAULT_ADDR: “https://$(HOSTNAME).vault-v1-internal.vault:8200”

I also disabled the injector caBundle, certName and keyName property

I also realized my leader_ca_cert_file in the retry_join was missing the T in cerT because the actual documentation here is missing it too https://www.vaultproject.io/docs/configuration/storage/raft

So know I have a working cluster with TLS, and Pods can have their secrets injected by the vault agent for as long as I keep “vault.hashicorp.com/tls-skip-verify” = “true” in their annotations. If I remove it I face the bug 2020-06-29T13:03:18.369Z [ERROR] auth.handler: error authenticating: error=“Put https://vault-v1.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority” backoff=2.16560675.

I tried fixing it with the annotaton “vault.hashicorp.com/ca-cert” = “/run/secrets/kubernetes.io/serviceaccount/ca.crt” but it’s not working.

I am ensure at this point if the container should use the ca cert of the certificate generated with cert-manager or the one of the kubernetes services account. I feel it’s missing a bit of guidance here. I’ll try to inject the CA of the cert-manager certificate in the container to see if I can fix it, but if the secret “vault-tls” is not in the same namespace I am ensure how I will make this work. As far as I know cert-manager generate a secret in the same namespace where the certificate resource is created, and vault cannot operate with multiple certificate.

So that’s where I am now, I’ll keep you posted.
If you know have some guidance for me that would be nice !
I’ll post my new values.yml in attachment
values.yml.txt (16.5 KB)

Thanks
–jm

So yeah I did not managed to make this annotation work :
http://vault.hashicorp.com/ca-cert” = “/run/secrets/kubernetes.io/serviceaccount/ca.crt”

However if I copy my vault-tls secret (containing the cert-manager certificate artifacts) in the namespaces that I need it, like:

kubectl get secret vault-tls --namespace=${var.namespace} -ojson | jq ‘del(.metadata.namespace)’ | kubectl apply --namespace=default -f -

I can then use the following annotations to make this work

vault.hashicorp.com/tls-secret” = “vault-tls”
vault.hashicorp.com/ca-cert” = “/vault/tls/ca.crt”