Vault PKI and Cert-Manager Issuer authorization with Vault - TLS problem

Vault
1

Hello,

I have troubles with TLS between Vault and Cert-Manager. I try to set up Vault PKI and let Cert-Manager use it.
Sorry if I mix concepts and terms, I have little knowledge on this topic.

So, I have HA Vault setup with Raft backend. TLS between Raft nodes works well, all Vault nodes are unsealed.
I prepared a self-signed certificate from own CA using official Vault tutorial (shown below), imported to Kubernetes and configured Vault raft to use it.

Then, I configured PKI in Vault, generated issuer, certificate, role (from UI), kubernetes Auth Method, kubernetes service account and binding with Cert-Manager in Kubernetes. So, I end-up having another CA and root cert.

Then, I created a Cert-Manager Issuer to authenticate with Vault as described in Cert-Manager documentation:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    path: pki_test/sign/services
    server: https://vault.vault:8200
    caBundle: LS0tL......
    auth:
      kubernetes:
        role: issuer
        mountPath: /v1/auth/kubernetes
        serviceAccountRef:
          name: vault-issuer

The vault-issuer resource returns following error:

Error initializing issuer: while requesting a Vault token using the Kubernetes auth: error calling Vault server: Post "https://vault.vault:8200/v1/auth/kubernetes/login": tls: failed to verify certificate: x509: certificate signed by unknown authority

Log from the Cert-Manager pod:

controller.go:167] "re-queuing item due to error processing" err="while requesting a Vault token using the Kubernetes auth: error calling Vault server: Post \"https://vault.vault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" logger="cert-manager.issuers" key="default/vault-issuer"
I0529 08:37:40.495958       1 setup.go:130] vault-issuer: Failed to initialize Vault client: while requesting a Vault token using the Kubernetes auth: error calling Vault server: Post "https://vault.vault:8200/v1/auth/kubernetes/login": tls: failed to verify certificate: x509: certificate signed by unknown authority

And, finally, Vault log:

[INFO]  http: TLS handshake error from 10.42.3.91:54086: remote error: tls: bad certificate
[WARN]  auth.kubernetes.auth_kubernetes_1cf47131: Configured CA PEM data contains no valid certificates, TLS verification will fail

As I understand, the caBundle is wrong. I don’t know which certificate to use (the one for UI and Raft or from PKI?) and how to generate that caBundle to satisfy Vault and Cert-Manager.

Here is how I did it:

I use the same certifiate as for Raft, the one located in each Vault pod at /vault/userconfig/vault-ha-tls/vault.ca, below I briefly describe how I generated it (followed Vault tutorial):

$ openssl genrsa -out ${WORKDIR}/vault.key 2048

$ openssl req -new -key ${WORKDIR}/vault.key -out ${WORKDIR}/vault.csr -config ${WORKDIR}/vault-csr.conf

$ cat > ${WORKDIR}/csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: vault.svc
spec:
  signerName: kubernetes.io/kubelet-serving
  expirationSeconds: 8640000
  request: $(cat ${WORKDIR}/vault.csr|base64|tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

$ kubectl create -f ${WORKDIR}/csr.yaml

$ kubectl certificate approve vault.svc

# retrieve certificate:
$ kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt

$ kubectl config view \
--raw \
--minify \
--flatten \
-o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
| base64 -d > ${WORKDIR}/vault.ca

$ kubectl create secret generic vault-ha-tls \
-n $VAULT_K8S_NAMESPACE \
--from-file=vault.key=${WORKDIR}/vault.key \
--from-file=vault.crt=${WORKDIR}/vault.crt \
--from-file=vault.ca=${WORKDIR}/vault.ca

Here is vault-csr.conf file

[req]
default_bits = 4096
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.vault.svc.cluster.local
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.vault-internal
DNS.2 = *.vault-internal.vault.svc.cluster.local
DNS.3 = *.vault
IP.1 = 127.0.0.1

Here is how I use it for UI and Raft:

config: |
  ui = true
  listener "tcp" {
    tls_disable = 0
    address = "[::]:8200"
    cluster_address = "[::]:8201"
    tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
    tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
    tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
  }
  storage "raft" {
    path = "/vault/data"

    retry_join {
      leader_api_addr = "https://vault-0.vault-internal:8200"
      leader_ca_cert_file = "/vault/userconfig/vault-ha-tls/vault.ca"
      leader_client_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
      leader_client_key_file = "/vault/userconfig/vault-ha-tls/vault.key"
    }

Within Vault it works like charm. So I tried to generate a caBundle from vault.ca and use it for Cart-Manager’s Issuer:

$ base64 vault.ca > test.ca.b64

# remove all new line characters, this will result in one-line string
$ tr -d "\n" < test.ca.b64 > caBundle

Resulting string in kubernetes manifest looks like caBundle: LS0tLS1CRUdJTiBD.....S0tLS0t

As I understand, it doesn’t matter how Cert-Manager authenticates with Vault (via Token or Kubernetes auth etc.), it should connect to Vault using https protocol and this step fails. Therefore I didn’t provide details regarding establishing auth itself (roles, bindings etc.).

I can provide more details on request, this post is already very long.

So, what do I do wrong? Do I use right certificate?

Edit: maybe the certificate is generated wrong? (CA:FALSE in vault-csr.conf file)