TLS handshake error: Bad certificate

RakeshRaj97 · August 30, 2021, 2:04am

Can i get a clue on what I’m missing? The detailed error is posted as a Github issue

TLS handshake error

opened 01:53AM - 30 Aug 21 UTC

I installed vault in HA mode using self-signed certificate. Process I followed, … ## Step 1: Create key & certificate using Kubernetes CA **Define environment variables** ``` SERVICE=vault-server-tls NAMESPACE=vault SECRET_NAME=vault-server-tls TMPDIR=/tmp ``` **Create a key for Kubernetes to sign** `openssl genrsa -out ${TMPDIR}/vault.key 2048` **Create a Certificate Signing Request (CSR)** ``` cat <<EOF >${TMPDIR}/csr.conf [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1 = ${SERVICE} DNS.2 = ${SERVICE}.${NAMESPACE} DNS.3 = ${SERVICE}.${NAMESPACE}.svc DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local DNS.5 = vault-0.vault-internal DNS.6 = vault-1.vault-internal DNS.7 = vault-2.vault-internal DNS.8 = vault-0.vault-internal.svc DNS.9 = vault-1.vault-internal.svc DNS.10 = vault-2.vault-internal.svc DNS.11 = vault-0.vault-internal.svc.cluster.local DNS.12 = vault-1.vault-internal.svc.cluster.local DNS.13 = vault-2.vault-internal.svc.cluster.local DNS.14 = vault-0 DNS.15 = vault-1 DNS.16 = vault-2 DNS.17 = vault-agent-injector-svc DNS.18 = vault-agent-injector-svc.${NAMESPACE} DNS.19 = vault-agent-injector-svc.${NAMESPACE}.svc DNS.20 = vault-agent-injector-svc.${NAMESPACE}.svc.cluster.local IP.1 = 127.0.0.1 EOF ``` **Create a CSR** `openssl req -new -key ${TMPDIR}/vault.key -subj "/CN=${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server.csr -config ${TMPDIR}/csr.conf` **Create the certificate** ``` export CSR_NAME=vault-csr cat <<EOF >${TMPDIR}/csr.yaml apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: ${CSR_NAME} spec: groups: - system:authenticated request: $(cat ${TMPDIR}/server.csr | base64 | tr -d '\n') usages: - digital signature - key encipherment - server auth EOF ``` **Send the CSR to Kubernetes** `kubectl create -f ${TMPDIR}/csr.yaml` **Approve the CSR in Kubernetes** `kubectl certificate approve ${CSR_NAME}` ## Step 2: Sort key, cert, and Kubernetes CA into Kubernetes secrets store **Retrieve the certificate** `serverCert=$(kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}')` **Write the certificate out to a file** `echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/vault.crt` **Create the namespace** `kubectl create namespace ${NAMESPACE}` **Store the key, cert, and Kubernetes CA into Kubernetes secrets** ``` kubectl create secret generic ${SECRET_NAME} \ --namespace ${NAMESPACE} \ --from-file=vault.key=${TMPDIR}/vault.key \ --from-file=vault.crt=${TMPDIR}/vault.crt \ --from-file=vault.ca=${TMPDIR}/vault.ca ``` ## Step 3: Install using Helm override-values.yaml ``` global: enabled: true tlsDisable: false server: extraEnvironmentVars: VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca VAULT_TLSCERT: /vault/userconfig/vault-server-tls/vault.crt VAULT_TLSKEY: /vault/userconfig/vault-server-tls/vault.key extraVolumes: - type: secret name: vault-server-tls ha: enabled: true replicas: 3 raft: enabled: true setNodeId: false config: | ui = true listener "tcp" { address = "0.0.0.0:8200" cluster_address = ""0.0.0.0:8201" tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" tls_key_file = "/vault/userconfig/vault-server-tls/vault.key" tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca" # tls_require_and_verify_client_cert = 1 } storage "raft" { path = "/vault/data" retry_join { leader_api_addr = "https://vault-0.vault-internal:8200" leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca" leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key" } retry_join { leader_api_addr = "https://vault-1.vault-internal:8200" leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca" leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key" } retry_join { leader_api_addr = "https://vault-2.vault-internal:8200" leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca" leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key" } autopilot { cleanup_dead_servers = "true" last_contact_threshold = "200ms" last_contact_failure_threshold = "10m" max_trailing_logs = 250000 min_quorum = 3 server_stabilization_time = "10s" } } service_registration "kubernetes" {} ``` Step 4: Initialised and unsealed vault ``` kubectl get pods --selector='app.kubernetes.io/name=vault' NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 1m49s vault-1 1/1 Running 0 1m49s vault-2 1/1 Running 0 1m49s vault-agent-injector-58446c77f4-7n6jp 1/1 Running 0 1m49s ``` Step 5: Create dummy secrets and configure K8s authentication ``` vault secrets enable -path=secret kv-v2 vault kv put secret/devwebapp/config username='giraffe' password='salsa' vault auth enable kubernetes vault write auth/kubernetes/config \ token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt vault policy write devwebapp - <<EOF path "secret/data/devwebapp/config" { capabilities = ["read"] } EOF vault write auth/kubernetes/role/devweb-app \ bound_service_account_names=internal-app \ bound_service_account_namespaces=default \ policies=devwebapp \ ttl=24h ``` Step 6: Deploy web application `kubectl create sa internal-app -n default` ``` apiVersion: v1 kind: Pod metadata: name: devwebapp namespace: default labels: app: devwebapp annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-status: "update" #vault.hashicorp.com/tls-skip-verify: "true" vault.hashicorp.com/role: "devweb-app-1" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config" vault.hashicorp.com/tls-secret: "vault-server-tls" vault.hashicorp.com/ca-cert: "/vault/tls/vault.ca" vault.hashicorp.com/client-cert: "/vault/tls/vault.crt" vault.hashicorp.com/client-key: "/vault/tls/vault.key" spec: serviceAccountName: internal-app containers: - name: devwebapp image: jweissig/app:0.0.1 ``` Logs: `k logs devwebapp -c vault-agent-init` ``` ==> Vault agent started! Log data will stream in below: ==> Vault agent configuration: Cgo: disabled Log Level: info Version: Vault v1.8.1 Version Sha: 4b0264f28defc05454c31277cfa6ff63695a458d 2021-08-30T01:38:17.530Z [INFO] sink.file: creating file sink 2021-08-30T01:38:17.530Z [INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r----- 2021-08-30T01:38:17.531Z [INFO] sink.server: starting sink server 2021-08-30T01:38:17.531Z [INFO] template.server: starting template server 2021-08-30T01:38:17.531Z [INFO] auth.handler: starting auth handler 2021-08-30T01:38:17.531Z [INFO] auth.handler: authenticating 2021-08-30T01:38:17.532Z [INFO] (runner) creating new runner (dry: false, once: false) 2021-08-30T01:38:17.534Z [INFO] (runner) creating watcher 2021-08-30T01:39:17.532Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s 2021-08-30T01:39:18.533Z [INFO] auth.handler: authenticating 2021-08-30T01:40:18.534Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.64s 2021-08-30T01:40:20.178Z [INFO] auth.handler: authenticating 2021-08-30T01:41:20.181Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=3.09s 2021-08-30T01:41:23.277Z [INFO] auth.handler: authenticating 2021-08-30T01:42:23.277Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=4.66s 2021-08-30T01:42:27.944Z [INFO] auth.handler: authenticating 2021-08-30T01:43:27.945Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=8.98s 2021-08-30T01:43:36.932Z [INFO] auth.handler: authenticating 2021-08-30T01:44:36.933Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=17.12s 2021-08-30T01:44:54.064Z [INFO] auth.handler: authenticating 2021-08-30T01:45:54.066Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=30.22s 2021-08-30T01:46:24.290Z [INFO] auth.handler: authenticating 2021-08-30T01:47:24.294Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=59.39s 2021-08-30T01:48:23.686Z [INFO] auth.handler: authenticating 2021-08-30T01:49:23.688Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1m51.4s ``` `k logs vault-0` ``` 2021-08-27T06:21:10.953Z [INFO] http: TLS handshake error from 10.244.15.1:59366: remote error: tls: bad certificate 2021-08-27T06:21:12.192Z [INFO] http: TLS handshake error from 10.244.15.1:59372: remote error: tls: bad certificate 2021-08-27T06:21:18.538Z [INFO] http: TLS handshake error from 10.244.15.1:59390: remote error: tls: bad certificate 2021-08-27T06:21:45.250Z [INFO] http: TLS handshake error from 10.244.15.1:59454: remote error: tls: bad certificate 2021-08-27T06:22:12.932Z [INFO] http: TLS handshake error from 10.244.15.1:47366: remote error: tls: bad certificate 2021-08-27T06:22:24.547Z [INFO] http: TLS handshake error from 10.244.15.1:59580: remote error: tls: bad certificate 2021-08-27T06:22:29.662Z [INFO] http: TLS handshake error from 10.244.15.1:59592: remote error: tls: bad certificate 2021-08-27T06:22:32.423Z [INFO] http: TLS handshake error from 10.244.15.1:47502: remote error: tls: bad certificate 2021-08-27T06:23:08.416Z [INFO] http: TLS handshake error from 10.244.15.1:60062: remote error: tls: bad certificate 2021-08-27T06:23:21.253Z [INFO] http: TLS handshake error from 10.244.15.1:60096: remote error: tls: bad certificate 2021-08-27T06:23:54.774Z [INFO] http: TLS handshake error from 10.244.15.1:60176: remote error: tls: bad certificate 2021-08-27T06:24:10.829Z [INFO] http: TLS handshake error from 10.244.15.1:60244: remote error: tls: bad certificate 2021-08-27T06:24:21.824Z [INFO] http: TLS handshake error from 10.244.15.1:60272: remote error: tls: bad certificate 2021-08-27T06:24:44.709Z [INFO] http: TLS handshake error from 10.244.15.1:60338: remote error: tls: bad certificate 2021-08-27T06:24:55.447Z [INFO] http: TLS handshake error from 10.244.15.1:60366: remote error: tls: bad certificate 2021-08-27T06:25:28.973Z [INFO] http: TLS handshake error from 10.244.15.1:60510: remote error: tls: bad certificate 2021-08-27T06:26:06.380Z [INFO] http: TLS handshake error from 10.244.15.1:60626: remote error: tls: bad certificate 2021-08-27T06:26:17.813Z [INFO] http: TLS handshake error from 10.244.15.1:60662: remote error: tls: bad certificate 2021-08-27T06:26:50.514Z [INFO] http: TLS handshake error from 10.244.15.1:60738: remote error: tls: bad certificate 2021-08-27T06:27:21.144Z [INFO] http: TLS handshake error from 10.244.15.1:60846: remote error: tls: bad certificate 2021-08-27T06:27:23.455Z [INFO] http: TLS handshake error from 10.244.15.1:60856: remote error: tls: bad certificate 2021-08-27T06:27:31.615Z [INFO] http: TLS handshake error from 10.244.15.1:49694: remote error: tls: bad certificate 2021-08-27T06:28:07.956Z [INFO] http: TLS handshake error from 10.244.15.1:49992: remote error: tls: bad certificate 2021-08-27T06:28:17.455Z [INFO] http: TLS handshake error from 10.244.15.1:50054: remote error: tls: bad certificate 2021-08-27T06:28:28.550Z [INFO] http: TLS handshake error from 10.244.15.1:32806: remote error: tls: bad certificate 2021-08-27T06:28:29.763Z [INFO] http: TLS handshake error from 10.244.15.1:32810: remote error: tls: bad certificate 2021-08-27T06:28:30.766Z [INFO] http: TLS handshake error from 10.244.15.1:50140: remote error: tls: bad certificate 2021-08-27T06:29:26.511Z [INFO] http: TLS handshake error from 10.244.15.1:32978: remote error: tls: bad certificate 2021-08-27T06:29:43.549Z [INFO] http: TLS handshake error from 10.244.15.1:33022: remote error: tls: bad certificate 2021-08-27T06:29:47.176Z [INFO] http: TLS handshake error from 10.244.15.1:33028: remote error: tls: bad certificate 2021-08-27T06:31:40.219Z [INFO] http: TLS handshake error from 10.244.15.1:33392: remote error: tls: bad certificate 2021-08-27T06:31:47.798Z [INFO] http: TLS handshake error from 10.244.15.1:33418: remote error: tls: bad certificate 2021-08-27T06:33:05.083Z [INFO] http: TLS handshake error from 10.244.15.1:33654: remote error: tls: bad certificate 2021-08-27T06:33:07.151Z [INFO] http: TLS handshake error from 10.244.15.1:33658: remote error: tls: bad certificate 2021-08-27T06:33:11.361Z [INFO] http: TLS handshake error from 10.244.15.1:33670: remote error: tls: bad certificate 2021-08-27T06:33:28.173Z [INFO] http: TLS handshake error from 10.244.15.1:52314: remote error: tls: bad certificate 2021-08-27T06:33:52.094Z [INFO] http: TLS handshake error from 10.244.15.1:33766: remote error: tls: bad certificate ``` Resources created ``` pod/devwebapp 0/2 Init:0/1 0 6m45s pod/vault-0 1/1 Running 0 2d20h pod/vault-1 1/1 Running 0 2d20h pod/vault-2 1/1 Running 0 2d20h pod/vault-agent-injector-58446c77f4-7n6jp 1/1 Running 0 2d20h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/vault ClusterIP 10.111.201.114 <none> 8200/TCP,8201/TCP 2d20h service/vault-active ClusterIP 10.100.17.194 <none> 8200/TCP,8201/TCP 2d20h service/vault-agent-injector-svc ClusterIP 10.103.55.211 <none> 443/TCP 2d20h service/vault-internal ClusterIP None <none> 8200/TCP,8201/TCP 2d20h service/vault-standby ClusterIP 10.105.249.28 <none> 8200/TCP,8201/TCP 2d20h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/vault-agent-injector 1/1 1 1 2d20h NAME DESIRED CURRENT READY AGE replicaset.apps/vault-agent-injector-58446c77f4 1 1 1 2d20h NAME READY AGE statefulset.apps/vault 3/3 2d20h ``` Is there anything I'm missing?

tirelibirefe · December 1, 2022, 6:53pm

Hello,
were you able to fix the problem?

Thanks & Regards