Hello 
Here I am again asking for help… I am trying to setup a HA Vault cluster on Kubernetes using the Raft integrated storage with TLS enabled.
Unfortunately I am having the following issues:
2023-04-12T14:47:49.494Z [ERROR] core: failed to get raft challenge: leader_addr=https://vault-4.vault-internal:8200 error="error during raft bootstrap init call: Put \"https://vault-4.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
2023-04-12T14:47:49.494Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2023-04-12T14:47:49.506Z [INFO] http: TLS handshake error from 10.112.0.21:48016: remote error: tls: bad certificate
2023-04-12T14:47:49.604Z [INFO] http: TLS handshake error from 10.112.1.22:49590: remote error: tls: bad certificate
- CSR config:
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.vault-internal.vault.svc
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.vault-internal
DNS.2 = *.vault-internal.vault
DNS.3 = *.vault-internal.vault.svc
DNS.4 = *.vault-internal.vault.svc.cluster.local
IP.1 = 127.0.0.1
- Kubernetes CertificateSigningRequest:
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: vault-internal.svc
spec:
groups:
- system:authenticated
signerName: kubernetes.io/kubelet-serving
expirationSeconds: 8640000
request: 'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRFVEQ0NBamdDQVFBd1NERVZNQk1HQTFVRUNnd01jM2x6ZEdWdE9tNXZaR1Z6TVM4d0xRWURWUVFERENaegplWE4wWlcwNmJtOWtaVG9xTG5aaGRXeDBMV2x1ZEdWeWJtRnNMblpoZFd4MExuTjJZekNDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlJzSjBPb0pGdGhmV0R3TmN5alhXQ3VpYUVBTXpKeVViMWQKaVJoQTlaZWVVbTE2UFBHZGphOFFBdStzQTdTSys0SlQ2TlpyeWlBWFVwZk1LL014ZnpkYkQxMGFpN1dPbm5hKwovbmp1d2ZmSDA1MHIvM3F3Nm5IN2xCa1cyNVFGSHlGYXBJL2ZiRXJpV2ZRWVhIdnpoSEJ4WjBBc3U0eWY2UDFsCm5uN3FjVTBHYXdMWWlaMjdPb0tUSjg3QzJYcC9OMzNLZDBzcWlTdVBhT2w5eTBVUVo3d2dBZjA5UG1rSFlMRUcKTUpLYWFwek5KOUxxc1UwdTFqTWcyMXM0bkRieVY1U3VqTGxaN2F1RGc3UnVJb2JWSEpsOG4xc1V5N1FxYm1OaQpTUlUrQWREcGtWL1F3L3dVMFRmS1RxK0Q5ZFJiUFBDOUgyQWV6TDlIK2ptTU1EblZDWjBDQXdFQUFhQ0J3akNCCnZ3WUpLb1pJaHZjTkFRa09NWUd4TUlHdU1Ba0dBMVVkRXdRQ01BQXdDd1lEVlIwUEJBUURBZ1hnTUJNR0ExVWQKSlFRTU1Bb0dDQ3NHQVFVRkJ3TUJNSDhHQTFVZEVRUjRNSGFDRUNvdWRtRjFiSFF0YVc1MFpYSnVZV3lDRmlvdQpkbUYxYkhRdGFXNTBaWEp1WVd3dWRtRjFiSFNDR2lvdWRtRjFiSFF0YVc1MFpYSnVZV3d1ZG1GMWJIUXVjM1pqCmdpZ3FMblpoZFd4MExXbHVkR1Z5Ym1Gc0xuWmhkV3gwTG5OMll5NWpiSFZ6ZEdWeUxteHZZMkZzaHdSL0FBQUIKTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFDK0J3ZmZsbHBrcmtXZS9MYWRnRHkrSHRXQjc1eithK2RLS3JEcgpxZlZYZVdDVWwrL2RxbWhpS2RHdUdQaXZ2RmFBWnJ5Q0lGeWhRSFk3ZHY4cGVaaDNwS2FFOWlvTTFyTlhQczNwCjFNOTVOZGJyRlpNLzJrc0pBSzZkRStLRmxJRGZaR0hRYW9rdmcxNHpoVDMxQVdwaTFHU2FUU3lhM01GcGxIbFMKUHdaWjRrRXlpcys2YjNBcmdONmdIRGVnRitwUjNKRWRxcGUxVlZya25RMGl2RElBRHFxYkhPZGF3WDNsRG15WgpQaEJiVU00eG5wZXU5SU96UUhydEZnSHRtUHJEMi9aM2N6eU1rdGpyZkxRbUorWmduNTJuL2xVWkFNQlpYOTZICnpOaFNqYTRMVWV1ZDh2NHB2cXdwRTJBYTA1NjM1M2ZpQjRseVBNUmNzNlRiNGdObAotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0='
usages:
- digital signature
- key encipherment
- server auth
- Certificate issued by Kubernetes:
❯ cat vault/tls/vault.crt | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
6c:6e:5d:62:35:d3:12:ff:14:dc:35:2e:43:a7:3c:a9
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=75586c7f-07dc-4f50-be37-5fb649deed10
Validity
Not Before: Apr 12 14:42:28 2023 GMT
Not After : Jul 21 14:44:28 2023 GMT
Subject: O=system:nodes, CN=system:node:*.vault-internal.vault.svc
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:d4:6c:27:43:a8:24:5b:61:7d:60:f0:35:cc:a3:
5d:60:ae:89:a1:00:33:32:72:51:bd:5d:89:18:40:
f5:97:9e:52:6d:7a:3c:f1:9d:8d:af:10:02:ef:ac:
03:b4:8a:fb:82:53:e8:d6:6b:ca:20:17:52:97:cc:
2b:f3:31:7f:37:5b:0f:5d:1a:8b:b5:8e:9e:76:be:
fe:78:ee:c1:f7:c7:d3:9d:2b:ff:7a:b0:ea:71:fb:
94:19:16:db:94:05:1f:21:5a:a4:8f:df:6c:4a:e2:
59:f4:18:5c:7b:f3:84:70:71:67:40:2c:bb:8c:9f:
e8:fd:65:9e:7e:ea:71:4d:06:6b:02:d8:89:9d:bb:
3a:82:93:27:ce:c2:d9:7a:7f:37:7d:ca:77:4b:2a:
89:2b:8f:68:e9:7d:cb:45:10:67:bc:20:01:fd:3d:
3e:69:07:60:b1:06:30:92:9a:6a:9c:cd:27:d2:ea:
b1:4d:2e:d6:33:20:db:5b:38:9c:36:f2:57:94:ae:
8c:b9:59:ed:ab:83:83:b4:6e:22:86:d5:1c:99:7c:
9f:5b:14:cb:b4:2a:6e:63:62:49:15:3e:01:d0:e9:
91:5f:d0:c3:fc:14:d1:37:ca:4e:af:83:f5:d4:5b:
3c:f0:bd:1f:60:1e:cc:bf:47:fa:39:8c:30:39:d5:
09:9d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
keyid:8D:7E:D0:C8:B0:C5:EC:98:E7:64:D6:DA:65:A2:F8:C0:2B:ED:D2:BD
X509v3 Subject Alternative Name:
DNS:*.vault-internal, DNS:*.vault-internal.vault, DNS:*.vault-internal.vault.svc, DNS:*.vault-internal.vault.svc.cluster.local, IP Address:127.0.0.1
Signature Algorithm: sha256WithRSAEncryption
54:55:30:10:b5:f9:7b:20:50:8c:6c:48:9c:c9:c2:40:a9:34:
40:42:4f:52:43:bb:3f:5a:3e:a7:37:85:1f:40:6f:4c:3f:f6:
9b:6d:23:4c:87:aa:8c:e9:57:4f:9b:df:06:44:96:28:ba:e4:
25:b4:23:e3:98:f7:2b:d5:69:22:ae:d9:36:ee:65:8c:41:ab:
cf:e5:91:d1:1d:73:fc:08:a7:0f:bf:8e:4c:d0:a1:48:bf:87:
c6:b1:26:c3:5b:c3:a4:f9:5f:76:e7:f2:11:68:19:70:77:33:
19:96:cd:98:d9:10:d1:92:a7:16:8a:a0:b1:b5:f1:b4:01:d5:
2c:c1:91:0c:8b:01:5e:64:fe:14:7c:12:c3:fe:7f:6a:58:f1:
b9:f5:f9:04:64:06:c7:0b:04:3c:3a:90:22:05:74:93:51:5f:
7e:5a:b7:52:74:0e:42:49:81:ba:fc:26:a9:3c:a1:63:48:ac:
0c:dd:6f:36:c1:76:ba:a2:73:05:cc:35:ba:90:51:23:9f:7d:
2b:82:2b:4a:07:cc:ed:77:ef:db:3a:0d:df:42:13:ec:f8:ef:
47:34:9b:e2:48:16:63:95:af:93:09:07:6d:59:1c:30:4f:88:
cf:6d:26:36:86:3b:67:e8:92:47:98:fc:6d:44:df:6d:64:2f:
41:ce:f2:24:3b:f6:0c:00:44:12:fa:53:e4:66:78:2f:e8:7e:
96:58:45:e7:89:d7:fa:0e:b5:a2:6f:4c:81:75:46:c8:71:56:
a2:e9:fe:63:ed:00:f0:5d:b9:32:87:3a:1a:47:5d:2d:cd:79:
e4:04:28:f7:9c:b1:5c:65:c4:ae:29:bf:7c:bf:d8:9a:a9:cb:
cb:bc:d8:20:40:8a:aa:d9:2b:29:41:94:fc:66:5e:58:26:34:
a1:76:89:f3:6e:51:d7:f8:b1:08:54:42:d6:2c:c3:7e:d4:05:
12:c7:09:8a:69:9e:0e:1d:40:66:63:1c:4c:ae:3f:24:b8:35:
fe:40:4d:a5:5c:da
- Kubernetes CA:
❯ cat vault/tls/vault.ca | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
c2:57:fd:a4:5c:22:e1:12:55:8c:a0:4f:cf:fd:ab:04
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=75586c7f-07dc-4f50-be37-5fb649deed10
Validity
Not Before: Apr 12 12:32:13 2023 GMT
Not After : Apr 4 13:32:13 2053 GMT
Subject: CN=75586c7f-07dc-4f50-be37-5fb649deed10
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (3072 bit)
Modulus:
00:b5:c8:17:c8:a3:9e:99:57:a0:54:09:9e:5d:9d:
bc:9b:ef:27:10:8f:91:4a:e2:b0:26:3d:d3:c1:ef:
c6:42:c0:c3:72:41:bd:b6:35:17:69:b9:40:49:ae:
7b:6f:20:e4:2b:0b:69:89:fe:76:dc:6b:3d:ba:fa:
fc:d6:06:fc:32:27:aa:86:5b:41:9b:3e:42:d2:68:
c7:c1:ac:78:9b:7a:d3:c8:9a:bd:28:ee:0b:30:ac:
e5:7b:96:c5:ed:c0:59:6c:a4:67:db:87:e2:19:3b:
0c:da:53:dd:de:83:4f:a5:11:fc:da:26:ce:9f:07:
44:10:13:3a:30:c8:f3:26:56:0e:4f:3c:29:1d:91:
52:c8:7d:42:ba:18:be:17:4e:c0:e0:e7:9e:62:f0:
d4:48:f2:6a:25:cf:1f:56:68:98:d8:dc:f0:f5:71:
c3:f1:26:e0:de:a5:38:22:2d:84:92:10:55:18:09:
76:ea:b0:bb:c0:06:41:42:3e:38:4d:1a:64:be:13:
b2:d9:fa:c1:44:68:49:31:8b:db:1b:b3:29:5d:e5:
17:f8:7e:17:cd:4e:8a:54:48:0e:d7:e4:8e:3e:e1:
c0:41:ca:df:c4:40:5d:d8:0c:3e:00:29:06:d5:1a:
ec:6d:a3:f0:d2:e4:1b:38:0b:4b:20:3a:a9:14:6d:
dd:8f:03:8c:d5:d6:eb:37:d5:24:05:b2:45:b4:40:
b8:11:d4:5b:c0:5f:47:14:35:da:59:22:b0:3b:ae:
22:9e:59:63:48:0b:fb:1d:89:9c:b1:80:e6:ee:88:
37:a5:85:cd:d5:ee:8e:ed:9c:13:a8:25:cc:33:b3:
8c:4a:d5:69:0f:90:d5:67:3e:14:aa:70:8e:3b:24:
b7:b0:83:e9:9c:b1:91:ea:d3:52:12:a2:4d:35:e5:
0d:1e:09:16:5b:ba:96:3d:c1:9c:2e:43:76:1b:fb:
d2:29:47:ca:93:ac:41:47:07:83:21:5d:78:24:fd:
e4:24:37:5d:17:f9:a7:e5:c7:d5
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
8D:7E:D0:C8:B0:C5:EC:98:E7:64:D6:DA:65:A2:F8:C0:2B:ED:D2:BD
Signature Algorithm: sha256WithRSAEncryption
75:86:30:c3:78:96:33:f7:a6:11:c3:b2:eb:22:3a:70:2c:bd:
c5:42:71:3d:07:75:13:72:db:3a:ba:04:76:6c:da:9a:1d:85:
d9:dc:f2:4e:ef:29:74:b8:fe:b1:35:09:08:30:2e:59:36:72:
63:cc:3e:4c:14:29:4c:44:a6:12:3a:dd:25:f4:58:cb:9e:ad:
dd:d7:10:22:7d:4e:b5:5c:6f:e8:36:cc:16:9d:49:33:ed:45:
24:57:b0:79:8a:7e:44:c2:1c:9d:53:d6:df:ad:1a:e2:e0:2f:
7a:b9:a2:18:c2:e5:1f:41:f7:fe:03:e2:86:5b:6a:17:4f:f1:
96:05:c0:5b:82:a0:7f:85:aa:0f:be:19:94:e9:7b:58:06:39:
30:de:d5:42:b5:fe:c3:aa:7c:15:5a:87:33:bc:06:b8:61:fd:
f2:38:3a:37:ea:12:eb:81:f1:dd:1f:dc:dc:fe:76:e2:61:90:
e0:07:20:45:f1:80:15:42:2b:30:09:57:29:bd:95:52:05:dd:
7b:6d:c9:18:68:e9:ba:b5:a5:73:02:75:64:70:ab:4c:b0:82:
5e:3f:2c:7a:17:fc:42:8c:4b:19:eb:e8:97:de:a9:d8:2e:9f:
93:73:73:19:82:2b:70:0b:b0:a5:98:31:61:20:9e:c7:94:38:
db:f2:1b:0f:46:dd:26:1a:53:ab:c6:c5:ce:a6:e3:96:82:b3:
33:f6:c8:35:2b:79:d8:70:52:66:d5:cc:ed:ea:1e:83:e0:38:
84:a4:1b:7a:e2:85:03:58:71:d5:5a:ca:e3:07:73:65:5c:4f:
6a:01:f2:46:9e:39:95:63:8e:40:6c:51:9c:3d:68:49:56:dd:
cd:d4:55:10:ea:2b:03:8e:b8:98:7d:9b:a4:aa:ee:da:a5:06:
d4:dd:6e:27:10:ee:01:ef:67:8f:a1:a1:e8:7e:e8:8a:8c:b6:
f4:b2:92:66:f8:6b:75:cf:5e:69:04:a1:80:41:e5:af:35:3c:
ac:fc:06:fa:96:55
I have created a vault-tls secret holding the vault.ca, vault.key and vault.crt files.
❯ kubectl create secret generic vault-tls -n vault --from-file=vault.key=vault/tls/vault.key --from-file=vault.crt=vault/tls/vault.crt --from-file=vault.ca=vault/tls/vault.ca
Below the server config:
apiVersion: v1
kind: ConfigMap
metadata:
name: server-config
namespace: vault
labels:
app.kubernetes.io/name: vault
app.kubernetes.io/instance: vault
data:
extraconfig-from-values.hcl: |-
disable_mlock = true
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/tls/vault.crt"
tls_key_file = "/vault/tls/vault.key"
tls_client_ca_file = "/vault/tls/vault.ca"
}
storage "raft" {
path = "/vault/data"
retry_join {
leader_api_addr = "https://vault-0.vault-internal:8200"
}
retry_join {
leader_api_addr = "https://vault-1.vault-internal:8200"
}
retry_join {
leader_api_addr = "https://vault-2.vault-internal:8200"
}
retry_join {
leader_api_addr = "https://vault-3.vault-internal:8200"
}
retry_join {
leader_api_addr = "https://vault-4.vault-internal:8200"
}
}
service_registration "kubernetes" {}
And finally the statefulset:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: vault
namespace: vault
labels:
app.kubernetes.io/name: vault
app.kubernetes.io/instance: vault
spec:
serviceName: vault-internal
podManagementPolicy: Parallel
replicas: 5
updateStrategy:
type: OnDelete
selector:
matchLabels:
app.kubernetes.io/name: vault
app.kubernetes.io/instance: vault
component: server
template:
metadata:
labels:
app.kubernetes.io/name: vault
app.kubernetes.io/instance: vault
component: server
spec:
hostNetwork: false
serviceAccountName: vault
terminationGracePeriodSeconds: 10
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
fsGroup: 1000
volumes:
- name: home
emptyDir: {}
- name: config
configMap:
name: server-config
- name: tls
secret:
secretName: vault-tls
- name: data
emptyDir: {}
containers:
- name: vault
image: hashicorp/vault:1.13.1
imagePullPolicy: IfNotPresent
command:
- "/bin/sh"
- "-ec"
args:
- |
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl
env:
- name: HOME
value: "/home/vault"
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: VAULT_K8S_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: VAULT_K8S_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SKIP_CHOWN
value: "true"
- name: SKIP_SETCAP
value: "true"
- name: VAULT_ADDR
value: "https://127.0.0.1:8200"
- name: VAULT_API_ADDR
value: "https://$(POD_IP):8200"
- name: VAULT_CLUSTER_ADDR
value: "https://$(HOSTNAME).vault-internal:8201"
- name: VAULT_LOG_LEVEL
value: debug
- name: VAULT_LOG_FORMAT
value: standard
- name: VAULT_RAFT_NODE_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: VAULT_TLSCERT
value: /vault/tls/vault.crt
- name: VAULT_TLSKEY
value: /vault/tls/vault.key
- name: VAULT_CACERT
value: /vault/tls/vault.ca
ports:
- name: http
containerPort: 8200
- name: https-internal
containerPort: 8201
- name: http-rep
containerPort: 8202
lifecycle:
# Vault container doesn't receive SIGTERM from Kubernetes
# and after the grace period ends, Kube sends SIGKILL. This
# causes issues with graceful shutdowns such as deregistering itself
# from Consul (zombie services).
preStop:
exec:
command: [
"/bin/sh", "-c",
# Adding a sleep here to give the pod eviction a
# chance to propagate, so requests will not be made
# to this pod while it's terminating
"sleep 5 && kill -SIGTERM $(pidof vault)",
]
livenessProbe:
httpGet:
path: "/v1/sys/health?standbyok=true"
port: 8200
scheme: HTTPS
failureThreshold: 2
initialDelaySeconds: 60
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
readinessProbe:
httpGet:
path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
port: 8200
scheme: HTTPS
failureThreshold: 2
initialDelaySeconds: 180
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
securityContext:
allowPrivilegeEscalation: false
volumeMounts:
- name: config
mountPath: /vault/config
- name: data
mountPath: /vault/data
- name: home
mountPath: /home/vault
- name: tls
mountPath: /vault/tls
Any help would be greatly appreciated
Best,