Unable to join initialized node vault-0, despite having the correct CA

Vault
1

i build a minimal docker image to run vault (for learning), and i am successful in launching the vault server. I have created a 3 replica statefulset each of them running the below config. More information about my setup and config here as well.

SSL Certs were all generated properly and stored in kubernetes secrets. it is mounted in /etc/ssl/certs and cat of the respective secrets shows that it is in the proper format.

i have no problem accessing vault ui via the browser.

problem
What i’m experiencing problems with is when i try to manually connect vault-1 to the initialized node vault-0.

>>> kubectl  exec -it -n securities vault-1 -- bash
>>> vault-1:/vault vault operator raft join https://vault-0.vault.securities.svc.cluster.local:8200
Key       Value
---       -----
Joined    true

>>> vault-1:/vault# vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  true
Total Shares            5
Threshold               3
Unseal Progress         0/3
Unseal Nonce            n/a
Version                 1.19.4
Build Date              2025-05-02T23:54:28Z
Storage Type            raft
Removed From Cluster    false
HA Enabled              true

But as I apply the unseal keys, vault-1 gets the error message Error unsealing: context deadline exceeded when i enter the 3rd key, and vault logs have the output:

  err=
  | failed to send answer to raft leader node: Error making API request.
  | 
  | URL: PUT https://vault-0.vault.securities.svc.cluster.local:8200/v1/sys/storage/raft/bootstrap/answer
  | Code: 500. Errors:
  | 
  | * Preventing server addition that would require removal of too many servers and cause cluster instability

prior to manually joining, i also noticed vault-1’s logs having:

2025-05-08T04:11:42.568Z [INFO]  core: security barrier not initialized
2025-05-08T04:11:42.568Z [INFO]  core: attempting to join possible raft leader node: leader_addr=https://vault-0.vault.securities.svc.cluster.local:8200
2025-05-08T04:11:42.572Z [ERROR] core: failed to retry join raft cluster: retry=2s err="waiting for unseal keys to be supplied"

config.hcl

ui = true
cluster_name = "vault-internal"
# HTTP listener with TLS enforced
listener "tcp" {
  address     = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/tls/vault.crt"
  tls_key_file  = "/vault/tls/vault.key"
  tls_client_ca_file = "/vault/tls/vault.ca"
  tls_require_and_verify_client_cert = false
  tls_disable_client_certs = false
}

# Vault runs in HA mode
storage "raft" {
  path    = "/vault/data"
  node_id = "HOSTNAME_PLACEHOLDER"  # initContainer will replace this with hostname (vault-0 | vault-1 | vault-2)
  # Recommended: enable raft TLS
  retry_join {
    leader_api_addr = "https://vault-0.vault.securities.svc.cluster.local:8200"
    leader_ca_cert_file = "/vault/tls/vault.ca"
    leader_client_cert_file = "/vault/tls/vault.crt"
    leader_client_key_file = "/vault/tls/vault.key"
  }
}

service_registration "kubernetes" {
  namespace = "securities"
}

# Advertise addresses to other nodes
api_addr      = "https://HOSTNAME_PLACEHOLDER.vault.securities.svc.cluster.local:8200"
cluster_addr  = "https://HOSTNAME_PLACEHOLDER.vault.securities.svc.cluster.local:8201"

# Logging
log_level = "info"

# Enable memory locking (if supported by OS)
disable_mlock = true

Not sure what I can do to get through this and join vault-1 and vault-2 with vault-0?