Vault ha cluster - node vault-1 didn't join the cluster after init the node vault-0

Vault
,
1

Hello,

I deployed a vault ha cluster on EKS via helm and using raft storage and AWS KMS autounseal, I applied these values.yaml code

global:
  enabled: true
  tlsDisable: true

injector:
  enabled: true
  image:
    repository: "hashicorp/vault-k8s"
    tag: "latest"
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.14.0"
  dataStorage:
    enabled: true
    size: 5Gi
    mountPath: "/vault/data"
    storageClass: null
    accessMode: ReadWriteOnce
    annotations: {}
  dev:
    enabled: false
  standalone:
    enabled: false
  extraSecretEnvironmentVars:
  - envName: AWS_ACCESS_KEY_ID
    secretName: eks-creds
    secretKey: AWS_ACCESS_KEY_ID
  - envName: AWS_SECRET_ACCESS_KEY
    secretName: eks-creds
    secretKey: AWS_SECRET_ACCESS_KEY
  ha:
    enabled: true
    replicas: 2
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
          leader_api_addr = "http://vault-internal:8201"
          }
        }

        seal "awskms" {
          region     = "us-east-1"
          kms_key_id = "alias/vault-autounseal"
        }

after deploying, it created 3 pods, 2 for vault and 1 for injector

kubectl get all -n vault

NAME                                      READY   STATUS    RESTARTS   AGE
pod/vault-0                               0/1     Running   0          26s
pod/vault-1                               0/1     Running   0          26s
pod/vault-agent-injector-6fbd655b-lg5rq   1/1     Running   0          26s

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/vault                      ClusterIP   172.20.170.149   <none>        8200/TCP,8201/TCP   28s
service/vault-active               ClusterIP   172.20.75.73     <none>        8200/TCP,8201/TCP   28s
service/vault-agent-injector-svc   ClusterIP   172.20.189.194   <none>        443/TCP             28s
service/vault-internal             ClusterIP   None             <none>        8200/TCP,8201/TCP   28s
service/vault-standby              ClusterIP   172.20.92.169    <none>        8200/TCP,8201/TCP   28s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           28s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-6fbd655b   1         1         1       28s

NAME                     READY   AGE
statefulset.apps/vault   0/2     28s

after that, I went to vault-0 and executed

kubectl exec -ti vault-0 -n vault – vault operator init

for auto unseal. in this case, the pods vault-0 became ready but the vault-1 running but still unready.

kindly check the logs of the two pods:

kubectl logs vault-0 -n vault

2023-07-07T09:12:52.977Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:12:52.990Z [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=1
2023-07-07T09:12:53.063Z [INFO]  core: post-unseal setup starting
2023-07-07T09:12:53.073Z [INFO]  core: loaded wrapping token key
2023-07-07T09:12:53.073Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2023-07-07T09:12:53.073Z [INFO]  core: no mounts; adding default mount table
2023-07-07T09:12:53.084Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.14.0+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2023-07-07T09:12:53.084Z [INFO]  core: successfully mounted: type=system version="v1.14.0+builtin.vault" path=sys/ namespace="ID: root. Path: "
2023-07-07T09:12:53.085Z [INFO]  core: successfully mounted: type=identity version="v1.14.0+builtin.vault" path=identity/ namespace="ID: root. Path: "
2023-07-07T09:12:53.110Z [INFO]  core: successfully mounted: type=token version="v1.14.0+builtin.vault" path=token/ namespace="ID: root. Path: "
2023-07-07T09:12:53.114Z [INFO]  rollback: starting rollback manager
2023-07-07T09:12:53.114Z [INFO]  core: restoring leases
2023-07-07T09:12:53.115Z [INFO]  expiration: lease restore complete
2023-07-07T09:12:53.122Z [INFO]  identity: entities restored
2023-07-07T09:12:53.122Z [INFO]  identity: groups restored
2023-07-07T09:12:53.123Z [INFO]  core: usage gauge collection is disabled
2023-07-07T09:12:53.126Z [INFO]  core: Recorded vault version: vault version=1.14.0 upgrade time="2023-07-07 09:12:53.122954132 +0000 UTC" build date=2023-06-19T11:40:23Z
2023-07-07T09:12:53.163Z [WARN]  core: post-unseal upgrade seal keys failed: error="no recovery key found"
2023-07-07T09:12:53.665Z [INFO]  core: post-unseal setup complete
2023-07-07T09:12:53.712Z [INFO]  core: root token generated
2023-07-07T09:12:53.724Z [INFO]  core: pre-seal teardown starting
2023-07-07T09:12:53.724Z [INFO]  core: stopping raft active node
2023-07-07T09:12:53.724Z [INFO]  rollback: stopping rollback manager
2023-07-07T09:12:53.725Z [INFO]  core: pre-seal teardown complete
2023-07-07T09:12:53.725Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:12:53.752Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2023-07-07T09:12:53.752Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2023-07-07T09:12:53.752Z [INFO]  storage.raft: creating Raft: config="&raft.Config{ProtocolVersion:3, HeartbeatTimeout:15000000000, ElectionTimeout:15000000000, CommitTimeout:50000000, MaxAppendEntries:64, BatchApplyCh:true, ShutdownOnRemove:true, TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, LeaderLeaseTimeout:2500000000, LocalID:\"vault-0\", NotifyCh:(chan<- bool)(0xc0000b41c0), LogOutput:io.Writer(nil), LogLevel:\"DEBUG\", Logger:(*hclog.interceptLogger)(0xc000ff6c00), NoSnapshotRestoreOnStart:true, skipStartup:false}"
2023-07-07T09:12:53.753Z [INFO]  storage.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:vault-0 Address:vault-0.vault-internal:8201}]"
2023-07-07T09:12:53.753Z [INFO]  core: vault is unsealed
2023-07-07T09:12:53.753Z [INFO]  core: unsealed with stored key
2023-07-07T09:12:53.753Z [WARN]  core: attempted unseal with stored keys, but vault is already unsealed
2023-07-07T09:12:53.754Z [INFO]  storage.raft: entering follower state: follower="Node at vault-0.vault-internal:8201 [Follower]" leader-address= leader-id=
2023-07-07T09:12:53.754Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-07-07T09:12:53.754Z [INFO]  storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=3
2023-07-07T09:12:53.754Z [INFO]  core: entering standby mode
2023-07-07T09:12:53.759Z [INFO]  storage.raft: election won: term=3 tally=1
2023-07-07T09:12:53.759Z [INFO]  storage.raft: entering leader state: leader="Node at vault-0.vault-internal:8201 [Leader]"
2023-07-07T09:12:53.764Z [INFO]  core: acquired lock, enabling active operation
2023-07-07T09:12:53.780Z [INFO]  core: post-unseal setup starting
2023-07-07T09:12:53.785Z [INFO]  core: loaded wrapping token key
2023-07-07T09:12:53.785Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2023-07-07T09:12:53.787Z [INFO]  core: successfully mounted: type=system version="v1.14.0+builtin.vault" path=sys/ namespace="ID: root. Path: "
2023-07-07T09:12:53.787Z [INFO]  core: successfully mounted: type=identity version="v1.14.0+builtin.vault" path=identity/ namespace="ID: root. Path: "
2023-07-07T09:12:53.787Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.14.0+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2023-07-07T09:12:53.789Z [INFO]  core: successfully mounted: type=token version="v1.14.0+builtin.vault" path=token/ namespace="ID: root. Path: "
2023-07-07T09:12:53.789Z [INFO]  rollback: starting rollback manager
2023-07-07T09:12:53.790Z [INFO]  core: restoring leases
2023-07-07T09:12:53.790Z [INFO]  identity: entities restored
2023-07-07T09:12:53.790Z [INFO]  identity: groups restored
2023-07-07T09:12:53.790Z [INFO]  expiration: lease restore complete
2023-07-07T09:12:53.790Z [INFO]  core: starting raft active node
2023-07-07T09:12:53.790Z [INFO]  storage.raft: starting autopilot: config="&{false 0 10s 24h0m0s 1000 0 10s false redundancy_zone upgrade_version}" reconcile_interval=0s
2023-07-07T09:12:53.791Z [INFO]  core: usage gauge collection is disabled
2023-07-07T09:12:53.823Z [INFO]  core: post-unseal setup complete
2023-07-07T09:13:15.403Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-internal:8201 error="error during raft bootstrap init call: Put \"http://vault-internal:8201/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.51.210:8201: connect: connection refused"
2023-07-07T09:13:15.403Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2023-07-07T09:13:17.403Z [INFO]  core: returning from raft join as the node is initialized

kubectl logs vault-1 -n vault

2023-07-07T09:36:16.986Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:16.986Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:21.298Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:21.298Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:21.986Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:21.986Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:24.009Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-internal:8201 error="error during raft bootstrap init call: Put \"http://vault-internal:8201/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.51.210:8201: i/o timeout"
2023-07-07T09:36:24.009Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2023-07-07T09:36:26.010Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:26.013Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-internal:8201
2023-07-07T09:36:26.332Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:26.332Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:26.987Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:26.987Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:31.363Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:31.363Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:31.987Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:31.988Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:36.274Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:36.274Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:36.988Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:36.988Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:41.250Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:41.250Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:41.989Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:41.989Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:46.275Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:46.275Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:46.989Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:46.989Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:51.273Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:51.273Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:51.547Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:51.547Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:51.989Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:51.989Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:56.014Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-internal:8201 error="error during raft bootstrap init call: Put \"http://vault-internal:8201/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.23.147:8201: connect: connection refused"
2023-07-07T09:36:56.014Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2023-07-07T09:36:56.260Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:56.260Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:36:56.989Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:36:56.990Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:36:58.014Z [INFO]  core: security barrier not initialized
2023-07-07T09:36:58.017Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-internal:8201
2023-07-07T09:37:01.292Z [INFO]  core: security barrier not initialized
2023-07-07T09:37:01.293Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:37:01.990Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:37:01.990Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:37:06.261Z [INFO]  core: security barrier not initialized
2023-07-07T09:37:06.261Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:37:06.991Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:37:06.991Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:37:11.310Z [INFO]  core: security barrier not initialized
2023-07-07T09:37:11.310Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T09:37:11.991Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T09:37:11.991Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T09:37:13.018Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-internal:8201 error="error during raft bootstrap init call: Put \"http://vault-internal:8201/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.51.210:8201: i/o timeout"
2023-07-07T09:37:13.018Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"

kindly advise

2

Welcome to the forum - please reformat your message … right now it’s just too hard to read

6

Noted, kindly check and advise

7

A 2 node Raft cluster is unlikely to behave as you would wish. Raft is a consensus/quorum system, meaning that 3 nodes are required to tolerate 1 being down.

This does not look like an URL which the vault-1 pod would be able to use to connect to the vault-0 pod. Furthermore, the port is incorrect, the Vault API port is 8200.

The following is what I would consider a fairly standard Vault Raft in Kubernetes configuration:

      storage "raft" {
        path = "/vault/data"
        retry_join {
          leader_api_addr = "http://vault-0.vault-internal:8200"
        }
        retry_join {
          leader_api_addr = "http://vault-1.vault-internal:8200"
        }
        retry_join {
          leader_api_addr = "http://vault-2.vault-internal:8200"
        }
      }
8

I change the value.yaml

global:
  enabled: true
  tlsDisable: true

injector:
  enabled: true
  image:
    repository: "hashicorp/vault-k8s"
    tag: "latest"
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.14.0"
  dataStorage:
    enabled: true
    size: 5Gi
    mountPath: "/vault/data"
    storageClass: null
    accessMode: ReadWriteOnce
    annotations: {}
  dev:
    enabled: false
  standalone:
    enabled: false
  extraSecretEnvironmentVars:
  - envName: AWS_ACCESS_KEY_ID
    secretName: eks-creds
    secretKey: AWS_ACCESS_KEY_ID
  - envName: AWS_SECRET_ACCESS_KEY
    secretName: eks-creds
    secretKey: AWS_SECRET_ACCESS_KEY
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
          leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
          leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
          leader_api_addr = "http://vault-2.vault-internal:8200"
          }

        }

        seal "awskms" {
          region     = "us-east-1"
          kms_key_id = "alias/vault-autounseal"
        }

but still the same issues

9

If you had actually deployed the updated configuration, some error messages in the logs would have changed, at least.

10 
2023-07-07T12:47:03.294Z [INFO]  core: post-unseal setup starting
2023-07-07T12:47:03.302Z [INFO]  core: loaded wrapping token key
2023-07-07T12:47:03.302Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2023-07-07T12:47:03.303Z [INFO]  core: no mounts; adding default mount table
2023-07-07T12:47:03.311Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.14.0+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2023-07-07T12:47:03.312Z [INFO]  core: successfully mounted: type=system version="v1.14.0+builtin.vault" path=sys/ namespace="ID: root. Path: "
2023-07-07T12:47:03.312Z [INFO]  core: successfully mounted: type=identity version="v1.14.0+builtin.vault" path=identity/ namespace="ID: root. Path: "
2023-07-07T12:47:03.343Z [INFO]  core: successfully mounted: type=token version="v1.14.0+builtin.vault" path=token/ namespace="ID: root. Path: "
2023-07-07T12:47:03.347Z [INFO]  rollback: starting rollback manager
2023-07-07T12:47:03.348Z [INFO]  core: restoring leases
2023-07-07T12:47:03.349Z [INFO]  expiration: lease restore complete
2023-07-07T12:47:03.360Z [INFO]  identity: entities restored
2023-07-07T12:47:03.360Z [INFO]  identity: groups restored
2023-07-07T12:47:03.362Z [INFO]  core: usage gauge collection is disabled
2023-07-07T12:47:03.364Z [INFO]  core: Recorded vault version: vault version=1.14.0 upgrade time="2023-07-07 12:47:03.360715441 +0000 UTC" build date=2023-06-19T11:40:23Z
2023-07-07T12:47:03.397Z [WARN]  core: post-unseal upgrade seal keys failed: error="no recovery key found"
2023-07-07T12:47:03.978Z [INFO]  core: post-unseal setup complete
2023-07-07T12:47:04.031Z [INFO]  core: root token generated
2023-07-07T12:47:04.054Z [INFO]  core: pre-seal teardown starting
2023-07-07T12:47:04.054Z [INFO]  core: stopping raft active node
2023-07-07T12:47:04.055Z [INFO]  rollback: stopping rollback manager
2023-07-07T12:47:04.055Z [INFO]  core: pre-seal teardown complete
2023-07-07T12:47:04.055Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:47:04.083Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2023-07-07T12:47:04.083Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2023-07-07T12:47:04.083Z [INFO]  storage.raft: creating Raft: config="&raft.Config{ProtocolVersion:3, HeartbeatTimeout:15000000000, ElectionTimeout:15000000000, CommitTimeout:50000000, MaxAppendEntries:64, BatchApplyCh:true, ShutdownOnRemove:true, TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, LeaderLeaseTimeout:2500000000, LocalID:\"1ae816bf-803d-d5f7-191c-60a18257d5d3\", NotifyCh:(chan<- bool)(0xc000e7a770), LogOutput:io.Writer(nil), LogLevel:\"DEBUG\", Logger:(*hclog.interceptLogger)(0xc000f0c9f0), NoSnapshotRestoreOnStart:true, skipStartup:false}"
2023-07-07T12:47:04.085Z [INFO]  storage.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1ae816bf-803d-d5f7-191c-60a18257d5d3 Address:vault-0.vault-internal:8201}]"
2023-07-07T12:47:04.086Z [INFO]  core: vault is unsealed
2023-07-07T12:47:04.086Z [INFO]  core: unsealed with stored key
2023-07-07T12:47:04.086Z [WARN]  core: attempted unseal with stored keys, but vault is already unsealed
2023-07-07T12:47:04.086Z [INFO]  storage.raft: entering follower state: follower="Node at vault-0.vault-internal:8201 [Follower]" leader-address= leader-id=
2023-07-07T12:47:04.086Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-07-07T12:47:04.086Z [INFO]  storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=3
2023-07-07T12:47:04.086Z [INFO]  core: entering standby mode
2023-07-07T12:47:04.090Z [INFO]  storage.raft: election won: term=3 tally=1
2023-07-07T12:47:04.090Z [INFO]  storage.raft: entering leader state: leader="Node at vault-0.vault-internal:8201 [Leader]"
2023-07-07T12:47:04.101Z [INFO]  core: acquired lock, enabling active operation
2023-07-07T12:47:04.115Z [INFO]  core: post-unseal setup starting
2023-07-07T12:47:04.119Z [INFO]  core: loaded wrapping token key
2023-07-07T12:47:04.119Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2023-07-07T12:47:04.120Z [INFO]  core: successfully mounted: type=system version="v1.14.0+builtin.vault" path=sys/ namespace="ID: root. Path: "
2023-07-07T12:47:04.121Z [INFO]  core: successfully mounted: type=identity version="v1.14.0+builtin.vault" path=identity/ namespace="ID: root. Path: "
2023-07-07T12:47:04.121Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.14.0+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2023-07-07T12:47:04.122Z [INFO]  core: successfully mounted: type=token version="v1.14.0+builtin.vault" path=token/ namespace="ID: root. Path: "
2023-07-07T12:47:04.122Z [INFO]  core: restoring leases
2023-07-07T12:47:04.123Z [INFO]  rollback: starting rollback manager
2023-07-07T12:47:04.123Z [INFO]  expiration: lease restore complete
2023-07-07T12:47:04.123Z [INFO]  identity: entities restored
2023-07-07T12:47:04.123Z [INFO]  identity: groups restored
2023-07-07T12:47:04.123Z [INFO]  core: starting raft active node
2023-07-07T12:47:04.123Z [INFO]  storage.raft: starting autopilot: config="&{false 0 10s 24h0m0s 1000 0 10s false redundancy_zone upgrade_version}" reconcile_interval=0s
2023-07-07T12:47:04.124Z [INFO]  core: usage gauge collection is disabled
2023-07-07T12:47:04.157Z [INFO]  core: post-unseal setup complete
2023-07-07T12:47:12.716Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-2.vault-internal:8200 error="error during raft bootstrap init call: Put \"http://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.42.38:8200: i/o timeout"
2023-07-07T12:47:12.716Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-1.vault-internal:8200 error="error during raft bootstrap init call: Put \"http://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.23.147:8200: i/o timeout"
2023-07-07T12:47:12.716Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2023-07-07T12:47:14.716Z [INFO]  core: returning from raft join as the node is initialized

vault-0 logs

2023-07-07T12:48:54.849Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-1.vault-internal:8200
2023-07-07T12:48:54.849Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-0.vault-internal:8200
2023-07-07T12:48:54.852Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-1.vault-internal:8200
  error=
  | error during raft bootstrap init call: Error making API request.
  |
  | URL: PUT http://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
  | Code: 503. Errors:
  |
  | * Vault is sealed

2023-07-07T12:48:55.238Z [INFO]  core: security barrier not initialized
2023-07-07T12:48:55.238Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:48:55.719Z [INFO]  core: security barrier not initialized
2023-07-07T12:48:55.719Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:48:55.813Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:48:55.813Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:00.357Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:00.358Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:00.813Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:00.813Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:05.429Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:05.429Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:05.814Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:05.814Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:10.206Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:10.207Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:10.814Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:10.814Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:15.296Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:15.297Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:15.814Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:15.814Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:20.268Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:20.268Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:20.815Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:20.815Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:24.850Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-2.vault-internal:8200 error="error during raft bootstrap init call: Put \"http://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.42.38:8200: i/o timeout"
2023-07-07T12:49:24.852Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-0.vault-internal:8200 error="error during raft bootstrap init call: Put \"http://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.41.254:8200: i/o timeout"
2023-07-07T12:49:24.852Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2023-07-07T12:49:25.361Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:25.361Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:25.816Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:25.817Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:26.853Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:26.854Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-0.vault-internal:8200
2023-07-07T12:49:26.854Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-1.vault-internal:8200
2023-07-07T12:49:26.854Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-2.vault-internal:8200
2023-07-07T12:49:26.857Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-1.vault-internal:8200
  error=
  | error during raft bootstrap init call: Error making API request.
  |
  | URL: PUT http://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
  | Code: 503. Errors:
  |
  | * Vault is sealed

2023-07-07T12:49:30.376Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:30.376Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery

vault-1 logs

2023-07-07T12:49:25.938Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:25.938Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:26.534Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:26.534Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:30.936Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:30.936Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:31.535Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:31.535Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:32.064Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:32.064Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:35.942Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:35.942Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:36.535Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:36.535Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:40.921Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:40.921Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:41.536Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:41.536Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:45.944Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:45.944Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:46.536Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:46.536Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:50.920Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:50.920Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:51.537Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:51.537Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:49:53.539Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-1.vault-internal:8200 error="error during raft bootstrap init call: Put \"http://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.23.147:8200: i/o timeout"
2023-07-07T12:49:53.539Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-0.vault-internal:8200 error="error during raft bootstrap init call: Put \"http://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.0.41.254:8200: i/o timeout"
2023-07-07T12:49:53.539Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2023-07-07T12:49:55.540Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:55.542Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-0.vault-internal:8200
2023-07-07T12:49:55.542Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-1.vault-internal:8200
2023-07-07T12:49:55.542Z [INFO]  core: attempting to join possible raft leader node: leader_addr=http://vault-2.vault-internal:8200
2023-07-07T12:49:55.545Z [ERROR] core: failed to get raft challenge: leader_addr=http://vault-2.vault-internal:8200
  error=
  | error during raft bootstrap init call: Error making API request.
  |
  | URL: PUT http://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
  | Code: 503. Errors:
  |
  | * Vault is sealed

2023-07-07T12:49:55.946Z [INFO]  core: security barrier not initialized
2023-07-07T12:49:55.946Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:49:56.538Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:49:56.538Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:50:00.987Z [INFO]  core: security barrier not initialized
2023-07-07T12:50:00.987Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:50:01.539Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:50:01.539Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-07-07T12:50:05.978Z [INFO]  core: security barrier not initialized
2023-07-07T12:50:05.978Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2023-07-07T12:50:06.539Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-07-07T12:50:06.539Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"

vault-2 logs

11

It would seem there is some problem with networking between your Kubernetes pods, since the cluster is trying to form, but can’t seem to communicate between the nodes:

12

how can I solve this issue? can you help me please

13

I am sorry, but Kubernetes networking is such a complex issue, I find it is not really practical to provide such assistance via a forum, unless the person on the other end is able to debug independently and volunteer relevant information.

14

okay thx, I will check it and feedback you