URL: PUT https://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge Code: 503. Errors:

leonxu730 · March 18, 2023, 12:21pm

Deploy vault with 3 replicas in K3S cluster,always got POD failure with following error log. how can solve this

?

==> Vault server configuration:

         Api Address: https://10.42.3.21:8200
                 Cgo: disabled
     Cluster Address: https://vault-0.vault-internal:8201
          Go Version: go1.14.4
          Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
           Log Level: info
               Mlock: supported: true, enabled: false
       Recovery Mode: false
             Storage: raft (HA available)
             Version: Vault v1.5.0+ent

==> Vault server started! Log data will stream in below:

2023-03-17T01:00:03.423Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2023-03-17T01:00:03.466Z [INFO] core: raft retry join initiated
2023-03-17T01:00:03.466Z [INFO] core: security barrier not initialized
2023-03-17T01:00:03.466Z [INFO] core: security barrier not initialized
2023-03-17T01:00:03.466Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-0.vault-internal:8200
2023-03-17T01:00:03.555Z [INFO] core: join attempt failed: error="error during raft bootstrap init call: Error making API request.

URL: PUT https://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
Code: 503. Errors:

Vault is sealed"
2023-03-17T01:00:03.555Z [INFO] core: security barrier not initialized
2023-03-17T01:00:03.555Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-1.vault-internal:8200
2023-03-17T01:00:03.600Z [INFO] core: join attempt failed: error="error during raft bootstrap init call: Error making API request.

URL: PUT https://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
Code: 503. Errors:

Vault is sealed"
2023-03-17T01:00:03.600Z [INFO] core: security barrier not initialized
2023-03-17T01:00:03.600Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-2.vault-internal:8200
2023-03-17T01:00:03.641Z [INFO] core: join attempt failed: error="error during raft bootstrap init call: Error making API request.

URL: PUT https://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
Code: 503. Errors:

Vault is sealed"
2023-03-17T01:01:14.895Z [INFO] core: security barrier not initialized
2023-03-17T01:01:14.896Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-1.vault-internal:8200
2023-03-17T01:01:14.910Z [INFO] core: join attempt failed: error=“error during raft bootstrap init call: Put “https://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge”: dial tcp 10.42.1.184:8200: connect: connection refused”
2023-03-17T01:01:14.910Z [INFO] core: security barrier not initialized
2023-03-17T01:01:14.910Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-2.vault-internal:8200
2023-03-17T01:01:14.922Z [INFO] core: join attempt failed: error=“error during raft bootstrap init call: Put “https://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge”: dial tcp 10.42.4.40:8200: connect: connection refused”
2023-03-17T01:01:14.922Z [ERROR] core: failed to retry join raft cluster: retry=2s
2023-03-17T01:01:16.922Z [INFO] core: security barrier not initialized
2023-03-17T01:01:16.922Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-0.vault-internal:8200
2023-03-17T01:01:16.944Z [INFO] core: join attempt failed: error="error during raft bootstrap init call: Error making API request.

URL: PUT https://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
Code: 503. Errors:

Vault is sealed"
2023-03-17T01:01:16.945Z [INFO] core: security barrier not initialized
2023-03-17T01:01:16.945Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-1.vault-internal:8200
2023-03-17T01:01:16.956Z [INFO] core: join attempt failed: error=“error during raft bootstrap init call: Put “https://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge”: dial tcp 10.42.1.184:8200: connect: connection refused”
2023-03-17T01:01:16.956Z [INFO] core: security barrier not initialized
2023-03-17T01:01:16.956Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-2.vault-internal:8200
2023-03-17T01:01:16.967Z [INFO] core: join attempt failed: error=“error during raft bootstrap init call: Put “https://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge”: dial tcp 10.42.4.40:8200: connect: connection refused”
2023-03-17T01:01:16.967Z [ERROR] core: failed to retry join raft cluster: retry=2s
==> Vault shutdown triggered

============================================================================
Vault Helm config override-values.ymal

Vault Helm Chart Value Overrides

global:
enabled: true
tlsDisable: false

injector:
enabled: true

Use the Vault K8s Image GitHub - hashicorp/vault-k8s: First-class support for Vault and Kubernetes.

image:
repository: “hashicorp/vault-k8s”
tag: “latest”

resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 256Mi
cpu: 250m

server:

Use the Enterprise Image

image:
repository: “hashicorp/vault-enterprise”
tag: “1.5.0_ent”

These Resource Limits are in line with node requirements in the

Vault Reference Architecture for a Small Cluster

#resources:

requests:

memory: 4Gi

cpu: 2000m

limits:

memory: 4Gi

#  cpu: 2000m

For HA configuration and because we need to manually init the vault,

we need to define custom readiness/liveness Probe settings

readinessProbe:
enabled: true
path: “/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204”
livenessProbe:
enabled: true
path: “/v1/sys/health?standbyok=true”
initialDelaySeconds: 60

extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be

used to include variables required for auto-unseal.

extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/rootca/tls.crt

extraVolumes is a list of extra volumes to mount. These will be exposed

to Vault in the path `/vault/userconfig/<name>/`.

extraVolumes:
- type: secret
name: rootca
- type: secret
name: tls-server
#- type: secret
# name: kms-creds

This configures the Vault Statefulset to create a PVC for audit logs.

See Audit Devices | Vault | HashiCorp Developer to know more

auditStorage:
enabled: true

standalone:
enabled: false

Run Vault in “HA” mode.

ha:
enabled: true
replicas: 3
raft:
enabled: true
setNodeId: true

  config: |
    ui = true
    listener "tcp" {
      address = "[::]:8200"
      cluster_address = "[::]:8201"
      tls_cert_file = "/vault/userconfig/tls-server/tls.crt"
      tls_key_file = "/vault/userconfig/tls-server/tls.key"
      tls_client_ca_file = "/vault/userconfig/rootca/tls.crt"
    }

    storage "raft" {
      path = "/vault/data"
        retry_join {
        leader_api_addr = "https://vault-0.vault-internal:8200"
        leader_ca_cert_file = "vault/userconfig/rootca/tls.crt"
        leader_client_cert_file = "/vault/userconfig/tls-server/tls.crt"
        leader_client_key_file = "/vault/userconfig/tls-server/tls.key"
      }
      retry_join {
        leader_api_addr = "https://vault-1.vault-internal:8200"
        leader_ca_cert_file = "vault/userconfig/rootca/tls.crt"
        leader_client_cert_file = "/vault/userconfig/tls-server/tls.crt"
        leader_client_key_file = "/vault/userconfig/tls-server/tls.key"
      }
      retry_join {
        leader_api_addr = "https://vault-2.vault-internal:8200"
        leader_ca_cert_file = "/vault/userconfig/rootca/tls.crt"
        leader_client_cert_file = "/vault/userconfig/tls-server/tls.crt"
        leader_client_key_file = "/vault/userconfig/tls-server/tls.key"
      }

      autopilot {
        cleanup_dead_servers = "true"
        last_contact_threshold = "200ms"
        last_contact_failure_threshold = "10m"
        max_trailing_logs = 250000
        min_quorum = 3
        server_stabilization_time = "10s"
      }

    }

    service_registration "kubernetes" {}

Vault UI

ui:
enabled: true
serviceType: “LoadBalancer”
serviceNodePort: null
externalPort: 8200

For Added Security, edit the below

#loadBalancerSourceRanges:

- < Your IP RANGE Ex. 10.0.0.0/16 >

- < YOUR SINGLE IP Ex. 1.78.23.3/32 >

==============================================================================

ahamadi · October 19, 2023, 1:14am

Hi, i have the same problem. Is anybody have solution ?

Topic		Replies	Views
Vault operator raft join getting : [ERROR] core: failed to join raft cluster: error="failed to join any raft leader node" Vault k8s , connect , vault	3	10993	January 25, 2021
Vault K8s HA Raft Certificate Error Vault k8s , azure	2	771	January 21, 2021
Kubernetes Vault new cluster raft join failing Vault k8s , raft , vault	2	865	April 8, 2024
Trying to setup Vault HA mode with Raft Vault k8s , vault	4	1317	December 16, 2022
Vault raft: Preventing server addition that would require removal of too many servers and cause cluster instability Vault raft	7	2506	July 27, 2021

URL: PUT https://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge Code: 503. Errors:

Vault Helm Chart Value Overrides

Use the Vault K8s Image GitHub - hashicorp/vault-k8s: First-class support for Vault and Kubernetes.

Use the Enterprise Image

These Resource Limits are in line with node requirements in the

Vault Reference Architecture for a Small Cluster

requests:

memory: 4Gi

cpu: 2000m

limits:

memory: 4Gi

For HA configuration and because we need to manually init the vault,

we need to define custom readiness/liveness Probe settings

extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be

used to include variables required for auto-unseal.

extraVolumes is a list of extra volumes to mount. These will be exposed

to Vault in the path /vault/userconfig/<name>/.

This configures the Vault Statefulset to create a PVC for audit logs.

See Audit Devices | Vault | HashiCorp Developer to know more

Run Vault in “HA” mode.

Vault UI

For Added Security, edit the below

- < Your IP RANGE Ex. 10.0.0.0/16 >

- < YOUR SINGLE IP Ex. 1.78.23.3/32 >

Related topics

to Vault in the path `/vault/userconfig/<name>/`.