"could not start clustered storage: HeartbeatTimeout is too low" on Initialization

Vault
1

Server config:

[QA] ottvault1:~ # cat /etc/vault/config.json 
{
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8443",
      "tls_disable": false,
      "tls_client_ca_file": "/etc/cbn/pki/CA2014/CA2014-cacert.pem",
      "tls_cert_file": "/etc/cbn/pki/CA2014/ottvault1.test.example.com-cert.pem",
      "tls_key_file": "/etc/cbn/pki/CA2014/ottvault1.test.example.com-privkey.pem"
    }
  },
  "storage": {
    "raft": {
      "path": "/data01/vault_data",
      "node_id": "apd_qa_vault_node_1",
      "performance_multiplier": "0",
      "retry_join": [
        {
          "leader_api_addr": "https://ottvault1.test.example.com:8443"
        }
      ]
    }
  },
  "default_lease_ttl": "720h",
  "disable_mlock": true,
  "ui": true,
  "api_addr": "https://ottvault1.test.example.com:8443",
  "cluster_addr": "https://ottvault1.test.example.com:8444",
  "cluster_name": "apd_qa_vault_cluster"
}

Server start command:
/usr/bin/vault server -config=/etc/vault/config.json

2022-11-04T09:38:56.700-0400 [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2022-11-04T09:38:58.701-0400 [INFO]  core: security barrier not initialized
2022-11-04T09:38:58.711-0400 [INFO]  core: attempting to join possible raft leader node: leader_addr=https://ottvault1.test.example.com:8443
2022-11-04T09:38:58.724-0400 [ERROR] core: failed to get raft challenge: leader_addr=https://ottvault1.test.example.com:8443
  error=
  | error during raft bootstrap init call: Error making API request.
  | 
  | URL: PUT https://ottvault1.test.example.com:8443/v1/sys/storage/raft/bootstrap/challenge
  | Code: 503. Errors:
  | 
  | * Vault is sealed

Error upon trying to initialize:
[QA] ottvault1:~ # export VAULT_ADDR='https://ottvault1.test.example.com:8443

[QA] ottvault1:~ # vault operator init 
Error initializing: Error making API request.

URL: PUT https://ottvault1.test.example.com:8443/v1/sys/init
Code: 400. Errors:

* could not start clustered storage: HeartbeatTimeout is too low

I’m simply trying to start a single node cluster for now for testing and experimentation purposes.

I’ve followed this documentation: Deploy Vault | Vault | HashiCorp Developer

Thanks in advance!

2 
[QA] ottvault1:/etc/vault # cat /etc/vault/config.json
{
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8443",
      "tls_disable": true
    }
  },
  "storage": {
    "raft": {
      "path": "/data01/vault_data",
      "node_id": "apd_qa_vault_node_1"
    }
  },
  "ui": true,
  "api_addr": "https://ottvault1.test.example.com:8443",
  "cluster_addr": "https://ottvault1.test.example.com:8444"
}

Reducing my config to the above allows it to start.

Currently trying to pinpoint which configuration element is causing the issue, but it’s not exactly obvious because none of them appear to be related to “HeartbeatTimeout” at the surface. Of course, this configuration is not ideal, and not production quality.

3

You tried to set performance_multiplier to zero - that’s the part that’s not valid.

It’s a multiplier, and multiplying by zero gives zero, which would be an invalid heartbeat timeout.

1 Like
4

According to documentation: Integrated Storage - Storage Backends - Configuration | Vault | HashiCorp Developer

performance_multiplier (integer: 0) - An integer multiplier used by servers to scale key Raft timing parameters. Tuning this affects the time it takes Vault to detect leader failures and to perform leader elections, at the expense of requiring more network and CPU resources for better performance. Omitting this value or setting it to 0 uses default timing described below. Lower values are used to tighten timing and increase sensitivity while higher values relax timings and reduce sensitivity.

By default, Vault will use a lower-performance timing that’s suitable for minimal Vault servers, currently equivalent to setting this to a value of 5 (this default may be changed in future versions of Vault, depending if the target minimum server profile changes). Setting this to a value of 1 will configure Raft to its highest-performance mode and is recommended for production Vault servers. The maximum allowed value is 10.

So would this be a bug, or incorrect documentation?

5

The documentation is wrong. The default multiplier is 5.

6

That was it. Appreciate the help! Cheers

Not sure where I can put a request to have the documentation tweaked.