[SOLVED] Advertised cluster address does not match cluster_addr setting

Vault
1

I have a single vault node that was just upgraded from filesystem storage to raft storage.

During the migration I set the cluster_addr to 127.0.0.1:8201 as in the migrate docs. Afterwards I updated the cluster_addr to be an address that new nodes would be able to access.

Based on the startup logs, this seems to have worked as expected:

==> Vault server configuration:

             Api Address: https://ccs-docker01.coldstorage.com:8200
                     Cgo: disabled
         Cluster Address: https://ccs-docker01.coldstorage.com:8201
              Go Version: go1.16.6
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: raft (HA available)
                 Version: Vault v1.8.1
             Version Sha: 4b0264f28defc05454c31277cfa6ff63695a458d

However, vault operator raft list-peers tells a different story:

vault operator raft list-peers
Node               Address           State     Voter
----               -------           -----     -----
docker-vault-01    127.0.0.1:8201    leader    true

vault operator raft autopilot state
Healthy:                      true
Failure Tolerance:            0
Leader:                       docker-vault-01
Voters:
   docker-vault-01
Servers:
   docker-vault-01
      Name:            docker-vault-01
      Address:         127.0.0.1:8201
      Status:          leader
      Node Status:     alive
      Healthy:         true
      Last Contact:    0s
      Last Term:       4
      Last Index:      4347

Since the advertised address is localhost, joining a new node doesn’t work. How do I get this address to be properly advertised?

Config:

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/srv/certificates/wildcard_cert/ccs-wildcard.pem"
  tls_key_file = "/srv/certificates/wildcard_cert/ccs-wildcard.key"
  tls_disable_client_certs = true
}

ui = true
plugin_directory = "/srv/vault/plugins"
disable_mlock = true
api_addr = "https://ccs-docker01.coldstorage.com:8200"
cluster_addr = "https://ccs-docker01.coldstorage.com:8201"

storage "raft" {
  path = "/vault/raft"
  node_id = "docker-vault-01"
}
2

I managed to bring this into line, but I was pretty heavy-handed about it.

I followed the manual recovery using peers.json guide to get this back in line.

I used the following peers.json file and then restarted vault:

[
  {
    "id":  "docker-vault-01",
    "address": "ccs-docker01.coldstorage.com:8201",
    "non_voter": false
  }
]

Once that was done, vault operator raft list-peers agreed with the startup logs.