I’ve setup two machines running vault v1.15.5, vault-vc-0 and vault-vc-1. They’re setup to use raft as storage, and to use HTTPS for everything. They run as docker containers on flatcar v3815. I can’t get vault-vc-1 to properly talk to vault-vc-0, and wondering what i am doing wrong.
I provision both nodes at the same time. When they are both up, I go to vault-vc-0 and perform the vault initialization via the web ui. I then go to vault-vc-1 where it asks for keys to unseal, which I provide. When the last key is entered the unseal button doesn’t do anything.
The logs for vault-vc-0 says:
[ERROR] storage.raft: failed to heartbeat to: peer=vault-vc-1.company.lan:8200 backoff time=2.5s error="remote error: tls: no application protocol"
[ERROR] storage.raft: failed to appendEntries to: peer="{Nonvoter raft_node_vault-vc-1 vault-vc-1.company.lan:8200}" error="remote error: tls: no application protocol"
The logs for vault-vc-1 says:
[INFO] core: security barrier not initialized
[INFO] http: TLS handshake error from <vc-0-IP>:<randport> tls: client requested unsupported application protocols ([raft_storage_v1])
[INFO] http: TLS handshake error from <vc-0-IP>:<randport> tls: client requested unsupported application protocols ([raft_storage_v1])
[INFO] http: TLS handshake error from <vc-0-IP>:<randport> tls: client requested unsupported application protocols ([raft_storage_v1])
...
[INFO] core: security barrier not initialized
containers are started like this:
docker run \
--restart unless-stopped \
--network host \
--name vault \
-p 8200:8200 \
-p 8201:8201 \
-v "/vault/config.json:/vault/config.json" \
-v "/vault/certs/:/vault/certs/" \
docker.company.net:5000/hashicorp/vault:1.15.5 server -config /vault/config.json
I generate my certs with the following script:
openssl req \
-new \
-newkey rsa:4096 \
-days 1825 \
-nodes \
-x509 \
-subj "/C=NO/ST=Oslo/L=Vault/O=company CA" \
-keyout "CA_key.pem" \
-out "CA_cert.pem"
for l in $(ls vault* --color=never -d); do
cat <<EOF > $l.cnf
[req]
default_bits = 4096
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName = NO
localityName = $l
organizationName = company CA
commonName = $l
[req_ext]
subjectAltName = @alt_names
[v3_req]
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = $l
DNS.2 = $(cut -d'.' -f1,2 <<< $l)
DNS.3 = $(cut -d'.' -f1 <<< $l)
DNS.4 = vault.company.lan
EOF
openssl req \
-sha256 \
-nodes \
-newkey rsa:4096 \
-keyout $l/key.pem \
-out $l.csr \
-config $l.cnf
openssl x509 \
-req \
-days 398 \
-in $l.csr \
-CA CA_cert.pem \
-CAkey CA_key.pem \
-CAcreateserial \
-out $l/cert.pem \
-extensions req_ext \
-extfile $l.cnf
cat $l/cert.pem CA_cert.pem > $l/fullchain.pem
done
config for vault-vc-0 is this:
{
"api_addr": "https://vault-vc-0.company.lan:8201",
"cluster_addr": "https://vault-vc-0.company.lan:8200",
"disable_mlock": true,
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"cluster_address": "0.0.0.0:8201",
"tls_cert_file": "/vault/certs/vault-vc-0.company.lan/cert.pem",
"tls_client_ca_file": "/vault/certs/CA_cert.pem",
"tls_key_file": "/vault/certs/vault-vc-0.company.lan/key.pem"
}
},
"storage": {
"raft": {
"node_id": "raft_node_vault-vc-0",
"path": "/vault",
"retry_join": null
}
},
"ui": true
}
config for vault-vc-1 is this:
{
"api_addr": "https://vault-vc-1.company.lan:8201",
"cluster_addr": "https://vault-vc-1.company.lan:8200",
"disable_mlock": true,
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"cluster_address": "0.0.0.0:8201",
"tls_cert_file": "/vault/certs/vault-vc-1.company.lan/cert.pem",
"tls_client_ca_file": "/vault/certs/CA_cert.pem",
"tls_key_file": "/vault/certs/vault-vc-1.company.lan/key.pem"
}
},
"storage": {
"raft": {
"node_id": "raft_node_vault-vc-1",
"path": "/vault",
"retry_join": {
"leader_api_addr": "https://vault-vc-0.company.lan:8200",
"leader_ca_cert": [REDACTED], # the contents of '/vault/certs/CA_cert.pem'
"leader_tls_servername": "vault-vc-0.company.lan"
}
}
},
"ui": true
}
