How to make a cluster accept Vault's provided auto-signed certificate

Vault
1

Hi all,

For learning purposes (I’m one in charge of bringing Vault into my company), I’m setting Vault Docker environments by hand, with increasing completeness, following the official Hashicorp tutorials and docs. The last successful step was a cluster with manual join and unseal, and without TLS, adapting the procedure in the Vault with Integrated Storage Deployment Guide.

Now I need help for the next step: use TLS, with the auto-signed certificate provided by Vault installer. I got stuck at making Vault accept that certificate.

I’m clueless about managing certificate authorities, so I’m looking (at this time) for a way to make Vault use the certificate, and be OK with it being auto-signed.

No-TLS working cluster config file:

disable_mlock = true
api_addr      = "http://vault_X:8200"
cluster_addr  = "https://vault_X:8201"

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault_X"
}

# HTTP listener
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable = "true"
}

Environment variable:
VAULT_ADDR="http://vault_ad_X:8200"

After initialization, join, unseal and login, I got all members present with vault operator members, and confirmed the cluster is working by managing a key in all nodes:

# Standby node 2
vault secrets enable -path=spam kv
vault kv put spam/ham knights=ni

# Standby node 3
vault kv get spam/ham
===== Data =====
Key        Value
---        -----
knights    ni

# Active node 1
vault secrets disable spam

# Standby node 3
vault kv get spam/ham
Code: 403. Errors:
* preflight capability check returned 403, please ensure client's policies grant access to path "spam/ham/"

To start in TLS, I changed the schema in api_addr to https and added the provided files in the listener directives of the config file, but disabling client authentication (not sure if that makes sense):

disable_mlock = true
api_addr      = "https://vault_X:8200"
cluster_addr  = "https://vault_X:8201"

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault_X"
}

# HTTPS listener
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
  tls_disable_client_certs = "true"
}

Then set properties and permissions of provided files as instructed in the same guide:

chown root:root /opt/vault/tls/tls.crt
chown root:vault /opt/vault/tls/tls.key
chmod 0644 /opt/vault/tls/tls.crt
chmod 0640 /opt/vault/tls/tls.key

Environment variable:
VAULT_ADDR="https://vault_ad_X:8200"

Then, I got stuck at initializing the active server:

vault operator init &
[INFO]  http: TLS handshake error from <Node IP>:36584: remote error: tls: bad certificate
Error initializing: Put "https://vault_1:8200/v1/sys/init": x509: certificate is not valid for any names, but wanted to match vault_1

Is there a way to make Vault skip that name match and be happy with the certificate? Or to add names to this certificate?

Thanks a lot,
Emerson

2

Additional info on my procedure

  • Build and access containers

    sudo docker network create --subnet 172.16.1.0/24 vault
for no in {1..3} ; do
  sudo docker run --cap-add=IPC_LOCK --entrypoint="/bin/bash" -dit \
    --network vault --name vault_$no rockylinux:9
done
sudo docker exec -it vault_X /bin/bash

  • Install and start server

    dnf -y install yum-utils
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
dnf -y install vault procps-ng

# Create /etc/vault.d/vault.hcl

export VAULT_ADDR="http(s)://vault_X:8200"
vault server -config=/etc/vault.d/vault.hcl &

  • Open active (first) node

    vault operator init &   # TLS attempt broke here
vault operator unseal
vault login

  • Open standby nodes

    vault operator raft join http://vault_1:8200
vault operator unseal
vault login

  • Validate cluster state

    vault status
...
Initialized             true
Sealed                  false
Storage Type            raft
HA Enabled              true
HA Cluster              https://vault_ad_1:8201
# In active node
HA Mode                 active
Active Since            <Timestamp>
# In standby nodes
HA Mode                 standby
Active Node Address     http://vault_ad_1:8200

vault operator members
Host Name   API Address               Cluster Address            Active Node
---------   -----------               ---------------            -----------
<ID>        http://vault_ad_1:8200    https://vault_ad_1:8201    true
<ID>        http://vault_ad_2:8200    https://vault_ad_2:8201    false
<ID>        http://vault_ad_3:8200    https://vault_ad_3:8201    false

My setup

  • Host

    uname -r
4.19.0-21-amd64
cat /etc/lsb-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=4
DISTRIB_CODENAME=debbie
DISTRIB_DESCRIPTION="LMDE 4 Debbie"
dpkg -l docker-ce
...
ii  docker-ce      5:20.10.17~3-0~debian-buster amd64        ...

  • Container

    uname -r
4.19.0-21-amd64
cat /etc/rocky-release
Rocky Linux release 9.0 (Blue Onyx)
rpm -q vault
vault-1.11.1-1.x86_64
3

Hi
I don’t see in you configuration:
a) CA certs config
b) tls_disable = 0 option

Below my piece of config (from k8s) but for configuration on hosts could be simmilar:

config: |
        ui = true
        listener "tcp" {
          tls_disable = 0
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
          tls_min_version = "tls12"
          telemetry {
            unauthenticated_metrics_access = "true"
          }
        }

        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "//vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
        }

Additionally, could you please paste config file for openssl to prepare csr for signing ?
Regards

4

That’s right. Questions:
a) Is there something included in Vault installer that can be used here? I really don’t know what can make Vault accept it’s own provided certificate.
b) Do this option disable TLS altogether in all connections?

I have just googled csr… My openssl config file is the factory default, so I bet there’s nothing about what you mentioned.

Should I sign the provided certificate and setup an authority file? This is what I can grasp so far - I’m really clueless about this.

Thanks!

5

Hi
First you must create own CA or maybe in your organisation you have PKI.
When you create CA you can create key and csr (for example with openssl command)
When you sign a csr, you should have all data for TLS: CA cert, key and certificate for Vault.

The openssl conf file you must prepare by your own, because it’s depend for example for you host domain if you use if not it can be a simmilar to configuration for k8s installation:

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = vault-0.local
IP.1 = 192.168.0.10

for every one host that you install vault.

1 Like
6

Good. Will check with security staff for a PKI. If I can’t use it, will fallback to building a CA, maybe with Vault itself.
Thanks!

7

Hi @EmersonPrado,

One thing you haven’t mentioned yet, is who and what is going to be connecting to your Vault instance.

TLS certificates are all about trust - both client and server agreeing which CAs they agree can sign statements about identity.

Fundamentally, you have this choice:

  • Either get certificates from a CA your clients already trust
  • Or plan how your clients are going to be reconfigured to trust a new CA

If you think about it this way, it should help you clarify what you actually need. To put it another way - creating a CA is only the beginning - a CA is useless until people/systems begin to trust it.