How to overcome the chicken and egg problem of consul and vault encryption?

What's Your Story?
1

For a full hashicorp stack, the guides and docs recommend that for production consul and vault deployments, they should talk to each other encrypted, and to benefit from the security of automatic key rotations, vault should be used as the certificate authority.

What is the recommended way to get consul, vault, and nomad securely communicating with each other, with the benefit of automatic key rotation, from initiation?

So far this question doesn’t have a simple answer, as the documentation avoids answering it for the sake of simplicity.

So far the options seem to be:

Without Automatic Key Rotation

Manually generating certs with terraform tls provider

  1. use terraform tls provider to generate the tls certs
  2. configure and boot consul, vault, nomad with the certs

Seems until v0.14 is out which supports a sensitive flag, this is insecure

Manually generating certs from Vault

  1. boot vault
    2 generate certs from vault
  2. reconfigure vault, and configure consul and nomad to use these certs
  3. reboot vault, and boot consul and nomad

Now that vault, nomad, and consul all support sans, the code here may now work

Manually generating certs from Consul

  1. boot consul
  2. generate certs from consul
  3. reconfigure consul, and configure vault and nomad to use these certs
  4. reboot consul, and boot vault and nomad

With Automatic Key Rotation

I just can’t seem to find instructions for this. All the instructions seem to be related to Consul Certificate Authority, which is only used automatically (with automatic key rotation) for Consul Connect, which Consul Connect is just used to secure the communication between apps, not the actual consul/vault/nomad services.

Hope

I really hope that there becomes a way to boot consul, nomad, and vault, with all of them just using either Consul or Vault as the certificate authority, and able to secure themselves automatically with TLS cert rotation.

What is essentially needed is fo rthe auto_encrypt flag to be extended to auto_encrypt not just the consul clients, but also the consul servers, as well as vault and nomad agents (clients+servers).

All the hashicorp products tout features that should make things extremely secure, yet the actual software doesn’t know how to use them.

Relevant Links

4 Likes
2

I am currently looking into the same issues.

For auto-renewing mTLS certificates in Consul with Vault, you can use consul-template. An example setup is described in “Generate mTLS Certificates for Consul with Vault”.

Now I’m wondering if it’s possible (or even a good idea at all) to follow a similar approach when setting up Vault itself: How to move Vault into his own intermediate?

3

After months I gave up. Hashicorp just seems focused on enterprise use cases and enterprise size clients. Trying to do any of this on your own is futile.

2 Likes
5

This is exactly how I feel right now, with this and most of the things in the higher scope than just this.