Vault as Consul Connect CA when Vault uses Consul as storage backend

Consul
,
1

If I have a vault cluster using a consul cluster as storage backend, and want to use the vault as the Connect CA - is it ever possible? It seems they depend on each other.

  • Vault: needs Consul for storage
  • Consul: needs Vault for Consul Connect CA

Does Consul defer running consul connect until vault is up (e.g. retrying)?

Or should I use other vault storage backend (e.g. now integrated storage seems a good option)?

1 Like
2

I’m running into the same seemingly circular dependency and wondering if you found a solution?

What I have working now is with consul config like this:

verify_incoming = false,
verify_outgoing = false,
verify_server_hostname = false,
ca_file   = "{{vault_agent_cert_path}}/consul.ca",
cert_file = "{{vault_agent_cert_path}}/consul.crt",
key_file  = "{{vault_agent_cert_path}}/consul.key",

ports = {
  "dns" = 8600
  "http"  = 8500
  "https" = 8501
}

and I generate some “bootstrap” certs using the consul built in CA and bake those into the VM image so consul will start up. This way consul is up to work with http and https, so vault is able to initialize and uses the plain http to establish the consul backend, then vault agent can replace the bootstrap certs with vault issued certs. Consul connect is able to work with this configuration (although I only got it working yesterday so still need to confirm it’s not using bootstrap certs for some bits)

At some point I’d like to have consul only support https, and I think maybe that’s possible if I include certs in the vault machine VM image which are issued by the same bootstrap CA (e.g. using a common base image that has the CA already), but I’m not sure if that’s going to cause a problem when rolling out new images or something, or if this is even the best practice approach. Maybe there’s a simpler way and we’re just missing something obvious here.

3

I’m no consul expert but are you guys confusing consul access with vault → consul backend storage?

  • Vault → Consul is already encrypted.
  • You can add a TLS tunnel around the connection if you like, unnecessary, but doable.
  • Consul can act as it’s own Cert Manager, or you can off-load that out to Vault if you already have a PKI engine setup in vault and you don’t want to manage two different CAs.
4

You’ve probably moved on, but for others that come across this, if you use a Consul cluster for Vault storage, you should not use that Consul cluster for regular service registration and discovery. Make a separate Consul cluster.

5

is there any documentation on why is that so? It is less appealing to maintain two Consul Clusters and peering them.

6

You’re not supposed to “peer” them.

It is quite simply that you shouldn’t use the same Consul deployment in both a security-critical internal storage role, and in a widely accessible service discovery role, for reasons of security defence in depth, and segmenting failure domains and maintenance planning requirements.

7

At this point, is probably more appropriate for all new Vault deployments to prefer Raft (Integrated Storage) over Consul storage anyway.

It has now matured enough to be a good option and appears to be the direction in which HashiCorp are prioritising future development.

8

to answer my own question: check the Integrated Storage vs. Consul as Vault Storage part in Storage Backends - Configuration | Vault by HashiCorp

2 Likes
9

I knew I saw it somewhere! Thanks for posting the answer.