Vault cross DC replication using Consul federation

Vault
1

Hello all,

Here is what I’m trying to achieve using the consul-helm and vault-helm charts:

  • A single source of truth in Europe. A vault instance on which we can write data
  • streaming the data to remote Datacenters
  • Use of Kubernetes sidecar agent to inject secrets into pods

Well this is not obvious to setup the consul backend with a cross datacenter configuration.
Finally, I managed to do that using the beta federation beta feature

kubectl exec statefulset/consul-server -- consul members -wan
Node                 Address          Status  Type    Build       Protocol  DC   Segment
consul-server-0.dc1  10.4.2.91:8302   alive   server  1.8.0beta1  2         dc1  <all>
consul-server-0.dc2  10.48.0.65:8302  alive   server  1.8.0beta1  2         dc2  <all>
consul-server-1.dc1  10.4.0.43:8302   alive   server  1.8.0beta1  2         dc1  <all>
consul-server-1.dc2  10.48.2.65:8302  alive   server  1.8.0beta1  2         dc2  <all>
consul-server-2.dc1  10.4.1.49:8302   alive   server  1.8.0beta1  2         dc1  <all>
consul-server-2.dc2  10.48.1.42:8302  alive   server  1.8.0beta1  2         dc2  <all>

This feature requires TLS to be enabled and this is a good practices when we deal with secrets. However now I have to connect my Vault instances to these consul and unfortunately the charts are not really easy to use if you want something production ready.

Note that the acl’s are enabled on Consul side so I had to create a policy and put the token in Vault’s configuration. The last thing I have to figure out is how to create a certificate using the same CA. I don’t know how all these pki settings are handled internally so I’m not able to create a certificate .

How would achieve this cross-dc architecture ?
Is consul the right choice and is the federation the best approach for cross-dc replication? How to connect Vault to consul using TLS ?

Thanks for your help

2

I managed to get Vault working on both datacenters now.

  • I created the proper acl policy on consul side
  • I created a certificate based on the ca created when consul has been deployed
consul-connect-injector-webhook-deployment-8bd4bd4b6-44gg8   1/1     Running   0          19h
consul-jdr22                                                 1/1     Running   0          19h
consul-mesh-gateway-747c8f77f9-bk8mn                         2/2     Running   0          19h
consul-mesh-gateway-747c8f77f9-dsf7g                         2/2     Running   0          19h
consul-server-0                                              1/1     Running   0          19h
consul-server-1                                              1/1     Running   0          19h
consul-server-2                                              1/1     Running   0          19h
consul-tfqp9                                                 1/1     Running   0          19h
consul-wssw5                                                 1/1     Running   0          19h
vault-0                                                      1/1     Running   0          15m
vault-1                                                      1/1     Running   0          15m
vault-2                                                      1/1     Running   0          15m
vault-agent-injector-5d4d8fd767-vgnr4                        1/1     Running   0          16m

Now I have another issue: It seems that the vault instances are not writing in the same location.
When I write something in the DC1 the changes are not seen in DC2

Digging further

3

Ok it seems that I did the bad choice with Consul federation. The data is not replicated.

4

Any advice to how to perform consul replication would be welcome.
I’ve noticed consul-replicate but the last commit was 3 years ago

5

I tried that

consul kv export -datacenter dc1 | consul kv import -datacenter dc2 -
Imported: vault/core/audit
Imported: vault/core/auth
Imported: vault/core/cluster/local/info
...

The data is properly replicated on Consul side but are not seen by the Vault in the other DC

6

Well the previous command just f****d up my second vault instance.
I had to reinstall it.

After that I tried to import only the vault/logical path but I’m probably missing something because the second vault instance can’t see these items.

consul kv export vault/logical | consul kv import -datacenter dc2 -

7

Long story short is that this isn’t a supported configuration. There is an old github issue about this and AFAIK nothing has fundamentally changed since then

8

Thank you @drawks I suppose that the only way to get a georeplicated Vault is to have the entreprise edition.