Connect to Vault via Consul service mesh


I have Consul WAN Federation via mesh gateways with two datacenters in kubernetes clusters.

In the first datacenter I have installed Vault with TLS and ACL enabled and registered it as Consul service.
And I want to connect to Vault from pods in the second datacenter through the service mesh.

I have implemented several examples from Consul tutorials and everything worked as expected. I was able to create two services in defferent datacenters and connect them to each other. But all examples used consul proxy sidecars for communications between services in the service mesh.

As I understand Vault doesn’t support consul dataplane, so I can’t inject proxy sidecar into it.
But it supports consul service registration and I assume that I should be able to connect to the vault thourgh consul service mesh.
But I still can’t find right configuration for my pod to achieve this.

I inject consul sidecar into my test pod. And also I have vault-injector installed in dc2.

Vault is available by address vault.service.dc1.consul and I can dig it from pods in the dc2.

root@static-client-55448b6d7-pzzvm:/# dig vault.service.dc1.consul

; <<>> DiG 9.11.3-1ubuntu1.18-Ubuntu <<>> vault.service.dc1.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23191
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e43140c038c3ca78 (echoed)
;vault.service.dc1.consul.  IN      A

vault.service.dc1.consul. 5 IN      A
vault.service.dc1.consul. 5 IN      A
vault.service.dc1.consul. 5 IN      A

But when I try to connect to it via vault-injector I’m getting error “connection refused”.

[ERROR] auth.handler: error authenticating: error="Put \"https://vault.service.dc1.consul:8200/v1/auth/k8s-dc2/login\": dial tcp connect: connection refused" backoff=2m47.26s

I use following annotations in my pod: 'true' "true" 'secrets/dev/vault-demo/demoapp' 'auth/k8s-dc2' kubernetes dev-read "https://vault.service.dc1.consul:8200"

I have tried vault address without port, just "https://vault.service.dc1.consul"

and to add 'false' 'vault:8200:dc1' "http://localhost:8200"

but still no any luck.

What is missing in my configuration?

It looks like I need to use terminating gateway.

My infrastructure is something like this:

I have tried following annotations in my test app: 'true' 'vault:8200:dc1' "true" 'secrets/dev/vault-demo/demoapp' 'auth/k8s-dc2' kubernetes ultron-dev-read "https://vault.service.dc1.consul:8200"

But in this case vailt-injector init container starts befor consul-dataplane sidecar and it not able to connect to vault
Following error is appeared in vault-injector log:

[ERROR] auth.handler: error authenticating: error="Put \"https://vault.service.dc1.consul:8200/v1/auth/k8s-dc2/login\": dial tcp connect: connection refused" backoff=4m17.6s is correct ip addres of vault pod in dc1.

If vault can register itself in consul - how I can use it?

You can try disabling the init container if your application can withstand that.