Hello,
I have Consul WAN Federation via mesh gateways with two datacenters in kubernetes clusters.
In the first datacenter I have installed Vault with TLS and ACL enabled and registered it as Consul service.
And I want to connect to Vault from pods in the second datacenter through the service mesh.
I have implemented several examples from Consul tutorials and everything worked as expected. I was able to create two services in defferent datacenters and connect them to each other. But all examples used consul proxy sidecars for communications between services in the service mesh.
As I understand Vault doesn’t support consul dataplane, so I can’t inject proxy sidecar into it.
But it supports consul service registration and I assume that I should be able to connect to the vault thourgh consul service mesh.
But I still can’t find right configuration for my pod to achieve this.
I inject consul sidecar into my test pod. And also I have vault-injector installed in dc2.
Vault is available by address vault.service.dc1.consul and I can dig it from pods in the dc2.
root@static-client-55448b6d7-pzzvm:/# dig vault.service.dc1.consul
; <<>> DiG 9.11.3-1ubuntu1.18-Ubuntu <<>> vault.service.dc1.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23191
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e43140c038c3ca78 (echoed)
;; QUESTION SECTION:
;vault.service.dc1.consul. IN A
;; ANSWER SECTION:
vault.service.dc1.consul. 5 IN A 10.42.23.47
vault.service.dc1.consul. 5 IN A 10.42.248.110
vault.service.dc1.consul. 5 IN A 10.42.225.233
But when I try to connect to it via vault-injector I’m getting error “connection refused”.
[ERROR] auth.handler: error authenticating: error="Put \"https://vault.service.dc1.consul:8200/v1/auth/k8s-dc2/login\": dial tcp 10.42.23.47:8200: connect: connection refused" backoff=2m47.26s
I use following annotations in my pod:
consul.hashicorp.com/connect-inject: 'true'
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-config.txt: 'secrets/dev/vault-demo/demoapp'
vault.hashicorp.com/auth-path: 'auth/k8s-dc2'
vault.hashicorp.com/auth-type: kubernetes
vault.hashicorp.com/role: dev-read
vault.hashicorp.com/service: "https://vault.service.dc1.consul:8200"
I have tried vault address without port, just vault.hashicorp.com/service: "https://vault.service.dc1.consul"
and to add
consul.hashicorp.com/transparent-proxy: 'false'
consul.hashicorp.com/connect-service-upstreams: 'vault:8200:dc1'
vault.hashicorp.com/service: "http://localhost:8200"
but still no any luck.
What is missing in my configuration?
