Getting error when trying to access vault as a consul service - unsupported protocol scheme ""

Vault
,
1

Hello,
I have a consul federation in K8s with two datacenters.
In the dc1 I have installed vault and registered it as consul service. According to the documentation vault should be available and accessible at vault.service.consul

I can dig vault in a pod in the same dc1 by address vault.service.consul

root@static-client-55448b6d7-pzzvm:/# dig vault.service.consul

; <<>> DiG 9.11.3-1ubuntu1.18-Ubuntu <<>> vault.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34176
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 0ccf495a039c3b34 (echoed)
;; QUESTION SECTION:
;vault.service.consul.          IN      A

;; ANSWER SECTION:
vault.service.consul.   5       IN      A       10.42.248.110
vault.service.consul.   5       IN      A       10.42.225.233
vault.service.consul.   5       IN      A       10.42.23.45

and in a pod in dc2 by address vault.service.dc1.consul

root@static-client-55448b6d7-pzzvm:/# dig vault.service.dc1.consul

; <<>> DiG 9.11.3-1ubuntu1.18-Ubuntu <<>> vault.service.dc1.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23191
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e43140c038c3ca78 (echoed)
;; QUESTION SECTION:
;vault.service.dc1-fra.consul.  IN      A

;; ANSWER SECTION:
vault.service.dc1.consul. 5 IN      A       10.42.23.45
vault.service.dc1.consul. 5 IN      A       10.42.248.110
vault.service.dc1.consul. 5 IN      A       10.42.225.233

But when I try to use this address in vault-injector I’m getting error

auth.handler: error authenticating: error="Put \"vault.service.dc1.consul/v1/k8s-dc2/login\": unsupported protocol scheme \"\"" backoff=1s

Vault-injector is installed in dc2.

And I use following annotations in the demo pod:

      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-config.txt: demo/dev/vault-demo/demoapp
        vault.hashicorp.com/agent-pre-populate-only: 'true'
        vault.hashicorp.com/auth-path: k8s-dc2
        vault.hashicorp.com/auth-type: kubernetes
        vault.hashicorp.com/role: dev-read
        vault.hashicorp.com/service: "vault.service.dc1.consul"
        vault.hashicorp.com/tls-skip-verify: "true"

Please help how to connect to vault as consul service from vault-injector.

2

The error message shows it is expecting an URL, but you’re giving it a bare hostname.

Add the missing https:// or http:// prefix.

3

yep, my fault
but now I have another error in the vault-agent init container is

auth.handler: error authenticating: error="Put \"https://vault.service.dc1.consul/v1/k8s-dc2/login\": dial tcp 10.42.225.233:443: connect: connection refused" backoff=1m26.86s

How can I get more details about the reason of the error above?

Vault server and vault injector are in deferent clusters.

Something wrong in Consul setup?
According to the documentation Vault doesn’t support Consul dataplane and I can’'t inject consul sidecar into it.

The main goal is to connect to Vault from different cluster through Counsul.

I have injected consul sidecar into my pod and created service intention in consul (not sure if it was required)

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: vault-allow-all
spec:
  destination:
    name: 'vault'
  sources:
    - name: '*'
      action: allow
4

They’re right there in the error message? Here:

I guess your Vault isn’t actually listening on port 443.

5

It is failed even if I add port 8200

[ERROR] auth.handler: error authenticating: error="Put \"https://vault.service.dc1.consul:8200/v1/k8s-dc2/login\": dial tcp 10.42.23.47:8200: connect: connection refused" backoff=1.87s

image

And I can’t find anything in logs of Consul or Vault

6

OK… have you investigated why your pod in dc2 is unable to reach that IP address and port, then?

Please bear in mind that you’re asking strangers on the internet, who have absolutely no idea about the networking topology of your Kubernetes cluster to suggest an answer to a problem, where that information is highly relevant.

Interactions on this forum are highly dependent on the person asking the question volunteering such information, otherwise they tend to reach a point where the people responding start to give up, because they are, metaphorically, being presented with not enough of the puzzle to actually provide help.

I’m going to throw out a couple of ideas, but they’re wild guesses:

  • Let me assume that 10.42.x.x is the pod IP range for your dc1 K8s cluster
  • If it is, does the dc2 K8s cluster have a different pod IP range? If not, cross-cluster pod-to-pod communication is never going to work.
  • Even if that’s fine, is there appropriate network routing for those IP ranges, such that pod-to-pod cross-cluster communication works at all?

Assuming you manage to get the networking layer sorted, I also see another problem with your Vault URL:

The above is wrong - all auth methods include auth/ in their URL, so probably https://vault.service.dc1.consul/v1/auth/k8s-dc2/login is intended.

1 Like
7

yes, exactly, auth-path was wrong, but it still doesn’t solve the problem.

I have Consul WAN Federation via mesh gateways.
According to the documentation

image
image767×217 13.3 KB

I have implemented several examples from Consul tutorials and everything worked as expected. I was able to create two services in defferent datacenters and connect to the service in defferent datacenter. But all examples used consul proxy sidecars for communications between services in the service mesh.

Vault doesn’t support consul dataplane, so I can’t inject proxy sidecar into it.
But it supports consul service registration and I assume that I should be able to connect to the vault thourgh consul service mesh.
But I still can’t find right configuration for my pod to achieve this.

8

Ah, so you are using Consul service mesh. That is highly relevant information which you had not previously shared.

I have no relevant experience with Consul service mesh. I suggest you would be better off asking in the Consul section of these forums (and making it clear this is a mesh-related question).

9

I see, thank you for your help.
Will continue in the consul section.