Connecting to Vault, from a Consul Connect Service

Vault
,
1

Can someone please assist with the correct and recommended way to talk to vault from a service that is using consul connect

What I have done thus far:
Registered a termination gateway for vault using:

Kind = "terminating-gateway"
Name = "vault-termination-gateway"
Services = [
 {
   Name = "vault"
 }
]

In my nomad job, I have:

upstreams {
  local_bind_port = 8200
  destination_name = "vault"
}

and for the variables: (Please excuse the double $$, had to escape it to post it here)

VAULT_HOST = $${NOMAD_UPSTREAM_IP_vault}
VAULT_PORT = $${NOMAD_UPSTREAM_PORT_vault}

Background Info:
This service has been working with vault for a while, we are now trying to move it to consul connect. The Service has the vault CA cert in the trusted store.

Error we are getting:

Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for “https://127.0.0.1:8200/v1/secret/data”: Connection reset; nested exception is javax.net.ssl.SSLException: Connection reset

On full trace mode, we can see the SSL handshake start, but then the connection is reset. Broken Pipe.

My Questions.
Is a terminating gateway required when accessing vault via a sidecar, or is there something else we should be doing?

Note, the other services we access over consul connect, works fine.

2

Hey @CarelvanHeerden,

There are a couple of options here and it largely depends if you want to secure the connection to Vault using Consul.

In general, even though you are using consul connect for your services there should be nothing which is restricting outbound traffic. If the Vault server was accessible from the job previously it should remain so after using connect.

If however you do want to secure the connection to Vault using connect then you will have to take one of the following approaches.

  1. Register and start a proxy for every vault server
  2. Run a terminating gateway and register vault as an upstream endpoint

In both of these cases you need to consider the way that Vault forwards requests, any request which is not sent directly to the active node will be forwarded to the active node. When you are using Vault registered with connect traffic will round robin to all instances, it does not single out the active node. The standby nodes should have communication between each other to form a cluster and therefore the request forwarding should work, but always good to double check. If you only have a single node cluster then this issue can be ignored.

Looking at your error message normally that means that there is no route to the destination, the proxy accepts the request but can not manage end to end communication. When you say you see the handshake start, do you actually get a response connection from the other server, or are you only seeing the proxy accept the local TCP connection?

With your setup the hops to vault would be like:
Spring Application >>
Local Envoy Proxy >>
Terminating Gateway >>
Vault Server

There are a number of points where the connection could be dropped before it gets to the server. Have you tried execing into the Task and running something like netcat to validate the E2E connection?

netcat -z -v 127.0.0.1:8200 1-1000
3

Thanks Nic.

Since we are talking to Vault over TLS, we decided to drop using a sidecar for this purpose.

Thanks for the comments and the assistance.