Can I use Consul Connect to proxy into the Nomad service itself as an upstream?

Nomad
,
1

We’d like one of our services (running on Nomad) to be able to interface with Nomad itself and access Nomad’s HTTP API. (Yes, we’re aware that’s very, very scary from a security perspective!)

And I noticed that consul lists nomad as one of the available upstreams.

So I slapped together a job to test it out: gist:83286c6fa051087794aa9fa5906639fc · GitHub

notably,

            upstreams {
              destination_name = "nomad"
              local_bind_port  = 1000
            }

I then attach to that container using docker exec <that container that's spun up by Nomad>, the idea being that I’d then be able to curl in the same network namespace.

So, once I’m in this container shell I try

root@aa54b9883229:/# env | grep "NOMAD_UPSTREAM_"
NOMAD_UPSTREAM_IP_nomad=127.0.0.1
NOMAD_UPSTREAM_ADDR_nomad=127.0.0.1:1000
NOMAD_UPSTREAM_PORT_nomad=1000

root@aa54b9883229:/# curl "http://127.0.0.1:1000/v1/nodes"
curl: (56) Recv failure: Connection reset by peer

# just for argument's sake, let's see what happens if I use a different port number
root@aa54b9883229:/# curl "http://127.0.0.1:9999/v1/nodes"
curl: (7) Failed to connect to 127.0.0.1 port 9999: Connection refused

So I guess my questions are:

  • is the nomad service listed in Consul a red herring?
  • if not, are there any code examples out there you know of showing [a service running in Nomad] consuming Nomad itself as an upstream?
  • if not, any idea what the missing sauce is here to be able to contact Nomad?
  • finally, is there any good way to trace a request going through Consul Connect to see where it’s getting caught up?

One last thought: I tried messing with Intentions and it didn’t seem to change anything.

(This is all on a local nomad-agent -dev / consul-agent -dev, for the time being)

2

Hi @wimax-grapl. Thanks for using Nomad.

I am pretty sure you can call the API server on the local client using ${attr.unique.network.ip-address}.

You could expose that to your workload with something like.

env {
  HOST_IP = "${attr.unique.network.ip-address}"
}

You might also need to add in the port, but get’s you started. Does that work with your use case? If not, maybe we can dig into the Consul questions.

3

Hey Derek,

Yeah, that does do the trick in bridge mode and unblocks me for the time being. (I guess I got too down-the-rabbit-hole on the Consul Connect stuff!)

4

And would this solution work even in a production, multi-agent-client-server setup?

5

It should. Every client is an agent and can forward API requests. Just be aware that if you are using mTLS, it’s important you have your mTLS client settings correct, and if you are using ACLs you’ll need a NOMAD_TOKEN and you’ll need to pass it in your request headers as X-Nomad-Token.

If you or anyone reading, are not familiar with how to set up mTLS in general, or with Nomad specifically, you can check out how we did it in the Nomad OpenAPI project. The mTLS setup in that project is a bit more straightforward to read quickly than in Nomad itself. Nomad has complex requirements around mTLS set up, while the OpenAPI client is very much in sync with this use case of I need a simple client that can talk to the API server please. That code also illustrates using the X-Nomad-Token header. Hope that helps.

In fact, I hope you consider using the Nomad OpenAPI project for your project. We built it for folks with exactly this use case, and if Go isn’t your language of choice, there a several other generated clients in other languages. If you don’t see the language you need, adding a new one takes about 15 minutes. I’d be happy to help.

Cheers!