Accessing Nomad REST API/UI via HTTPS

Nomad
1

I have a nomad cluster successfully configured to use tls for both tls and http communication. The server config looks like

datacenter            = "$instance_region"
name                  = "$instance_id"
region                = "$instance_region"
bind_addr             = "0.0.0.0"
disable_update_check  = true

advertise {
  http = "internal-ip:4646"
  rpc  = "$instance_ip_address"
  serf = "$instance_ip_address"
}

server {
  enabled = true
  bootstrap_expect = $num_servers

  encrypt = "${NOMAD_ENCRYPT_KEY}"
  server_join {
    $retry_join_json
  }
}

tls {
  http = true
  rpc  = true

  ca_file = "/etc/ssl/ssc/nomad-ca.pem"
  cert_file = "/etc/ssl/ssc/server.pem"
  key_file = "/etc/ssl/ssc/server-key.pem"

  verify_server_hostname = true
  verify_https_client    = true
}

The Nomad CLI and Nomad clients have to problem connecting with the server. e.g. the cli responds correctly to nomad job status -address https://localhost:4646 -client-cert=/etc/ssl/ssc/cli.pem -client-key=/etc/ssl/ssc/cli-key.pem -ca-cert=/etc/ssl/ssc/nomad-ca.pem.

However, making API requests via curl fails when using any of client.pem/client-key.pem, cli.pem/cli-key.pem, or server.pem/server-key.pem. E.g.

curl --cacert /etc/ssl/ssc/nomad-ca.pem --cert /etc/ssl/ssc/client.pem --key /etc/ssl/ssc/client-key.pem https://localhost:4646/ui
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

Ultimately the goal is to load the UI in a browser, either via an ssh tunnel and registering the correct certs with the browser, or by setting up an nginx proxy (as described in this tutorial). However, if I can’t even get a curl request from within the cluster to work, I don’t see how a browser outside the cluster could have much success.

2

Probably not solving your problem, but we run caddy as a reverse proxy for TLS (with letsencrypt cert).

We configured traefik to passthrough tcp 443 to caddy (and proxy 80) and store the certs in consul so we can run a 2 or more caddy instances with the same cert.

with official certs we don’t need to pass anything on nomad cli. Of course this only works if you use official public domain names and public IPs.

3

Hi @jameslaneconkling .

I’ll take a look at this and see if I can replicate. Do you happen to have Consul enabled and if so, can you provide that configuration? Also, can you provide your NGINX config?

Also you stated:
“I have a nomad cluster successfully configured to use tls for both tls and http communication”.

Did you use this tutorial or or some other resource?

Derek

4

It actually looks like this is a limitation of curl running on CentOS: see Stack Exchange posts here and here. When I ran curl on an Ubuntu machine with the same certs, it succeeded.

Haven’t actually set up the nginx proxy yet, though I don’t expect any issues as the above curl issue shouldn’t apply to nginx.

5

Great to hear you figured it out! Let us know if you have any NGINX issues. Sorry, I should have realized from you previous comments that you hadn’t done that part yet.