Best way to provide users access to Nomad UI and API

Nomad
1

I’m trying to find out the best way to provide users access to the Nomad cluster to run and check jobs, tasks, view logs etc.

I have a configuration, which enabled mTLS between the nodes and verifies the clients using client certificates. So now I have to provide also users a client certificate for the users, so they can hit the UI or API. For now, I’m not using ACLs.

I find it a bit inconvenient to generate TLS certificates for every user (and manage the expiration of those). They also need to keep them and set every time the environment variables so the Nomad CLI works. This gets even more complicated, when there are multiple clusters.

I’m curious, if there are some scripts, tools or configuration, which allow making it easier. We have also Vault integrated with our IdP, so we could allow the users to generate TLS certificates for Nomad cluster with Vault, but this would require additional tooling or scripts.

I’m attaching the Nomad configuration:

datacenter = "dc1"

tls {
  http = true
  rpc  = true

  ca_file   = "/etc/nomad.d/certificates/tls/ca.pem"
  cert_file = "/etc/nomad.d/certificates/tls/cert.pem"
  key_file  = "/etc/nomad.d/certificates/tls/key.pem"

  verify_server_hostname = true
  verify_https_client    = true
}

acl {
  enabled = false
}

server {
  enabled          = true
  bootstrap_expect = 3
  encrypt = "<hidden>"
}

client {
  enabled       = true
}

consul {
  address = "127.0.0.1:8501"

  ssl = true
  verify_ssl = false
  ca_file   = "/etc/nomad.d/certificates/tls/ca.pem"
  cert_file = "/etc/nomad.d/certificates/tls/cert.pem"
  key_file  = "/etc/nomad.d/certificates/tls/key.pem"
}

vault {
  enabled = true
  address = "<hidden>"
  token = "<hidden>"
}
2

Hi @Trojan295,

you could use a proxy in front of nomad with the client TLS certificates.

In combination with vaults PKI engine and consul template you could make use of short lived TLS certificates that get automatically renewed.

To control access to the cluster itself you can use nomads ACL system directly, or couple it with vault again to use eg. and existing LDAP auth.

1 Like
3

Hey, thanks for the idea!

I like it much better as everything could be managed by Vault and the users can come from an IdP like Okta.

4

Hi @Trojan295 and @schlumpfit,

I just wanted to point out there is a tutorial available for running a UI reverse proxy which might be of interest and use here.

Thanks,
jrasell and the Nomad team

5

Thanks @jrasell! Didn’t know there was a tutorial on the Nomad website for it.

One thing I think could be improved by the Hashicorp guys is how to set up a production Nomad cluster.

I know there is Installing Nomad for Production | Nomad | HashiCorp Developer, but to be honest, after reading and actually setting up a cluster with Consul, Vault, mTLS and securing the stuff I don’t really get the simplicity argument for Nomad compared to e.g. Kubernetes.

Kubernetes had kubeadm, which simplifies creating a cluster a lot, while in Nomad I feel like in a DIY shop.

6

Hi @Trojan295,

Thanks for the feedback and this is something we are certainly currently aware of. I am not familiar with kubeadm, so will do some research in this area. Thanks again.

jrasell and the Nomad team

7

Hi @jrasell and @Trojan295 ,

I would say that this is kind of the equivalent of kubeadm or kubespray or kops:

But still quite a hustle to get everything right.

Is there anything on the roadmap about how this could be simplified?

I think an ideal scenario would be some one-liner to get something like

  • vault-cluster with PKI, app-role, token-auth, acl-enabled
  • consul-cluster using vault pki with service mesh enabled
  • nomad-cluster using vault,consul docker driver ready and a reverse-proxy added to service mesh exposing all 3 APIs dynamically configured via consul.

I think just adding more and more features to nomad itself is not the right way for production setups. I understand that it helps for evaluation, but I think that it would be great to have some best-practices setup. While flexibility is great to have it can make your life pretty hard in the real world :smiley: