I’m looking for suggestions on how to easily secure my Nomad cluster. I’ve been trying to use mTLS, but I’m finding it to be quite difficult. Generating certificates, ensuring hostname match, distributing them to clients, etc., is all a lot of work.
Is there any way to make this process easier? Are there any other security mechanisms that I should consider? Would just ACL be sufficient?
Once we switched to mTLS many of our tools stoped working. Such as:
$# nomad status
Error querying jobs: Unexpected response code: 400 (Client sent an HTTP request to an HTTPS server.)
Also http interface doesn’t allow us to connect. Basically we need to add CA into a browser and each client needs a certificate? With introduction of mTLS it becomes unclear how things work together.
I would suggest the security concepts page as a place to start in order to understand how to better secure a cluster. I would suggest both mTLS and ACLs being required to secure a cluster, rather than just ACLs.
If you have any questions or require guidance on mTLS, please let me know.
We are trying to setup mTLS but the servers won’t connect.
[WARN] nomad.rpc: failed TLS handshake: remote_addr=xxx.xxx.xxx.xxx:55092 error="remote error: tls: bad certificate"
[WARN] nomad.rpc: failed TLS handshake: remote_addr=xxx.xxx.xxx.xxx:33162 error="invalid role or region for certificate"
We generated server certificates for each server but it doesn’t seem to work properly. Do we need extra certificate for each server?
Also on cli I get:
Error submitting job: Put "https://myserver.mydomain.com:4646/v1/jobs": tls: failed to verify certificate: x509: certificate is valid for server.europe.nomad, server.global.nomad, localhost, not myserver.mydomain.com
or submitting job without verification:
nomad job run -tls-skip-verify my.job.hcl
Error submitting job: Put "https://myserver.mydomain.com:4646/v1/jobs": remote error: tls: bad certificate
When I disable server verification the servers will connect. But it’s not safe. I think there should be some simper way. Frankly, this is one of the biggest issues which keeps us from Nomad for now. It’s supposed to be easy but mTLS is very complicated to setup, debug and maintain.
Along with what @dbd mentioned, are you able to provide the commands you’re using the generate the TLS certificates and any server configuration objects? The error message indicates the problem might be with the certificate DNS names.
I agree mTLS can be complicated to setup and initially debug, however, it provides robust mutual authentication for Nomad which is why it is included and recommended.