Vault integration with Nomad - Workload identities, JWT, and custom CA

Vault
1

Hello,

I was following this tutorial:

and found myself stuck at this line:

vault write auth/jwt-nomad/config '@vault-auth-method-jwt-nomad.json'

Where vault-auth-method-jwt-nomad.json contains the following:

{
  "jwks_url": "https://192.168.250.2:4646/.well-known/jwks.json",
  "jwt_supported_algs": ["RS256", "EdDSA"],
  "default_role": "nomad-workloads"
}

My HashiStack is using MTLS, the above command failed because it didn’t accept the Nomad server certificate and didn’t offer up a valid client certificate either.

I disabled MTLS completely on Nomad and reran the above command and Vault accepted it(it was able to fetch the data from the Nomad server), however running Nomad without MTLS isn’t a thing I want to do.

I looked through the documentation and was unable to find a way to set this.

listener “tcp” {

This stanza has reference to certs, but only for the TCP listener.

service_registration “consul” {

This stanza has reference to certs, but only for interacting with Consul.

This page’s deprecation warning seems to imply that the certificates need to be configured in the jwt secrets engine, but I see no reference to TLS anywhere else in the page.

I tried using an AI to answer this question and it just hallucinated fields that don’t exist in jwt or the Vault configuration.

How do I tell the jwt engine to recognize a custom CA?
How do i provide the jwt engine with a client cert to authenticate?

Thanks,

David

2

Asking some people internally, but have you tried jwks_ca_pem?

3

That looks right. I will try that today, thank you!

David

1 Like
4

I am having exactly this same problem.
Did you have to leave Nomad’s mTLS turned off?
Or, were you able to figure out how to configure Vault to supply a client certificate? If so, how?

EDIT: For others who may be coming here, after encountering the same challenge… After doing some further research on this, I found a couple of related Git issues;

Based on that, it seems the way to move forward with this (at least for the moment) is to set tls.verify_https_client=false in the Nomad server config.

1 Like
5

I changed to tls.verify_https:client = false and still can’t get it working:

vault \
    write auth/jwt-nomad/config \
    jwks_url=https://server.global.nomad:4646/.well-known/jwks.json \
    bound_issuer=nomad \
    default_role=nomad-workload \
    disable_bound_issuer_validation=true \
    disable_jwks_validation=true
Error writing data to auth/jwt-nomad/config: Error making API request.

URL: PUT https://vault.dev.premis.app:8200/v1/auth/jwt-nomad/config
Code: 400. Errors:

* error checking jwks URL https://server.global.nomad:4646/.well-known/jwks.json

I have been using mTLS clients as well and it’s not something I wanted to give up, but I tried the JWT implementation to get clients able to access vault secrets, but I keep getting 400 errors. In the vault server log I see:

2025-05-12T02:01:31.790Z [ERROR] auth.jwt.auth_jwt_9da9937f: error checking jwks URL: url=https://server.global.nomad:4646/.well-known/jwks.json error="fetching keys oidc: get keys failed Get \"https://server.global.nomad:4646/.well-known/jwks.json\": remote error: tls: certificate require

and in the nomad server log I see these errors:

    2025-05-12T02:10:15.213Z [TRACE] http: http: TLS handshake error from 172.21.0.20:38076: tls: client didn't provide a certificate

I’m also using a custom/private vault CA for my client certs. Any idea what I could do to to get past this?

6

@will It appears your Nomad server still has mTLS enabled.
The tls section of the Nomad server configuration must set verify_https_client = false
For example;

..
....

tls {
  http = true
  rpc  = true

  ca_file   = "/etc/nomad.d/tls/nomad-agent-ca.pem"
  cert_file = "/etc/nomad.d/tls/global-server-nomad.pem"
  key_file  = "/etc/nomad.d/tls/global-server-nomad-key.pem"

  verify_server_hostname = true
  verify_https_client = false
}

....
..

Dont forget to restart the Nomad server(s) after you change the config