Failed to verify certificate authority when creating nomad auth method

Consul
,
1

Hi, I am a Consul / Nomad newbie and I’m in the process of setting up my first cluster. I’ve deployed Nomad and Consul successfully in a basic configuration with server A running consul server / nomad server, and server B running consul client / nomad client. I have ACL and TLS enabled on both Consul and Nomad and everything seems OK in the Consul dashboard.

I am following the tutorial at Consul ACL with Nomad Workload Identities | Nomad | HashiCorp Developer to set up Nomad so that it can authenticate with the consul ACL when submitting jobs. However, I’ve run into an issue adding the authentication method. Firstly, I had to change the snippet from the tutorial to be HTTPS and to target the IP my nomad server was listening on at 10.0.0.2. When I try to execute the command consul acl auth-method create -name 'nomad-workloads' -type 'jwt' -description 'JWT auth method for Nomad services and workloads' -config '@nomad-workload-auth-method.json' I get the following error:

Failed to create new auth method: Unexpected response code: 500 (Invalid Auth Method: 
error checking JWKSURL: fetching keys oidc: get keys failed Get "https://10.0.0.2:4646/.well-
known/jwks.json": tls: failed to verify certificate: x509: certificate signed by unknown 
authority)

My assumption here based on the 500 error code is this is a request the Consul agent is making, rather than my CLI, and it is failing to verify the certificate authority.

What I’ve tried: I realized that I had configured the cluster with two separate CAs for nomad and Consul following the separate tutorials, however I unified these under one CA but the same error persisted. I also found this note about mutual TLS but setting tls.verify_https_client to false in the Nomad client and server configuration hasn’t changed anything either.

What’s interesting is that I am able to do curl https://10.0.0.2:4646/.well-known/jwks.json --cacert ~/certs/cluster-agent-ca.pem (with my shared CA) and I get the JWKS returned just fine. So I suppose it may be possible to set up a proxy to the Nomad JWKS endpoint, which is one of the other recommendations in the documentation, but I’m wondering if there may be an easier way out…

Thanks in advance for any help you may be able to provide :slight_smile:

1 Like
2

Solved my own problem almost immediately after posting by giving the docs a closer look - and found there is a parameter for this (see: JSON Web Token (JWT) Auth Method | Consul | HashiCorp Developer). I simply had to set “JWKSCACert” to the contents of the CA .pem file (replacing any actual newlines with \n) and it successfully made the policy!

3

Thank you so much! This saved me a lot of time debugging.
This same problem occurred on my cluster when upgrading to Nomad 1.9.6 (previously 1.9.5) i.c.w. Consul v1.20.4 (previously 1.20.2).

According to the docs, the system CA store should be used when JWKSCACert is not set.

If not set, system certificates are used.

I was able to curl the well-known jwks endpoint without specifying the --cacert flag from all hosts that run Nomad/Consul so the CA should be installed in the system certificate store correctly. Before the update there was no issue with the omitted JWKSCACert field.

Perhaps the system CA store has no effect on this mechanism anymore after the update?
In any case, the documentation seems to be out of sync with actual behavior.

Setting the JWKSCACert field brought everything back online. :slight_smile: