Remote error: tls: certificate required

Nomad
1

Hi, Nomad community! I’m newbee in Nomad!
Try more 2 days to resolve how to setup TLS with verify_https_client = true. I can’t find in documentation and Google what I need to do when enable this parameter.

I have Nomad v1.10.5 and Consul v1.19.2. When verify_https_client = false all jobs working fine and services can register in Consul. When true I get this error:

Setup Failure: failed to setup alloc: pre-run hook "consul" failed: 1 error occurred: * failed to derive Consul token for service nginx: Unexpected response code: 500 (rpc error making call: auth method validator for "nomad-workloads" could not be initialized: error checking JWKSURL: fetching keys oidc: get keys failed Get "https://10.0.0.2:4646/.well-known/jwks.json": remote error: tls: certificate required)

I’ve checked Consul token (working):

curl --cacert /etc/nomad.d/certs/nomad-agent-ca.pem \
     --cert /etc/nomad.d/certs/global-client-nomad.pem \
     --key /etc/nomad.d/certs/global-client-nomad-key.pem \
     -H "X-Consul-Token: d5002003-49be-5196-a536-c430f2ef8103" \
     https://10.0.0.2:8501/v1/agent/services
{"_nomad-client-qunv5tuilhvasig5jasuldn3qgeth7c7":{"ID":"_nomad-client-qunv5tuilhvasig5jasuldn3qgeth7c7","Service":"nomad-client","Tags":["http"],"Meta":{"external-source":"nomad"},.....

Certs Nomad (working):

curl --cacert /etc/consul.d/certs/consul-agent-ca.pem \
--cert /etc/consul.d/certs/dc1-server-consul-0.pem \
--key /etc/consul.d/certs/dc1-server-consul-0-key.pem \
     https://10.0.0.2:4646/.well-known/jwks.json
{"keys":[{"use":"sig","kty":"RSA","kid":"7b2066ea-34ff-da01-d8f5-5fa05c9a6bfe","alg":"RS256","n":"s--kD4I7qKVPNBreS3lquz1JqqQZaf7lNQZXCYNn0oPhCFPL7pRI_3EaJ4hmiuso0X5IhKtbfv4ILbVt0-UtamdHnsdPL2cHtgdy4rYHSv_xr8bZjaoKzFIetLmBC8Y73i2HjYm0NqLVlW-1uFRZoX9Uckzic3tvQpTqF3y_-kgPi9EC4Kym-3kpSJBhYECEfcJY7SThEmjiIe9AH0Ljf45Vy3iaCbLk4iZX41dA5U9iWjosMRxvw8YHcKEg2Y7hVIaiWS2_bbSxXoMtfJDPRy3_2hi1cnaOGlz8FRmmDN4J_xdHmRky95XLEd6Wwjko23qshopQRsXSLhtBpizmmw","e":"AQAB"}]}

Certs Consul (working):

curl --cacert /etc/consul.d/certs/consul-agent-ca.pem \
--cert /etc/consul.d/certs/dc1-server-consul-0.pem \
--key /etc/consul.d/certs/dc1-server-consul-0-key.pem \
-H "X-Consul-Token: d5002003-49be-5196-a536-c430f2ef8103" \
     https://10.0.0.2:8501/v1/agent/self
{"Config":{"Datacenter":"dc1","PrimaryDatacenter":"dc1","NodeName":"nomad-server1","NodeID":"18eeb6a3-9fcc-2cfb-1a38-183172895cd7","Revision":"048f1936","Server":true,.......

Here nomad.hcl server config (this is not full config, I add necessary only:

datacenter = “dc1”
data_dir   = “/opt/nomad”
server {enabled = true
……

tls {
  http = true
  rpc  = true

  ca_file   = "/etc/nomad.d/certs/nomad-agent-ca.pem"
  cert_file = "/etc/nomad.d/certs/global-server-nomad.pem"
  key_file  = "/etc/nomad.d/certs/global-server-nomad-key.pem"

  verify_server_hostname = true
  verify_https_client    = true
  #verify_https_client    = false
}

consul {
  enabled = true
  
  #token = "d184ce6b-3049-9e60-623f-295fc2fc625d" #bootstrap token
  token = "d5002003-49be-5196-a536-c430f2ef8103"
  server_service_name = "nomad-master"
  client_service_name = "nomad-client"
  service_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }
  task_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }

  auto_advertise = true

  server_auto_join = false
  client_auto_join = true

  ##### test ####
  #address = "10.0.0.2:8500"
  address = "10.0.0.2:8501"
  verify_ssl = true
  ssl       = true
  ca_file   = "/etc/consul.d/certs/consul-agent-ca.pem"
  cert_file = "etc/consul.d/certs/dc1-server-consul-0.pem"
  key_file  = "/etc/consul.d/certs/dc1-server-consul-0-key.pem"
}

Client config looking the same in these sections.

So where did I go wrong?

2

Is it answer?

It is highly recommended to use mutual TLS in production deployments of Nomad. With mTLS enabled, the tls.verify_https_client configuration must be set to false since it is not possible to provide client certificates to the Consul auth method.

https://developer.hashicorp.com/nomad/docs/secure/acl/consul#mutual-tls-in-nomad