Unable to make nomad and consul to work with tls configuration

Nomad
1

Dear hashicorp Nomad community,

I have a Nomad + Consul cluster and I wanted to configure TLS encryption, I followed this tutorial Enable TLS Encryption for Nomad | Nomad | HashiCorp Developer and this one for the nomad consul tls integration Hashicorp Nomad - Working with Consul Connect | Mike Polinowski

These are my config files:

consul server (server.hcl)

server = true
bootstrap_expect = 3
ui_config {
  enabled = true
}
client_addr = "0.0.0.0"

consul client (client.hcl)

datacenter = "home"
data_dir = "/opt/consul"
encrypt = "--REDACTED--"

consul.hcl

retry_join = ["nid010010"]

Nomad server (server.hcl)

server {
  enabled = true
  bootstrap_expect = 3
}
consul {
  address             = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
  ssl = true
  ca_file   = "/root/nomad-agent-ca.pem"
  cert_file = "/root/global-server-nomad.pem"
  key_file  = "/root/global-server-nomad-key.pem"
  verify_ssl = true
}

tls {
  http = true
  rpc  = true

  ca_file   = "/root/nomad-agent-ca.pem"
  cert_file = "/root/global-server-nomad.pem"
  key_file  = "/root/global-server-nomad-key.pem"

  verify_server_hostname = true
  verify_https_client    = true
}

nomad client (client.json)

{
  "client": [
    {
      "enabled": true,
      "meta": [
        {{ meta | to_json }}
      ],
    }
  ],
  "plugin": [
    {
      "raw_exec": [
        {
          "config": [
            {
              "enabled": true
            }
          ]
        }
      ]
    }
  ],
  "consul": [
    {
      "address"             : "127.0.0.1:8500",
      "server_service_name" : "nomad",
      "client_service_name" : "nomad-client",
      "auto_advertise"      : true,
      "server_auto_join"    : true,
      "client_auto_join"    : true
    }
  ],

  "tls": {
    "http" : true
    "rpc"  : true

    "ca_file"   : "nomad-agent-ca.pem"
    "cert_file" : "global-client-nomad.pem"
    "key_file"  : "global-client-nomad-key.pem"

    "verify_server_hostname" : true
    "verify_https_client"    : true
  }
}

nomad.hcl

datacenter = "--REDACTED--"
data_dir = "/opt/nomad"

consul info shows a leader…

Nomad seems in bad shape. I can’t check the web ui and the client complains about Error querying node status: Unexpected response code: 500 (No cluster leader)

An extract of the Nomad logs is below:

Jun 06 13:38:32 nid001314 nomad[240038]:     2023-06-06T13:38:32.099+0200 [WARN]  nomad.rpc: non-TLS connection attempted with RequireTLS set: remote_addr=10.100.4.27:54932
Jun 06 13:38:33 nid001314 nomad[240038]:     2023-06-06T13:38:33.280+0200 [WARN]  nomad.rpc: non-TLS connection attempted with RequireTLS set: remote_addr=10.100.4.27:54936

I created all the certs using nomad cli in my local laptop and then copied the files to the Nomad nodes. I am using the same Nomad certs in the consul stanza inside Nomad server configuration

Any idea of what could be wrong?

thank you very much

2

Hi @masuberu how many Nomad Server agents have you started?

In the server.hcl for Nomad you have

bootstrap_expect = 3

which means you’re expecting 3 Nomad servers, and Nomad will not be able to elect a leader until at least 2 of them are running and able to communicate.

3

Hi Seth,

I had an issue with the deployment and only 2 nodes were deployed, thats the reason of not being able to elect a leader…,
I fixed that error, then I redeployed and now seems to work

thank you