Nomad can't communicate with consul

Nomad
,
1

Hi,

I am trying to configure nomad + consul with TLS certificates.

My issue is that Nomad service can’t start and I see this in the logs:

Jun 08 19:26:56 nid001308 nomad[52332]:     2023-06-08T19:26:45.965+0200 [ERROR] nomad: error looking up Nomad servers in Consul: error="server.nomad: unable to query Consul datacenters: Get \"https://127.0.0.1:8500/v1/catalog/datacenters\": http: server gave HTTP response to HTTPS client"
...
Jun 08 19:26:56 nid001308 nomad[52332]:     2023-06-08T19:26:47.789+0200 [WARN]  nomad.raft: no known peers, aborting election
Jun 08 19:26:56 nid001308 nomad[52332]:     2023-06-08T19:26:55.994+0200 [ERROR] worker: failed to dequeue evaluation: worker_id=577f6465-589a-0b60-774f-b28a586d460b error="No cluster leader"

And

$ export NOMAD_ADDR=https://localhost:4646
$ export NOMAD_CACERT=/root/nomad-agent-ca.pem
$ export NOMAD_CLIENT_CERT=/root/global-cli-nomad.pem
$ export NOMAD_CLIENT_KEY=/root/global-cli-nomad-key.pem
$ nomad node status

Error querying node status: Unexpected response code: 500 (No cluster leader)

Consul shows a leader and all expected members:

$ consul operator raft list-peers
Node       ID                                    Address            State     Voter  RaftProtocol
nid001308  bdce84f4-d981-3302-c6e8-4c1f68a9fc5f  10.100.4.181:8300  follower  true   3
nid001309  bf64830a-5d0b-73dc-2f35-e6b88b583134  10.100.4.197:8300  leader    true   3
nid001310  3546b013-4d74-6248-11ee-279a98bc345c  10.100.4.247:8300  follower  true   3

NOTE: I am not specifying any CA or TLS certs when using the consul cli (I thought it would not work?)

config files:
consul - client.hcl

data_dir = "/opt/consul"

consul - server.hcl

server = true
data_dir = "/opt/consul"
bootstrap_expect = 3
ui_config {
  enabled = true
}
client_addr = "0.0.0.0"
tls {
  internal_rpc {
    verify_incoming = true
    verify_outgoing = true
    verify_server_hostname = true
  }
}

consul -consul.hcl

log_level = "DEBUG"
datacenter = "dc"
retry_join = ["nid001308","nid001309","nid001310"] 
data_dir = "/opt/consul"
tls {
  defaults {
    ca_file   = "/root/consul-agent-ca.pem"
    cert_file = "/root/dc-server-consul-0.pem"
    key_file  = "/root/dc-server-consul-0-key.pem"
    verify_incoming = true
    verify_outgoing = true
  }
}

nomad - client.json

{
  "client": [
    {
      "enabled": true,
      "meta": [
        {"mem.total_mb": "257330", "processor.nproc": "256", "processor.count": "2", "processor.cores": "64"}
      ],
    }
  ],
  "plugin": [
    {
      "raw_exec": [
        {
          "config": [
            {
              "enabled": true
            }
          ]
        }
      ]
    }
  ],
}

nomad - server.hcl

server {
  enabled = true
  bootstrap_expect = 3
}

nomad - nomad.hcl

datacenter = "dc"
data_dir = "/opt/nomad"
consul {
  address             = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
  ssl = true
  ca_file   = "/root/consul-agent-ca.pem"
  cert_file = "/root/dc-server-consul-0.pem"
  key_file  = "/root/dc-server-consul-0-key.pem"
  verify_ssl = true
}

tls {
  http = true
  rpc  = true

  ca_file   = "/root/nomad-agent-ca.pem"
  cert_file = "/root/global-server-nomad.pem"
  key_file  = "/root/global-server-nomad-key.pem"

  verify_server_hostname = true
  verify_https_client    = true
}

Could someone please help me troubleshoot/fix why Nomad does not start?

thank you

2

Hi @masuberu,

I can see that you have configured the TLS related settings in your Consul configuration (using the tls {} block). However, Consul by default won’t configure the https port even with the tls{} settings. You have to explicitly enable the https port by setting the ports.https in the Consul configuration file.

eg:

ports {
    https = 8501
}

Once you make the above changes and restart Consul, change the consul.address in your Nomad configuration file to use the TLS port of Consul (127.0.0.1:8501).

I hope this helps.