Port 4646 not accessible in nomad ui

Nomad
1

I’m new to nomad. I found the rep hashicorp-education/learn-nomad-cluster-setup (sorry, I couldn’t make this a link due to new user restrictions) to try to give me a quick overview.

Using the above repositoru, I was able to create a cluster in DigitalOcean. However, not everything came up as expected. In the nomad services, I see the error message:

Get “http://0.0.0.0:4646/v1/agent/health?type=server”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

If I SSH into the machine, I get the following error:

curl -X GET http://0.0.0.0:4646/v1/agent/health?type=server
{“server”:{“ok”:false,“message”:“No cluster leader”}}

I’m a little confused why the Consul UI has a different error.

For context, this is what the consul config looks like:

data_dir  = "/opt/nomad/data"
bind_addr = "0.0.0.0"

# Enable the server
server {
  enabled          = true
  bootstrap_expect = 2
}

consul {
  address = "127.0.0.1:8500"
  token = "8684x13x-x88x-3x61-xxxx-7x4x40x0041x"
}

acl {
  enabled = true
}

vault {
  enabled          = false
  address          = "http://active.vault.service.consul:8200"
  task_token_ttl   = "1h"
  create_from_role = "nomad-cluster"
  token            = ""
}

Any advice on how to correct this issue?

2

Hey @mike12, those errors seem a little unusual to me. For clarification, are you saying you’re not able to access the web UI at all? If you enter nomad ui from the command line, you should get Opening URL ....:4646 and your browser should open to the jobs index page. Does it load but fail to get resources?

3

Hi mike,

This occur because you configure your nomad to expect at min 2 nodes to create a cluster, whit this part of your configuration:

server {
  enabled          = true
  bootstrap_expect = 2
}

But 2 is not a good number for raft consensus (algorithm used for some products of hashicorp), i suggest change for 1 for test purpose, and the ui will be accessible.

When are you ready for use in production, i will recommend to change to 3.

:smile:

4

@phil.renaud - sorry for being vague. I can indeed see the UI, I’ve included a screenshot of the UI and the error in particular.

nomad-4646-error
nomad-4646-error2720×1908 402 KB

5

curl -X GET http://0.0.0.0:4646/v1/agent/health?type=server
{“server”:{“ok”:false,“message”:“No cluster leader”}}

when this error occurr is because for some reasons:

  • you dont have sufficient menbers to form a leader in election ( 3 or 5 nomad servers)
  • you had problem with nomad cache data with a broken nomad server node, when a node is broked

I suggested the first use case because the bootstrap expect is strange for me ( i use 1 or 3 nodes) and if you create 2 nodes servers and expect 2, you dont had a leader because when you choose to work with nomad cluster, the minimum is 3.

6

Thanks everyone. I’ve marked the thread that ended up as the solution.