Can't connect to Nomad API

Nomad
1

Hi All,
I just started exploring Nomad and hitting issues with the tutorial itself. I am not able to hit the http api of nomad. I am starting a dev agent using the command s

sudo nomad agent -dev -acl-enabled true -bind 0.0.0.0 -log-level INFO
  • tried to access UI and can’t even access that.
  • ran a netstat to check if something is running on 4646 and I don’t see any thing running
  • I don’t see anything errors when I started the dev agent
  • running this on RedHat 7.6 server and nomad version Nomad v1.1.4
usr:~ > nomad status
No running jobs

can some one please help me debug this ?

2

Hey @tarun-teja :wave:

Looking at the command line snippets you’ve pasted in gives me a couple of questions and ideas for solutions, so I’m going to see if I can give you a few troubleshooting angles to look into. I’m going to start with my experience reproducing your issue.

I brought up a RHEL7 VM in Vagrant using the following Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "generic/rhel7"
end

I added an entitlement through subscription-manager so that I could have access to the yum repositories. I installed unzip, and I fetched Nomad with wget. Unzipped Nomad, tossed it into /usr/bin (for easiest sudo-ing), and removed the download.

sudo subscription-manager register
sudo subscription-manager attach
sudo yum install unzip
sudo wget https://releases.hashicorp.com/nomad/1.1.4/nomad_1.1.4_linux_amd64.zip
sudo unzip nomad_1.1.4_linux_amd64.zip
sudo mv nomad /usr/bin/nomad
sudo rm nomad_1.1.4_linux_amd64.zip

Now, I made an additional vagrant ssh session so that I could launch the Nomad dev agent in one window and have a shell in the other. Next, I ran your supplied command.

sudo nomad agent -dev -acl-enabled true -bind 0.0.0.0 -log-level INFO

Everything started up like it should, but this is where I see my first difference. When I try to runnomad status against that dev agent I get a 403 error.

$ nomad status
Error querying jobs: Unexpected response code: 403 (Permission denied)

This error is expected until you have bootstrapped the Nomad ACL system. Since you did not appear to get this error, it leads me to believe that there might be another Nomad process running on your node that doesn’t have ACLs enabled. That said, if you can run nomad status and do not have values set for NOMAD_ADDR or NOMAD_TOKEN, you should be able to connect to the API at http://127.0.0.1:4646. For example, http://127.0.0.1:4646/v1/agent/self

Now, if you have already bootstrapped your ACLs, to get the “No running jobs” message, you will have had to supply your Nomad token using the NOMAD_TOKEN environment variable (or passed it in with a command-line flag, which you did not mention). If that’s the case, you will need to pass the token as a header to your API requests. For example:

curl -H "X-Nomad-Token: 11111111-2222-3333-4444-555555555555" http://127.0.0.1:4646/v1/agent/self

But it does seem suspect that you started an ACL-enabled dev agent and didn’t encounter a permissions issue with nomad status, so I would definitely start there.

Hopefully this gives you some ideas,
Charlie

1 Like
3

Hi @angrycub , thanks for responding.
I have bootstrapped ACL and I have an environment variable for NOMAD_TOKEN.
passing the token to curl gives me a gateway timeout html

curl -H "X-Nomad-Token: my_token" http://127.0.0.1:4646/v1/agent/self

end of the html is :

<hr>
<table>
  <tr valign="top">
    <td valign="top" width="10%">Notification&nbsp;codes:&nbsp;</td>
    <td valign="top" class="code" width="90%">(1, GATEWAY_TIMEOUT, 127.0.0.1)</td>
  </tr>
</table>

</body>

tried to curl localhost and still the same timeout error. Is there a way to know if the API is actually running on port 4646 ?

-Tarun

4

I can tell from that output that you didn’t make it to a Nomad agent. The /v1/agent/self endpoint would have returned a large JSON output or a 403. Gateway timeout errors tend to go with proxy servers, and since curl can directly support proxy server configurations I think that might be what is biting you. It appears that you can turn off proxies explicitly with the --noproxy * flag, or at least that was how the curl manpage read. Running your curl command with the -v flag may also give you some additional feedback to determine what could be happening.

Hopefully this can get you unjammed!
Charlie

1 Like
5

That was it. I can’t believe it was the proxy. After unsetting the proxy variable ( which was set from my bash_profile) and adding an ACL token
curl -H "X-Nomad-Token: my_token" http://127.0.0.1:4646/v1/agent/self
gives a JSON string as a response. Thanks for bringing me out of this jam.