Nomad Agent not Starting

Nomad
1

Trying to run Nomad on MACOS but the following error is observed everytime.

=> nomad: failed to start RPC layer: error=“listen tcp 127.0.0.1:4647: bind: address already in use”

Tried the following

sudo lsof -i :4647

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nomad 86016 UBIL1 9u IPv4 0x481d7c9f443befcd 0t0 TCP localhost:4647 (LISTEN)
nomad 86016 UBIL1 12u IPv4 0x481d7c9f4e15bb0d 0t0 TCP localhost:65328->localhost:4647 (ESTABLISHED)
nomad 86016 UBIL1 13u IPv4 0x481d7c9f44a7e38d 0t0 TCP localhost:4647->localhost:65328 (ESTABLISHED)

sudo kill -9 86016

But instantly after killing they are again there with new PID.

sudo lsof -i :4647
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nomad 86226 UBIL1 9u IPv4 0x481d7c9f450a89ad 0t0 TCP localhost:4647 (LISTEN)
nomad 86226 UBIL1 12u IPv4 0x481d7c9f409fe12d 0t0 TCP localhost:49376->localhost:4647 (ESTABLISHED)
nomad 86226 UBIL1 14u IPv4 0x481d7c9f4dff3fcd 0t0 TCP localhost:4647->localhost:49376 (ESTABLISHED)

How can I resolve this?

2

Is there a service running and restarting your failed nomad agent? Or a docker container or something else?

3

Nothing is running on my local machine right now. And amazingly it shows that there is nomad running which is not.

4

Reboot? Is it than back up again?

5

Can you lookup the parent process of the rouge Nomad process? Sounds like you have some kind of supervisor trying to keep Nomad running, like docker or launchd.

6

Yes tried reboot just now. Same issue.

7

Yes, it is launchd. How to stop this from doing it?
Sorry, I am a bit new.

8

launchd 1 root cwd DIR 1,4 704 2 /
launchd 1 root txt REG 1,4 386976 1152921500313048922 /sbin/launchd
launchd 1 root txt REG 1,4 1568368 1152921500313047901 /usr/lib/dyld
launchd 1 root 0w CHR 3,2 0t0 313 /dev/null
launchd 1 root 1u CHR 3,2 0t0 313 /dev/null
launchd 1 root 2w CHR 0,0 0t1189 311 /dev/console
launchd 1 root 3w CHR 0,0 0t1189 311 /dev/console
launchd 1 root 4u unix 0x4687c0a280402339 0t0 /private//var/run/syslog
launchd 1 root 5u unix 0x4687c0a280402271 0t0 ->0x4687c0a280402339
launchd 1 root 6u unix 0x4687c0a280402401 0t0 /private/var/run/cupsd
launchd 1 root 7u unix 0x4687c0a2804024c9 0t0 /var/rpc/ncalrpc/NETLOGON
launchd 1 root 8u unix 0x4687c0a280402591 0t0 /var/run/vpncontrol.sock
launchd 1 root 9u unix 0x4687c0a280402659 0t0 /var/run/portmap.socket
launchd 1 root 10u unix 0x4687c0a2804021a9 0t0 /private/var/run/.sim_diagnosticd_socket
launchd 1 root 11u unix 0x4687c0a280402721 0t0 /var/run/systemkeychaincheck.socket
launchd 1 root 12u unix 0x4687c0a2804020e1 0t0 /var/run/mDNSResponder
launchd 1 root 13u unix 0x4687c0a280402019 0t0 /var/rpc/ncacn_np/lsarpc
launchd 1 root 14u unix 0x4687c0a280401f51 0t0 /var/rpc/ncalrpc/lsarpc
launchd 1 root 15u unix 0x4687c0a280401e89 0t0 /var/rpc/ncacn_np/mdssvc
launchd 1 root 16u unix 0x4687c0a280401dc1 0t0 /var/rpc/ncacn_np/wkssvc
launchd 1 root 17u unix 0x4687c0a2804027e9 0t0 /var/rpc/ncalrpc/wkssvc
launchd 1 root 18u IPv4 0x4687c0a27fde21b1 0t0 UDP *:netbios-ns
launchd 1 root 19u IPv4 0x4687c0a27fde2499 0t0 UDP *:netbios-dgm
launchd 1 root 20u unix 0x4687c0a280402721 0t0 /var/run/systemkeychaincheck.socket
launchd 1 root 21u unix 0x4687c0a288f454c9 0t0 /private/tmp/com.apple.launchd.QPPfbS2ajz/Listeners
launchd 1 root 22u unix 0x4687c0a280402591 0t0 /var/run/vpncontrol.sock
launchd 1 root 23u unix 0x4687c0a280402659 0t0 /var/run/portmap.socket
launchd 1 root 24u unix 0x4687c0a2804021a9 0t0 /private/var/run/.sim_diagnosticd_socket
launchd 1 root 25u unix 0x4687c0a288f45019 0t0 /var/tmp/filesystemui.socket
launchd 1 root 26u unix 0x4687c0a280402401 0t0 /private/var/run/cupsd
launchd 1 root 27u unix 0x4687c0a280402019 0t0 /var/rpc/ncacn_np/lsarpc
launchd 1 root 28u unix 0x4687c0a280401f51 0t0 /var/rpc/ncalrpc/lsarpc
launchd 1 root 29u unix 0x4687c0a280401e89 0t0 /var/rpc/ncacn_np/mdssvc
launchd 1 root 30u unix 0x4687c0a280401dc1 0t0 /var/rpc/ncacn_np/wkssvc
launchd 1 root 31u unix 0x4687c0a2804027e9 0t0 /var/rpc/ncalrpc/wkssvc
launchd 1 root 32u unix 0x4687c0a2804028b1 0t0 /var/rpc/ncacn_np/srvsvc
launchd 1 root 33u unix 0x4687c0a288f45019 0t0 /var/tmp/filesystemui.socket
launchd 1 root 34u unix 0x4687c0a2804024c9 0t0 /var/rpc/ncalrpc/NETLOGON
launchd 1 root 35u unix 0x4687c0a280401cf9 0t0 /var/rpc/ncalrpc/srvsvc
launchd 1 root 36u unix 0x4687c0a280401c31 0t0 /var/run/usbmuxd
launchd 1 root 37u unix 0x4687c0a2804028b1 0t0 /var/rpc/ncacn_np/srvsvc
launchd 1 root 38u unix 0x4687c0a280401cf9 0t0 /var/rpc/ncalrpc/srvsvc
launchd 1 root 39u unix 0x4687c0a280402979 0t0 /var/run/com.docker.vmnetd.sock
launchd 1 root 41r REG 1,4 358 17242395 /private/etc/security/audit_control
launchd 1 root 43r REG 1,4 652 17242393 /private/etc/security/audit_class

9

which one to stop. How can I know which one is launching the nomad?

10

@shoenig some assistance please.

11

Hi @ubaidilyas.

I believe you will need to attempt to find the plist associated with Nomad and then unload it, which may look like the following set of commands:

$ sudo ls -l /System/Library/* |grep nomad

$ sudo launchctl unload  /System/Library/<path_to_nomad_plist>

Thanks,
jrasell and the Nomad team.

12

Hello @jrasell,

Nothing was found using the

$ sudo ls -l /System/Library/* |grep nomad

whereas the nomad process is still there…

13

Just for the future readers, It was running under brew services. Stopped the service using brew stop and now its working fine.

2 Likes