How to verify that consul started properly

Consul
1

I am deploying consul to AWS EC2 instance that has resolvers in /etc/resolv.conf.

The consul agent will recurse to these resolvers, and I would like to update /etc/resolv.conf to point to consul after it has started, ie. remove the original nameservers and then add in

nameserver 127.0.0.1

However, since the consul command runs forever if it succeeds, I am not sure how to tell when to update the /etc/resolv.conf to point to consul.

Any suggestions?

2

Looks like I was setting up the systemd unit file in a way that was causing it to be in an “Activating” state forever.

Though, the command “consul members” displayed that the node was part of the datacenter. Thus it seems that consul had started properly. The following was causing “systemd start consul.service” to block indefinitely.

[Service]
…
Type=notify
ExecStart=usr/bin/sudo /usr/local/bin/consul agent

When I changed type to simple:
Type=simple

or stopped using Sudo:
ExecStart= /usr/local/bin/consul agent

The command “systemd start consul.service” does not hang forever in “Activating” state.

3

Hi @munali,

When type=notify is used, Consul will wait until it has properly joined one of the agents specified in either join or retry_join before signaling to systemd that the process has successfully started (this is mentioned in Consul Deployment Guide: Configure systemd).

I’m not sure why the signal was sent in your environment. Changing the ExecStart type to simple or exec as you have done is a viable workaround. However, it is probably still worth investigating why the notify did not properly reach systemd.

1 Like
4

Hi @blake,

If I remove the sudo bit from ExecStart, and keep “Notify” it still works. So I think it has something to do with the sudo command. But I am not sure why. Any ideas how to debug furthur?

e.g. This works:
[Service]
…
Type=notify
ExecStart=/usr/local/bin/consul agent

5

@munali,

I suspected sudo might be causing issues here. If your intention is run Consul as the root user, then you do not actually need to use sudo. By default systemd will execute processes as the root user.

If you need to run Consul as a specific user and/or group, you can use systemd’s User= or Group= directives.

6

@blake,

I was able to resolve this by removing the “sudo” in the ExecStart and instead enabling the capabilities I needed:

AmbientCapabilities=CAP_NET_BIND_SERVICE

1 Like