Consul service stuck in activating mode

Consul
1

Hello Guys,

I’m trying to deploy a single-node consul server in my testing environment (multipass on MacOS) but it is stuck in activating state while the server is running.

Here is my consul.hcl file:

datacenter = "dc-1"
data_dir = "/opt/consul"
client_addr = "0.0.0.0"
ui_config{
  enabled = true
}
server = true
advertise_addr = "{{ GetInterfaceIP \"enp0s1\" }}"
acl {
  enabled = true
  enable_token_persistence = true
}

Here is the journal output

Aug 14 15:50:58 accurate-crake consul[21294]: ==> Starting Consul agent...
Aug 14 15:50:58 accurate-crake consul[21294]:                Version: '1.16.1'
Aug 14 15:50:58 accurate-crake consul[21294]:             Build Date: '2023-08-05 21:56:29 +0000 UTC'
Aug 14 15:50:58 accurate-crake consul[21294]:                Node ID: '5a55a11d-197f-4ef1-4e43-6b6da89542e4'
Aug 14 15:50:58 accurate-crake consul[21294]:              Node name: 'accurate-crake'
Aug 14 15:50:58 accurate-crake consul[21294]:             Datacenter: 'dc-1' (Segment: '<all>')
Aug 14 15:50:58 accurate-crake consul[21294]:                 Server: true (Bootstrap: false)
Aug 14 15:50:58 accurate-crake consul[21294]:            Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: >
Aug 14 15:50:58 accurate-crake consul[21294]:           Cluster Addr: 192.168.64.14 (LAN: 8301, WAN: 8302)
Aug 14 15:50:58 accurate-crake consul[21294]:      Gossip Encryption: false
Aug 14 15:50:58 accurate-crake consul[21294]:       Auto-Encrypt-TLS: false
Aug 14 15:50:58 accurate-crake consul[21294]:            ACL Enabled: true
Aug 14 15:50:58 accurate-crake consul[21294]:      Reporting Enabled: false
Aug 14 15:50:58 accurate-crake consul[21294]:     ACL Default Policy: allow
Aug 14 15:50:58 accurate-crake consul[21294]:              HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Ve>
Aug 14 15:50:58 accurate-crake consul[21294]:               gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Aug 14 15:50:58 accurate-crake consul[21294]:       Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify>
Aug 14 15:50:58 accurate-crake consul[21294]: ==> Log data will now stream in as it occurs:
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.737+0200 [WARN]  agent: skipping file /etc/consul.d/consul>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.741+0200 [WARN]  agent.auto_config: skipping file /etc/con>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.753+0200 [INFO]  agent.server.raft: initial configuration:>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.754+0200 [INFO]  agent.server.serf.wan: serf: EventMemberJ>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.754+0200 [INFO]  agent.server.serf.lan: serf: EventMemberJ>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.754+0200 [INFO]  agent.router: Initializing LAN area manag>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.754+0200 [INFO]  agent.server.raft: entering follower stat>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.754+0200 [WARN]  agent.server.serf.wan: serf: Failed to re>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.754+0200 [WARN]  agent.server.serf.lan: serf: Failed to re>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.755+0200 [INFO]  agent.server: Adding LAN server: server=">
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.755+0200 [INFO]  agent.server.autopilot: reconciliation no>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.755+0200 [INFO]  agent.server: Handled event for server in>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.755+0200 [INFO]  agent.server.cert-manager: initialized se>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.756+0200 [INFO]  agent: Started DNS server: address=0.0.0.>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.756+0200 [INFO]  agent: Started DNS server: address=0.0.0.>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.756+0200 [INFO]  agent: Starting server: address=[::]:8500>
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.756+0200 [INFO]  agent: started state syncer
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.756+0200 [INFO]  agent: Consul agent running!
Aug 14 15:50:58 accurate-crake consul[21294]: 2023-08-14T15:50:58.757+0200 [INFO]  agent: Started gRPC listeners: port_name=>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.328+0200 [WARN]  agent.server.raft: heartbeat timeout reac>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.330+0200 [INFO]  agent.server.raft: entering candidate sta>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.336+0200 [INFO]  agent.server.raft: election won: term=212>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.336+0200 [INFO]  agent.server.raft: entering leader state:>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.337+0200 [INFO]  agent.server: cluster leadership acquired
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.337+0200 [INFO]  agent.server: New leader elected: payload>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.398+0200 [INFO]  agent.server: initializing acls
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="a>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.server.autopilot: reconciliation no>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="f>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="f>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="s>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="m>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="p>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  connect.ca: initialized primary datacente>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="i>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="C>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="C>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="C>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="v>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.399+0200 [INFO]  agent.leader: started routine: routine="c>
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.400+0200 [INFO]  agent.leader: stopping routine: routine=">
Aug 14 15:51:05 accurate-crake consul[21294]: 2023-08-14T15:51:05.400+0200 [INFO]  agent.leader: stopped routine: routine="v>
Aug 14 15:51:08 accurate-crake consul[21294]: 2023-08-14T15:51:08.015+0200 [INFO]  agent: Synced node info

And the service status

● consul.service - "HashiCorp Consul - A service mesh solution"
     Loaded: loaded (/lib/systemd/system/consul.service; disabled; vendor preset: enabled)
     Active: activating (start) since Mon 2023-08-14 15:52:34 CEST; 10s ago
       Docs: https://www.consul.io/
   Main PID: 21339 (consul)
      Tasks: 7 (limit: 4624)
     Memory: 30.8M
     CGroup: /system.slice/consul.service
             └─21339 /usr/bin/consul agent -config-dir=/etc/consul.d/

Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.759+0200 [INFO]  agent.leader: started routine: routine="i>
Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.759+0200 [INFO]  agent.leader: started routine: routine="C>
Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.759+0200 [INFO]  agent.leader: started routine: routine="C>
Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.759+0200 [INFO]  agent.leader: started routine: routine="C>
Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.759+0200 [INFO]  agent.leader: started routine: routine="v>
Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.759+0200 [INFO]  agent.leader: started routine: routine="c>
Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.760+0200 [INFO]  agent.leader: stopping routine: routine=">
Aug 14 15:52:42 accurate-crake consul[21339]: 2023-08-14T15:52:42.760+0200 [INFO]  agent.leader: stopped routine: routine="v>
Aug 14 15:52:44 accurate-crake consul[21339]: 2023-08-14T15:52:44.216+0200 [ERROR] agent.server.autopilot: Failed to reconci>
Aug 14 15:52:45 accurate-crake consul[21339]: 2023-08-14T15:52:45.068+0200 [INFO]  agent: Synced node info
2

Hi @mrxdane,

Welcome to the HashiCorp Forums!

It looks like the systemd unit file has been recently changed to type notify, which previously was simple.

According to the documentation, Consul would only notify systemd when a LAN join has completed.

When running under systemd on Linux, Consul notifies systemd by sending READY=1 to the $NOTIFY_SOCKET when a LAN join has completed. For this either the join or retry_join option has to be set and the service definition file has to have Type=notify set.
ref: Agents Overview | Consul | HashiCorp Developer

You can get it fixed by adding retry_join or join in the config file and setting the IP to the bind address of Consul on the host and restarting the agent. For example, in your case:

retry_join = ["{{ GetInterfaceIP \"enp0s1\" }}"]

The alternate option is to change the Unit file type to simple by editing it.

ref:

I hope this helps.

3

Thank you very much Ranjandas, the service started successfully.