Consul agent nodes won't start on my Raspberry Pi 4s

Consul
1

I’m using my home lab for an online Consul tutorial. The agents get “consul.service: Failed with result ‘resources’”. I’m not sure where to get more debugging information. What else should I look at to debug what is happening?

The master nodes start successfully and I can see them on the Consul Dashboard. The servers used a package install on 3 Ubuntu 8GB and 16GB MacMinis. The 3 agent nodes are ARM64 binary installs on Ubuntu 8GB Raspberry Pi 4s. Running consul agent -dev seems to work and consul validate /etc/consul.d/consul.hcl responds “Configuration is valid!”. When I try to start normally, I get this error:

ubuntu@lab:~$ sudo systemctl start consul
Job for consul.service failed because of unavailable resources or another system error.
See "systemctl status consul.service" and "journalctl -xe" for details.

ubuntu@lab:~$ systemctl status consul.service
consul.service - "HashiCorp Consul - A service mesh solution"
Loaded: loaded (/lib/systemd/system/consul.service; enabled; vendor preset: enabled)
Active: failed (Result: resources) since Sun 2021-08-01 12:24:03 UTC; 2min 29s ago
Docs: https://www.consul.io/

Aug 01 12:24:03 lab.byteworksinc.com systemd[1]: consul.service: Scheduled restart job, restart counter is at 5.
Aug 01 12:24:03 lab.byteworksinc.com systemd[1]: Stopped "HashiCorp Consul - A service mesh solution".
Aug 01 12:24:03 lab.byteworksinc.com systemd[1]: consul.service: Start request repeated too quickly.
Aug 01 12:24:03 lab.byteworksinc.com systemd[1]: consul.service: Failed with result 'resources'.
Aug 01 12:24:03 lab.byteworksinc.com systemd[1]: Failed to start "HashiCorp Consul - A service mesh solution".

Here is an agent config:

node_name = "homelab-client-0"
datacenter = "homelab"
data_dir = "/opt/consul"
client_addr = "192.168.1.80"
advertise_addr = "192.168.1.80"
server = false
encrypt = "[REDACTED]"
retry_join = ["192.168.1.4", "192.168.1.5", "192.168.1.8"]
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/homelab-client-consul-0.pem"
key_file = "/etc/consul.d/homelab-client-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
log_level = "DEBUG"
enable_syslog = true

Here is a server Config:

node_name = "homelab-server-0"
datacenter = "homelab"
data_dir = "/opt/consul"
ui_config{
  enabled = true
}

server = true
client_addr = "192.168.1.4"
advertise_addr = "192.168.1.4"
bootstrap_expect=3
encrypt = "[REDACTED]"
retry_join = ["192.168.1.4", "192.168.1.5", "192.168.1.8"]
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/homelab-server-consul-0.pem"
key_file = "/etc/consul.d/homelab-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
2

Hi @smitchell, apologies for the very late reply. Have you since resolved this issue? If not, a quick search indicates this is caused by a permissions issue on the /tmp or /var/tmp directories.

https://forums.raspberrypi.com/viewtopic.php?t=249619

I recommend modifying the /tmp permissions as suggested in these other posts to see if it resolves the error you are receiving.