Unable to start consul service status=203

i am trying to install consul on a single node deployment but i am unable to start the service after following the steps mentioned at https://learn.hashicorp.com/tutorials/consul/deployment-guide?in=consul/datacenter-deploy#setup-consul-environment-variables

this is my /etc/consul.d/consul.hcl:

datacenter = “main-dc”

data_dir = “/opt/consul”

encrypt = “RluMloPrB04sYOk/952AhpXLAJO7aGSfSCkbGIsyTlk=”

ca_file = “/etc/consul.d/consul-agent-ca.pem”

cert_file = “/etc/consul.d/main-dc-server-consul-0.pem”

key_file = “/etc/consul.d/main-dc-server-consul-0-key.pem”

verify_incoming = true

verify_outgoing = true

verify_server_hostname = true

retry_join = [“dns-server-ip”]

acl = {

enabled = true

default_policy = “allow”

enable_token_persistence = true

}

performance {

raft_multiplier = 1

}

bind_addr = [“main-node-server”]

ports = {

dns = 53

}

this is my /etc/consul.d/server.hcl

server = true

bootstrap_expect = 1

ui = true

client_addr = “127.0.0.1”

and this is systemctl output:

node ~]# systemctl status consul

consul.service - “HashiCorp Consul - A service mesh solution”

Loaded: loaded (/etc/systemd/system/consul.service; enabled; vendor preset: disabled)

Active: failed (Result: exit-code) since Thu 2020-08-20 18:14:44 UTC; 4min 10s ago

Docs: https://www.consul.io/

Process: 12488 ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ (code=exited, status=203/EXEC)

Main PID: 12488 (code=exited, status=203/EXEC)

Aug 20 18:14:44 ovirt-node systemd[1]: consul.service: Service RestartSec=100ms expired, scheduling restart.

Aug 20 18:14:44 ovirt-node systemd[1]: consul.service: Scheduled restart job, restart counter is at 5.

Aug 20 18:14:44 ovirt-node systemd[1]: Stopped “HashiCorp Consul - A service mesh solution”.

Aug 20 18:14:44 ovirt-node systemd[1]: consul.service: Start request repeated too quickly.

Aug 20 18:14:44 ovirt-node systemd[1]: consul.service: Failed with result ‘exit-code’.

Aug 20 18:14:44 ovirt-node systemd[1]: Failed to start “HashiCorp Consul - A service mesh solution”.

when i run node ~]# /usr/local/bin/consul agent -config-dir=/etc/consul.d/ I can see this message showing up:

2020-08-20T18:24:59.754Z [INFO]  agent: started state syncer

==> Consul agent running!
2020-08-20T18:24:59.754Z [WARN] agent: (LAN) couldn’t join: number_of_nodes=0 error="1 error occurred:

  • Failed to join dns-server-ip: dial tcp dns-server-ip:8301: connect: no route to host

it does not check for port 53 even if declared and if i add :53 to the retry join address directly is can see this with the same command:

2020-08-20T18:29:41.071Z [WARN] agent: (LAN) couldn’t join: number_of_nodes=0 error="1 error occurred:

  • Failed to join dns-server-ip: read tcp :52360->dns-server-ip:53: i/o timeout

on my dns server i can see the port:

dnsmasq ~]# grep 53 /etc/services -w
domain 53/tcp # name-domain server
domain 53/udp

I simply followed the steps on a centos 8 fresh install, what is missing here?
ask me for any other needed info, thanks!

node ~]# cat /etc/release

CentOS Linux release 8.2.2004 (Core)
Derived from Red Hat Enterprise Linux 8.2 (Source)
NAME=“CentOS Linux”
VERSION=“8 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“8”
VARIANT=“oVirt Node 4.4.1.5”
VARIANT_ID=“ovirt-node”
PRETTY_NAME=“oVirt Node 4.4.1”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:8”
HOME_URL=“https://www.ovirt.org/
BUG_REPORT_URL=“https://bugzilla.redhat.com/
PLATFORM_ID=“platform:el8”
CentOS Linux release 8.2.2004 (Core)
CentOS Linux release 8.2.2004 (Core)
cpe:/o:centos:centos:8

Hi @servicelevel,

The IPs or hostnames listed in join or retry_join should be addresses of other Consul agents (clients or servers) within the environment. The error “no route to host” indicates that the operating system does not have a route to the network on which that host exists.

Try configuring an address of another Consul agent in join / retry_join and ensure that subnet is accessible from this Consul host. Let us know if you are still experiencing issues.