Consul Server outside Kubernetes Cluster

Consul
1

We are running Consul server outside our Kubernetes cluster. Upong deploying Consul client on Kubernetes the Server dashboard frequently updates the state of the Kubernetes nodes between “Agent live and reachable” and immediately shows “Agent not live or unreachable”. This is happening continuously in a loop.

externalServers:
enabled: true
hosts: [“****”]
httpsPort: 8501
tlsServerName: null
useSystemRoots: false
k8sAuthMethodHost: null
client:
enabled: true
image: null
join:
- *********
dataDirectoryHostPath: “”
grpc: true
exposeGossipPorts: true
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
fsGroup: 1000
extraVolumes:
tolerations: “”
nodeSelector: null
affinity: null
priorityClassName: “”
annotations: null
extraLabels: null
extraEnvironmentVars: {}
dnsPolicy: “ClusterFirst”
hostNetwork: false
updateStrategy: null

2

Hi @debjit.kar,

Could you validate that all the consul agents in your cluster (including server and client agents) are able to talk to Kubernetes Nodes on port 8301 (both UDP and TCP) and similarly the Kubernetes nodes are able to talk to all other Consul agents outside K8S cluster on port 8301 UDP and TCP? Refer to the following link for the network requirements and also how health check works in Consul.
(ref: Consul Reference Architecture | Consul - HashiCorp Learn)

To me, it sounds like your LAN gossip health checks are failing as some of the nodes in your cluster is not able to ping the K8S nodes and they are marking K8S nodes as failed.

If you look at the logs from Consul nodes you should be able to find out why the nodes are marked as unreachable.

3

Agreed, also see Consul Clients Outside of Kubernetes - Kubernetes | Consul by HashiCorp

4

Thanks @Ranjandas. It is working fine now.

1 Like