Consul error :" memberlist: Was able to connect to <Container ID>.prod but other probes failed, network may be misconfigured"

Consul
1

Hello Team,

We are having an issue in Prod Setup with Vault and Consul. Here is the Setup.

We migrated the consul and vault from the system process to the docker container.
Consul & Vault running on servers 1,2,3 and Current consul version is 1.12.3 and vault 1.11.1

After migrating, all 3 vault and consul containers are up. Both are connecting with each other.

Consul can identify the leader node. The vault is unsealed and able to retrieve key values.

Here in consul if I check docker logs am seeing some errors related to “network” as below. FYI, PORTS all are enabled and the telenet works across the network.

Error:
2023-01-24T11:12:44.137Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:12:49.138Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:12:54.139Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:12:59.140Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:13:04.142Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:13:09.143Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:13:14.143Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:13:19.144Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured
2023-01-24T11:13:24.144Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to .prod but other probes failed, network may be misconfigured

For more information, please check the configuration as below :

consul config:

{
“server”: true,
“datacenter”: “PROD”,
“domain”: “consul”,
“bootstrap”: false,
“bootstrap_expect”: 2,
“encrypt”: “******************”,
“enable_script_checks”: true,
“data_dir”: “/consul/data”,
“log_level”: “INFO”,
“enable_syslog”: true,
“disable_update_check”: true,

“advertise_addr”: “IP3”,
“addresses”: {
“http”: “0.0.0.0”,
“dns”: “0.0.0.0”
},

“dns_config”: {
“only_passing”: true,
“service_ttl”: {
“*”: “2s”,
“web”: “3s”
}
},

“start_join”: [
“IP1”,
“IP2”,
“IP3”
],
“retry_join”: [
“IP1”,
“IP2”,
“IP3”
],

“leave_on_terminate”: true,
“rejoin_after_leave”: true,

“ports” : {
“dns” : 8600,
“http” : 8500,
“serf_lan” : 8301,
“serf_wan” : 8302,
“server” : 8300
},
“ui”: true
}

===================================================================
Vault config:
backend “consul” {
address = “local-consul:8500”
advertise_addr = “http://127.0.0.1:8200”
path = “vault”
scheme = “http”
}

api_addr = “http://IP3:8200”
cluster_addr = “https://vault.com”

listener “tcp” {
address = “0.0.0.0:8200”
tls_disable = 1
}

disable_mlock = true
ui = true

===================================================================

Docker compose file:

version: ‘2’
services:

consul:
container_name: “consul”
image: gitlab.com:6443/devops/vault_consul/consul:1.12.3-permi
command: “agent -config-file=/consul/config/config.json -client 0.0.0.0”
volumes:
- /var/lib/consul_docker_data:/consul/data:rw
- ./config/consul/config.json:/consul/config/config.json:rw
- /dev/log:/dev/log
ports:
- “8500:8500”
- “8300:8300”
- “8301:8301/tcp”
- “8302:8302/tcp”
- “8301:8301/udp”
- “8302:8302/udp”
- “8600:8600/tcp”
- “8600:8600/udp”
restart: always

vault:
container_name: “vault”
depends_on:
- consul
image: gitlab.com:6443/devops/vault_consul/vault:1.11.1
hostname: “vault”
links:
- “consul:consul”
environment:
VAULT_ADDR: http://IP3:8200
dns:
- IP3
extra_hosts:
- “local-consul:IP3”
ports:
- “8200:8200”
- “8201:8201”
volumes:
- ./tools/wait-for-it.sh:/wait-for-it.sh
- ./config/vault/config:/config
- ./config/vault/policies:/policies
- /dev/log:/dev/log
entrypoint: sh -x /wait-for-it.sh -t 20 -h consul -p 8500 -s – vault server -config=/config/with-consul.hcl
restart: always

=====
Here IP1, IP2 and IP3 are the IP address of 3 servers

Can someone kindly help me to resolve this issue.

2

Hi @CKShetty,

The error you are seeing is caused due to the agent not being able to talk to the other servers over port 8302/udp. This is the WAN Gossip port, and even if you are not using Consul WAN Federation, Consul will add all the server agents in a single cluster to its WAN pool.

For Gossip, Consul, by default, uses the UDP protocol and falls back to TCP when UDP doesn’t work. So in your case, TCP is working, and UDP is not for 8302.

You can get rid of this error by fixing the UDP connectivity.

I hope this helps.

1 Like
3

Hi @CKShetty @Ranjandas, I am facing same problem with consul server and agent. Can any of you help me how can I resolve this issue. Your help will be appreciated

4

Hi @anurag-appperfect,

Ensure all your Consul Server agent nodes can talk to each other on port 8302 UDP and TCP. In addition, all the servers and clients should be able to talk to each other on port 8301 UDP and TCP. This should fix the error that is reported in this topic.

5

@Ranjandas thank you for your answer