Connect a VM to Consul running in Azure AKS

Consul
1

Hi,

I am new to Consul and I have some issues to connect Consul running in Azure AKS with a VM.

I have deployed Consul using the following Helm Chart hashicorp https://helm.releases.hashicorp.com

On the VM I have installed the version 1.8.0

From the consul server pods running in AKS, I have used the command consul join {ip address of the VM} and I get the message “Successfully joined cluster by contacting 1 nodes.”

I then checked the Consul dashboard and the VM which I added is showning as Healthy.

But,when I open the logs both in Azure and on the VM, I see lots of warnings and error messages:

Consul Server in AKS:

[WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-consul-server-1)

[ERROR] agent.server.rpc: failed to read byte: conn=from=11.233.0.1:13458 error=“read tcp 11.233.0.73:8300->11.233.0.1:13458: i/o timeout”

[WARN] agent.server.memberlist.lan: memberlist: Was able to connect to vmtest but other probes failed, network may be misconfigured

Consul on VM

[ERROR] agent.client: RPC failed to server: method=Catalog.NodeServiceList server=11.233.1.104:8300 error=“rpc error getting client: failed to get conn: dial tcp ->11.233.1.104:8300: i/o timeout”

[ERROR] agent.anti_entropy: failed to sync remote state: error=“rpc error getting client: failed to get conn: dial tcp ->11.233.1.104:8300: i/o timeout”

On the VM, I have allowed the following ports (for TCP and UDP / incoming and outgoing): 8300, 8301, 8302, 8500, 8600

Could you tell me what I am missing in my configuration?

Thanks in advance for time and attention,

Mike

2

Hi Mike,
When you ran consul join <ip> you were telling Consul that the VM and Kubernetes will have full connectivity between:

  • All Consul Server Pods on their advertise IPs (pod ips) and port 8300
  • All Consul client pods (the daemonset) and their advertise ips (pod ips) on port 8301
  • The VM on its advertise IP and port 8301

The error messages you’re seeing make me think that perhaps one of the above is not true?

Also the one message Was able to connect to vmtest but other probes failed, network may be misconfigured will be logged if UDP or TCP are working but not both.

3

Hi lkysow,

Thanks for getting back to me.
I actually solved our issue by exposing the hostPort of the servers as recommended by you in this post Communication between consul on local VM and consul in Kubernetes .

Thanks again and have a great week ahead,
Mike

4

Great to hear Mike! Let us know if you need anything else.