Services are not available on "other" member

I have two physical machines. I run consul on both. They appear to know about each other
Node Address Status Type Build Protocol DC Segment
btaylor03 10.21.101.64:8301 alive server 1.8.0dev 2 lzlabs-dev
devlzonline01 10.21.101.69:8301 alive server 1.8.0dev 2 lzlabs-dev
I dynamically register a service on one machine(devlzonline01), but it is not visible on the other

[lzsystem@devlzonline01 ~]$ consul catalog services
LZLTE.EXCI.LTEPRODA
LZLTE.LTEPRODA.EXCITEST.TCP
LZLTE.LTEPRODA.GENLISNR.TCP
LZLTE.LTEPRODA.IPS1LSR.TCP
LZLTE.LTEPRODA.IPS4LSR.TCP
LZLTE.LTEPRODA.LTE3270.TCP
LZLTE.LTEPRODA.LTE3270.XIPC
LZLTE.LTEPRODA.LTEPRODD.TCP
LZLTE.LTEPRODA.LZ1ALSTR.TCP
catalogServices
consul
iosServices
sdmMetrics
smfServices
spoolServices

[lzsystem@btaylor03 ~]$ consul catalog services
RESTServices
catalogServices
consul
cpxExitServices
iosServices
rdbServices
sdmMetrics
smfServices
spoolServices

Hi @brad.taylor the command consul catalog services defaults to the local datacenter. It sounds like the other Consul server is a different Consul datacenter, so to see the services registered in another Consul datacenter use the CLI flag --datacenter=<name>

Example: consul catalog services --datacenter=dc2

No the data centers are the same. Is there a consul command I can issue that shows this?

@brad.taylor are you saying you only have two Consul servers and they are both in the same datacenter? Any reason for why you are not using the recommended number of 3? In order to achieve leader election, there needs to be an odd number of Consul server nodes.

Issue the command consul members to see the membership of the cluster. You should be seeing all participating members of the curent Consul datacenter.

I was able to get this to work, but I do not think it is what I want. I had to make 1 machine a “client”. I am just starting to set this up and am going slow. I would like to be able to have several machines in a “cluster”. I would like them to all know about each other. I am having some difficulty in understanding the various terms that seem to be used interchangeably but have similar meanings. I also can not find a reference document that explains these terms. There is the word “datacenter” which seems to mean “cluster”. Is that true.
Then there is the word “agent”. Is this a process on my linux machine? Can I have more than one of them on a machine. What is the difference between a “client” and a “server”? The tutorials are nice and show a way to accomplish what the tutorial is describing, but does not help me gain a deeper understanding on how to accomplish the various things. I will continue to look and post questions. The answers are encouraging.

Consul uses only one binary, which we call the agent. This binary can be configured to be a client or server. A server is is what manages the mesh, participates in leader election and keeps track of the service catalog. The service catalog contains all the services that are registered. It’s normal to have 3 Consul servers, the rest are clients. When you read the term Consul datacenter or Consul cluster, it refers to the collection of Consul server nodes. Each Consul cluster has a unique Consul datacenter id. So you can think of Consul cluster= Consul datacenter. The image below is an example of a Consul datacenter.

The Consul agent clients are what communicate with the server and help applications participate in the mesh. Consul clients, will leverage a sidecar proxy, such as Envoy to handle traffic between applications.

As for your question, can you have more than one on your machine. Technically yes, but it is not recommended. We prefer that each application is isolated to its own compute environment (VM/container). If you spin up multiple Consul agents in one machine you will have to address the issue of ports, as all Consul agents will default to the same ports unless configured differently.

Which tutorials did you start out with, if I may ask ?

@brad.taylor if you want to, here is a Docker compose example of an application called HashiCups deployed with Consul. This example uses two Consul datacenters. Feel free to spin it up and play with it. The README has all the directions needed to get started. I have a feeling seeing a real application will help your understanding of Consul :slight_smile: