Hi @pekuz!
Welcome to the forums, and thank you for your first post!
Let’s walk through your questions a bit.
What are pros and cons of connecting directly to a remote server agent versus via a local client agent, please?
Consul, in general, will always send the request to the client, and that is then forwarded to the appropriate servers. This intentional design distributes the request/read load, handles resolution for the request during leader elections or other changes in the cluster, and stops single point failures during the request path. This also allows the application to cache responses for better performance and lower network traffic, as @Wolfsrudel pointed out in his snippet.
From the Architecture Doc that @Wolfsrudel linked;
It is expected that there be between three to five servers. This strikes a balance between availability in the case of failure and performance …
I also recommend you take a look at the Gossip Protocol documentation, as this describes the libraries used to distribute communications and handle failure scenarios.
Hopefully these help cover the pros of using this set up.
As a general rule, there shouldn’t be any reason to connect directly to the server, except for restoring from a snapshot as an operator.
While you can design your application to read in any server IP at start up, that doesn’t mean that that server is even available, let alone able be the leader. This is where Consul agents work well, because you can have your application connect to any agent (client OR server) for initial start up. Because of this, we recommend launching your app with a locally available agent.
Just for some clarification;
I just guess that the consul tool REST-calls the agent client and the agent client REST-calls the agent server.
I just wanted to clarify the request path. Request goes to client agent, which forwards request to a server agent, which forwards the request to the leader agent.
So, for your second question;
how to discover agent servers, please?
To reiterate, we discourage connecting your application directly to the server agents directly. As part of our bootstrapping guide, we describe how you can connect to any Consul node to facilitate connections.
Clients are much easier as they can join against any existing node. All nodes participate in a gossip protocol to perform basic discovery, so once joined to any member of the cluster, new clients will automatically find the servers and register themselves.
Finally, I want to address your message about performance.
If the library REST-calls the agent client and the agent client REST-calls the agent server it would be a performance hit comparing to directly calling a remote agent server but again lets hope the library smartly avoids that after the discovery phase.
While it is true that there is additional “hops” for your application to communicate with the server, consider that your application is part of a larger ecosystem. The additional hops give you them benefits of caching of data, depending on your consistency mode. If your application can handle stale data, then you can gain a lot more benefits from querying your local client rather than connecting to the server.
Thanks for all your questions! I know there is a lot to unpack here, but I hope this helps!
Best,
Jono