I need to use consul only for service discovery, health checking, alerts etc. In which mode should I host consul.
Client
Server
Both
Currently, 2 nodes are enough for me to host consul and as per load, I will increase the count in future.
We are already using a 8 node cluster of hashicorp vault and consul where we’ve hosted a combination of consul server and client. But, we followed this approach because we have used consul server as backend storage for vault.
How to manage in this use case where the sole purpose of using consul is the management of other services. If possible, please share the consul configuration scripts as well.
You need at least three Consul servers for the Raft protocol to reach a quorum and have high availability. This way, one node can shortly go down without impacting your Consul setup. You don’t gain anything by running only two Consul nodes over a single node.
A Raft cluster of 3 nodes can tolerate a single node failure while a cluster of 5 can tolerate 2 node failures. The recommended configuration is to either run 3 or 5 Consul servers per datacenter. This maximizes availability without greatly sacrificing performance.
@fhemberger , Great. Thanks a lot for the details. One last query. If I host 3 nodes then at random one will become a leader.
Now, for integrating any service do we need to use only the leader node of the consul. Or, can we access the UI or shell prompt from any node and infuse the service details.
Because I remember, while hosting hashicorp vault consul cluster 4 years ago I was stuck while enabling acl’s. At that time I get to know, for any administrative work, we can run commands via leader node only.
@smartaquarius10, sorry, I don’t know off the top of my head if there are any ACL/leader node limitations. If you find something, it would be good to know.
@smartaquarius10, there are few different ways in which you can register services to Consul. You could choose the one that best suits your need. Having said that, one usual pattern is to run Consul Client Agent on the node where your service is running and register the service on that node via the local Consul agent.
By doing this, you are registering both the node and the service to the Consul catalog. If the node health check fails for some reason, Consul will stop advertising the services running on the node to Service Discovery queries. (You should be familiar with this pattern as you are already running a Vault + Consul setup).
Regarding your question on whether you should talk directly to the Leader node, No you don’t have to. In Consul all the Consul Agents will forward the RPC request it receives to the Leader node. You will get to read about it here: Consul Architecture | Consul by HashiCorp
Yes, you definitely need Consul Server agents (unless you are planning to reuse the Consul used with your Vault Infrastructure) and a minimum of 3 nodes for high availability. It’s the servers that store the Service Discovery Catalog (using Raft protocol).
Just to make it clear, agent is the term used for the Consul process running on a node (irrespective of client or server). Ref: Agent | Consul by HashiCorp
Additionally, there is first-class support for services hosted in Kubernetes using Consul connect if you are looking for a full-service mesh solution that spans across infrastructure, deployment types, etc.
Please have a look at the reference architectures linked below
Non-server agents run in client mode. A client is a lightweight process that registers services, runs health checks, and forwards queries to servers. A client must be running on every node in the Consul datacenter that runs services, since clients are the source of truth about service health.
Do I really need consul client. If I have my rest api’s are in azure as app service then can register them directly into consul server and setup alerts. Or, do I need a separate node of consul client for health checking and monitoring. My use case is a small scale so can I avoid using consul client completely.
If you are deploying services on VM’s that you have control of, or on Kubernetes (or other container stacks), having a Consul client is the right way to go.
I don’t know how much work would be involved and would it be worth the effort in the case of Azure App Services. I think the only easy option if at all you want to register the services into Consul will be as External Services Consul and External Services. But then whenever the service moves to a different IP etc, you will have to update the registration.
@Ranjandas , Its not of IP. I have my url. So I’ll add that url into it rather than using any IP. URL will not change that is confirmed.
Could you please tell me one more thing. In the url you’ve shared this line is written
Consul uses a push-based model where agents send events only upon status changes. As such, even large clusters will have very few requests.
Is it referring to consul client + consul server mechanism where client will be sending updates to server. But, in my case it will always be pull based. Am I correct here.