Is the health state returned by `/v1/health/checks/<service-name>` the most updated at system boot up?

Consul
1

Hi, I’m making this post to confirm whether my understanding is correct. I’m using Consul 1.16.1.

In my application, I actually have only one node which runs several functional services as Docker containers. These containers are registered to Consul by the Registrator service.

I’m writing another service that does this: When the system is booted up, poll Consul to get the health states of all the other functional services. When all the functional services become healthy, create a special file.

Right now, I’m calling /v1/health/checks/<service-name> to get the health check states of each service in order to determine if the service is already healthy. However, it seems that the health state that’s returned by this endpoint is stale and it may return the health state from the previous run.

By reading Consul’s log, I noticed the log messages such as “Synced check”. It then occurred to me that Consul is deployed in a distributed structure. Even if it’s just one single node, there are still the two roles: server and agent. Then I found the post by Blake here which talks about the difference between the HTTP endpoint for the server and the HTTP endpoint for the agent. It seems that I have been using the endpoint that returns the information on the server instead of the agent, but it may take a bit time for the information to be synchronized from the agent to the server.

So am I correct that:

  1. It may take some nondeterministic period of time for the Consul local agent to synchronize the service/health check info to the Consul server, so before the information is synchronized, the info on the Consul server can be stale.
  2. If I want to get the updated health state of the functional services running on my node, I should probably call the HTTP endpoint for the agent which is /v1/agent/health/service/<service-name>.
  3. Is the synchronization between the Consul server and the local agent(s) really nondeterministic? Is there a way to configure the synchronization? (I read somewhere that Consul uses gossip protocol to synchronize data so it’s not deterministic when that would happen, but I wonder if there is any way.)
2

I realized my question post may cause some confusion. Here is my clarification:

  • I only have one node.
  • I only have one Consul instance that’s configured as a “server”.
  • I only observed that Consul returned me stale health check results when Consul was started up. And after a short period of time, Consul would start to return updated information.
  • In Consul logs, I saw messages like “synced check” and “synced service”. I’m aware that Consul is usually deployed in a cluster and has more than one instance (with one being the server and others the clients). But in my case, I only have one Consul instance configured as the “server”, so I thought the information should be always updated. But the “Synced check” messages seemed to suggest there was some internal synchronization going on, and, if my observation was correct, I wonder if it was why I could get some stale information.