Consul auto recovery in case of failures

In our case, we are using Consul only for service discovery. We might need to create some ACLs, tokens and policies to ensure some level of security. We are using docker containers.

Consider we have a cluster of 3 agents,

  1. Consider a situation we don’t have –data-dir

a. If one/two of agent goes down for some reason and comes up, will it be in the same state of what it was previously? I mean and tokens/policies are retained as it is?

  1. Consider a situation we have –data-dir which is a docker volume which is mounted from hosting machine.

a. If one/two of agent goes down for some reason and comes up, will it be in the same state of what it was previously? I mean and tokens/policies are retained as it is?

  1. If the complete cluster goes down, is there a way we can bring up the complete cluster automatically without any human intervention or it needs to be created manually? Considering we have the snapshot stored in S3.

Hi @klnlakshmi,

I would not recommend operating a cluster in the first scenario where you are not providing persistent storage.

In the second case where you are providing a persistent volume, your data will be preserved upon restarts of the containers. It is still recommend to take periodic snapshots using consul snapshot save or GET /snapshot HTTP API.

If the entire cluster is destroyed, you can restore the data by re-creating the server agents and then loading the snapshot from backup using consul snapshot restore or PUT /snapshot via the HTTP API.