Consul auto recovery in case of failures

In our case, we are using Consul only for service discovery. We might need to create some ACLs, tokens and policies to ensure some level of security. We are using docker containers.

Consider we have a cluster of 3 agents,

  1. Consider a situation we don’t have –data-dir

a. If one/two of agent goes down for some reason and comes up, will it be in the same state of what it was previously? I mean and tokens/policies are retained as it is?

  1. Consider a situation we have –data-dir which is a docker volume which is mounted from hosting machine.

a. If one/two of agent goes down for some reason and comes up, will it be in the same state of what it was previously? I mean and tokens/policies are retained as it is?

  1. If the complete cluster goes down, is there a way we can bring up the complete cluster automatically without any human intervention or it needs to be created manually? Considering we have the snapshot stored in S3.

Hi @klnlakshmi,

I would not recommend operating a cluster in the first scenario where you are not providing persistent storage.

In the second case where you are providing a persistent volume, your data will be preserved upon restarts of the containers. It is still recommend to take periodic snapshots using consul snapshot save or GET /snapshot HTTP API.

If the entire cluster is destroyed, you can restore the data by re-creating the server agents and then loading the snapshot from backup using consul snapshot restore or PUT /snapshot via the HTTP API.

Are you able to restore the snapshot to the newly enabled engine.
I am not able to restore the secrets from the snapshot to the newly enabled vault secret kv/ backend.
If i backup and restore to the same secret engine with the same accessor id, it works but it doesn’t work when everything is created from scratch and you are using the previously taken snapshot to restore secrets to this new vault consul stack (even if path and secret engine you are restoring to is same)