Hi Steven,
It is dangerous to run Consul servers without durable storage however it is possible to do so. The Consul servers can be thought of as the database for Consul. They duplicate the data across each server, so as long as you have quorum, your data won’t be lost. But if you lose enough servers that there’s no quorum, you’re in trouble and could lose all your data. You’d then need to restore from backup.
Some more notes from asking the team:
If one server crashes and loses its data, the other servers will continue to function and when the crashed server restarts, it will get sent the data it lost. So on the face of it, as long as you don’t lose more than one server (assuming you have 3) then you don’t need durable storage.
However if you lose more than one server or there are network partitions then Raft’s (the consensus algorithm) guarantees no longer hold - it’s possible for a restarting server combined with a network partition to cause a split brain, lose committed writes or otherwise violate linearizability.
We also have a Helm chart for deploying on Kubernetes in case you haven’t seen it: GitHub - hashicorp/consul-helm: Helm chart to install Consul and other associated components..
To your question about zones, yes the server will need to come back up in the same zone. Although due to the durability requirements outlined above, it’s best to spread your servers out by zone anyway.