When deploying Consul servers within Kubernetes, it’s possible to use a StatefulSet to give the pods stable identity in DNS, and associate external storage with each pod. I’m wondering whether the files Consul writes to the path specified by the
data_dir configuration field need to survive a crash of the server container, and whether they should outlast a given server pod to be used by a replacement pod later.
If the files are just a local optimization, we can store them in an “emptyDir” Kubernetes volume, and let them die with the pod. If instead the files are critical to Consul’s operation and we need to make them available for replacement servers to adopt, we can store them in a bound PersistentVolume in, say, AWS EC2 EBS volumes. That does mean, though, that the storage is zonally confined, mandating the a replacement for each Consul server come back within the same availability zone as its predecessor.
I see raft control files in that
data_dir directory, and other things I don’t understand yet. Are they precious? Please advise.