Immutable VM + raft


I’m thinking to have immutables Vault Virtuals Machines (Packer is my friend !).
But the question is: is it easy to “replace” a Vault node in a raft cluster ?

What do you think about this idea ?

Provided you have an orchestrator that ensures nodes will only be taken offline in cases where the quorum will not be compromised, and provide a way for new nodes to automatically join the cluster, I don’t see a problem with this.

But, do expect to have to build the automation around making this safe.

You’ll need to inspect the current Raft peer set:

I think that node IP addresses are baked in to the Raft peer set, and nodes need to be explcitly removed and re-joined when changing IP.

You’ll need to be aware that old nodes need to be removed from the raft peer set either explicitly or via the dead server handling in the autopilot subsystem:

Nodes joining a cluster need:

  • If using auto-unseal, proper configuration to use the auto-unseal method
  • If using Shamir seal, a quorum of key-holders to provide their unseal keys to the node

And lastly … test extensively before trusting node replacement to run automatically on a production cluster.

Thank you for all those informations. I now know which points to pay the most attention to.
I use an orchestrator and I have already fully automated the deployment of a Vault + auto-unseal cluster. So, I think I can build a tasks to automate node remplacement like you explain.
I also use Packer to create Linux templates. I can easily create a chain for this template to become a Vault template.
Thanks you !