HI,
I’m thinking to have immutables Vault Virtuals Machines (Packer is my friend !).
But the question is: is it easy to “replace” a Vault node in a raft cluster ?
What do you think about this idea ?
Provided you have an orchestrator that ensures nodes will only be taken offline in cases where the quorum will not be compromised, and provide a way for new nodes to automatically join the cluster, I don’t see a problem with this.
But, do expect to have to build the automation around making this safe.
You’ll need to inspect the current Raft peer set: https://www.vaultproject.io/api-docs/system/storage/raft#read-raft-configuration
I think that node IP addresses are baked in to the Raft peer set, and nodes need to be explcitly removed and re-joined when changing IP.
You’ll need to be aware that old nodes need to be removed from the raft peer set either explicitly https://www.vaultproject.io/api-docs/system/storage/raft#remove-a-node-from-raft-cluster or via the dead server handling in the autopilot subsystem: https://www.vaultproject.io/api-docs/system/storage/raftautopilot#set-configuration
Nodes joining a cluster need:
- If using auto-unseal, proper configuration to use the auto-unseal method
- If using Shamir seal, a quorum of key-holders to provide their unseal keys to the node
And lastly … test extensively before trusting node replacement to run automatically on a production cluster.
Thank you for all those informations. I now know which points to pay the most attention to.
I use an orchestrator and I have already fully automated the deployment of a Vault + auto-unseal cluster. So, I think I can build a tasks to automate node remplacement like you explain.
I also use Packer to create Linux templates. I can easily create a chain for this template to become a Vault template.
Thanks you !