I’m deploying Vault on premises in Kubernetes using Raft and 3 replicas. Things were working fine until I had to delete the Kubernetes workers and recreate them (ClusterAPI so it’s technically a new node). The nodes weren’t deleted until I deleted the Vault pods on them because as I understand it, statefulset pods won’t be scheduled on other pods.
I’m trying to understand what is the recommended design for a situation like this? I want the nodes to be able to disappear and have Vault pods rescheduled, especially if there’s a normal eviction. Am I thinking about this the wrong way or is this how it should work and something went wrong in my situation?