Vault HA Cluster raft with keepalived

Vault
1

I have a pre-production setup on Hetzner, with Vault HA Cluster (4 nodes setup) with Vault 1 being Unseal Node and 3 Cluster nodes. I followed the Hashicorp development tutorial with Vault Ha Cluster Raft integrated storage and all works perfectly. I even included TLS with self-signed SSL with SANs I also included keepalived to make Cluster IP, which I pointed to api_addr which now acts as Cluster access, which resolved to keepalived virtual IP.

This cluster.sh script, created Vault_1 with inmemory storage, and Cluster nodes with Raft storage. So my question is, after I restart all VMs, How I am supposed to restart the whole thing? I’ve spend last day trying to come up with the solution or do i need to change storage type of the first “unseal” node. What is the best way to make this setup works and use it later in small production.

2

This architecture doesn’t sound safe at all to me.

I’m assuming by “unseal” node, you mean you have a single node “cluster”, that the other cluster is using for transit auto-unseal.

Don’t do this. If you lose the single node acting as the auto-unseal device, your entire cluster is doomed and unrecoverable.

3

You are correct. Single Node is used for Unseal for transit Auto-unseal. I went over this tutorial as I provided and yes it works, until it doesn’t (when Unseal node IN MEMORY one is restarted) because Vault_1 is not working I cannot unseal my Cluster. As I mentioned, this is TEST environment, but I still need to have Vault_1 (Unseal node better configured) so that i can unseal Cluster even when all VMs restart. So what do you suggest? like Unseal node have different kind of storage?

On the production, i was planning on using AWS KEY Management for the unseal…is that ok?

Thank you for your answer!

4

Oh, OK. If it’s strictly for testing and will never contain valuable secrets, then just switching the in-memory storage to a persistent storage type of your preference would be fine. The file storage type is the simplest.

AWS KMS is… well, is it acceptable to you that your Vault data is entirely dependent on that key remaining accessible, forever?

Even if you make a backup of your Vault data outside of AWS, it will be forever unreadable if you don’t have access to the AWS KMS key.

A quorum of Vault recovery keys will NOT, repeat NOT, allow you to recover a backup without the AWS KMS key!

Depending on what data you are storing in Vault, that might be a desirable security property, or a fatal flaw.