First time deployer - Need architecture rollout sanity check for bootstrapping Vault and entire architecture

Vault
1

I’m a fairly seasoned computer guy but this is my first time deploying Hashicorp products, although I’ve admired them from afar :slight_smile: I would like a few minutes of your time to review my architecture rollout plan and poke holes in it or give me advice on the Hashicorp pieces, with Vault being the first.

Scenario

New everything, switches, servers, etc. and the requirement to automatically bootstrap everything as much as possible. This will start with a single server running Debian Trixie and plugged into the network - I’ll call this the control station. This station will run a combination of Ansible, Vault, Packer, Terraform, and Ubuntu Mass to deploy a Proxmox cluster. I’ll give more detail about the plan a little later but start with the summary questions in case you don’t want to read a lot.

Before a single machine is deployed, a lot of sensitive information needs to be generated and stored so I plan to spin up a single instances of Vault on the control station, and have Ansible and the various tools store info there. Also will use Vault to create PKI certs and be the root and intermediate CA. After the cluster is created (three nodes), I’m hoping to spin up three Vault VMs and migrate the Vault data off of the control node into the master Vault cluster.

Q1 - To get the Vault data into the master cluster and off of the control station, would it be easiest just to spin up the Proxmox VM Vault nodes one at a time, have them join the control station’s Vault in a cluster, and after all three Vault VMs are running I can kill off the Vault instance running on the control station? Does this make sense or is there a better way to do it? Note that the control station is going away eventually after the cluster is bootstrapped.

Q2 - The cluster will have three Proxmox hosts. On each host will be a VM (I’ll call them infracore VMs) that runs Vault, Consul, and Nomad. Also on each host will be a VM that hosts the apps (I’ll call them appvms) that are deployed via Nomad. I’ve read varied recommendations on which of the three Hashicorp products to deploy first. If I was just reasoning about it without knowing much detail about the products, I would assume I would deploy Nomad first on the infracore VMs and then deploy Vault and Consul with Nomad since Nomad is the deploy control plane. However, since so much will depend on Vault I’m thinking I might want to reduce the variables and NOT deploy it on Nomad. Given that thinking (or you can argue against my thinking of course), assuming I’m deploying Vault directly in the OS, does it matter which I deploy next, Consul or Nomad? Would it be fine to deploy Consul via Nomad? All of this is the biggest conundrum in my head hehe.

Rollout Plan

If you’re interested, here is a summary draft of the rollout plan:

  1. Ansible spins up Vault on control station (ansible running against localhost)
  2. Ansible spins up DNS and Netbox on control station. Some data will be auto-populated into Netbox by scripts so that Netbox can be the source of truth and Ansible, Terraform, and Maas etc. can get data like IP addresses, mac addresses, etc. from Netbox as it needs to.
  3. Ansible configures KVM on control station
  4. Packer build an image for and Ubuntu Maas region and rack controller.
  5. Terraform deploy Mass image as a VM running on control station.
  6. Ansible configure Maas VM
  7. Packer build Proxmox images for Maas
  8. Plug in the three servers to the network and let Maas install Proxmox on them
  9. Packer build base image for cluster VM node running Debian 13 (Trixie)
  10. Terraform deploy the three infracore VMS
  11. Ansible configures Vault cluster on the infracore VMs
  12. Do the migration approach from the result of Q1 above to get the Vault data over into the cluster and then kill the local Vault instance that is on the control station.
  13. After Vault migrated, ansible deploy Consul and Nomad in the order that resulted from Q2 above.
  14. Terraform deploy the three appvms and then Ansible configures them with vault, consul, and nomad agents.
  15. Deploy PostgreSQL in some type of HA cluster configuration on Nomad
  16. Migrate Netbox data to PostgreSQL, spin up Netbox on Nomad running against the new Postgres “cluster”. Too bad some of the apps don’t officially support Cockroachdb :slight_smile:
  17. Kill off Netbox on the control station.
  18. Migrate Mass vm to Proxmox cluster and kill off the Maas VM on the control station
  19. Now the cluster is ready for other app deployments. This will include Authentik and some other things.

If you’ve read this far, seriously thank you for your time. I wish a case of the drink of your choice to magically appear on your doorstep.

Regards

2

Hi there - welcome!

Q1 - you could do the migration you’re talking about, or you might consider leaving that single instance Vault node, should your proxmox cluster or nomad go down you still have access to secrets/certs you may need to bring things back up…at the cost of operational overhead, but given its not the tier 0 Vault cluster, may be worth it.

Q2 - My general answer here is…it depends. Is Vault a truly tier 0 app where everything else relies on Vault to start? If so my general recommendation is to run Vault on the platform you/your team/your company is best suited to support and troubleshoot. If you’re comfortable with Nomad/containers - thats a perfectly viable solution to run Vault with Nomad. If thats somewhat new, or you’re just more comfortable with Proxmox/VMs that may be the better option.

3

Hey Jonathan,

Thanks for take the time and giving my questions some thought. I think that is a great idea about Q1 and keeping the Vault instance. I’ll keep it unless the situation changes.

Regarding Q2, I could definitely say I’m more comfortable running Proxmox because I’ve never run Nomad, but I’m familiar with container tech and running containers in K8 /GKE and the like. I"m actually a software guy that does distributed systems stuff but am helping out a friend and his company here who is somewhat technical but wants everything as automated as possible. Thus, I’m writing copious documentation because with all this automation has the associated complexity vs just spinning up apps on a host. I have a follow-up question:

Q3: From a technical standpoint, assuming the three (Consul, Nomad, Vault) are all going to be deployed before we start using them, does it matter which order I spin up the Consul, Vault, and Nomad stuff as far as integration complexity? I don’t want to run into a scenario where I say, do Consul last and find out it is twice as hard and complex to integrate with Vault and Nomad as it would have been if it had started by deploying Consul first.

I had assumed I would need to do Vault first so that it could issue certs and store the creds etc. of both Consul and Nomad. However, if the answer to Q3 is that as far as complexity the order doesn’t matter that much, then I would spin up Nomad first, and then use it to deploy Vault and then Consul. This seems the most natural to my way of thinking since I’m actually using our deployment system (Nomad) to deploy Vault and Consul, rather than just via Ansible.

Regards,

David

4

You can use Consul WITH Nomad, but not run it on Nomad.

Since you’re going with the “backup” Vault node, that gives you the flexibility if you decide to run Vault on Proxmox VMs, or as containers. If you go the proxmox route, order doesnt matter I would think since you have that Vault backup node.

One note, the Nomad docs team has a similar warning about not running Vault on Nomad, which contradicts https://www.hashicorp.com/en/blog/running-vault-on-hashicorp-nomad-part-1 - asking around internally to get clairification. Wondering if the statement on Nomad not running Vault is because Nomad would then fail to find the Vault cluster defined in the Nomad configuration. I’ll get verification on the Consul statement too since they are worded the same.

5

Okay, confirmed, the statements are just poorly phrased. You can choose to run both Vault and Consul via Nomad. On the Vault side, if you reference Vault in your Nomad config it should start even if Vault isn’t running yet. If you are using Consul to let Nomad discover itself, don’t run that on Nomad.

6

ahhh… I think I get it. If I run Consul first then Nomad can use it for self-discovery, otherwise I’ll need to tell Nomad about its other initial members in its initial config. I’m assuming that in the Nomad config I’ll go ahead and reference Vault as if it were there and the same for Consul based on what you said. I’ll give it a shot and see. I’m playing with Nomad now and am going to have to head over there and post a question cause I can’t hit the hello_world http://localhost:8000 in my browser after starting the hello_world pack. Great start, huh? hehe :slight_smile: :slight_smile: Thanks much!

7

Sounds like an absolutely great start - I learn way more from not getting it right the first time though (I say that while super frustrated I can’t get something Boundary related working lol) ! I’ll help as best I can, but Nomad and Consul and def weak spots for me.