Hi everyone,
It is better to perform a standalone instalation of vault on my nodes on use a nomad job to deploy vault on my nomad cluster?
I ask question for consul but I don’t think that will be efficient to put consul in nomad cluster.
I have a feeling that there is no definitive answer to your question and that it depends on what your final goal is. I provide some of my experience so that you can compare your requirements with it.
I faced this same question during the design of a datacentre. My experience, for what it is worth, has led me to the conclusion that it really depends on what you are doing, but it helps to consider failure modes. It also helps to make it explicit what you consider as part of the infrastructure layer and what as part of the application layer
If you deploy Vault into a Nomad cluster, you need to consider the failure modes of Nomad. What would happen if the cluster loses quorum? Will other applications which need secrets be affected? Will taking action at the Nomad layer result in your Vault becoming sealed or unavailable, e.g. due to a migration? What happens to the data (i.e. the secrets) ?
I wasn’t able to answer these questions a-priori, so I decided that it was much clearer (to me), to separate the two. Vault is piece of infrastructure. Consul is a piece of infrastructure. These two are deployed first and independently.
The Nomad control plane then depends on these two, and is a second-level layer of infrastructure. In my experience, there is an explicit dependency between Vault and Nomad. Nomad needs Vault, and not the other way around. This allows you to use all of the Vault integrations that Nomad provides, to inject secrets, template files, etc into workloads.
You can deploy Vault into Nomad first and then use it to inject secrets into all the other jobs, so at some level, there is no difference between having it deployed into Nomad or aside Nomad. However, I prefer the latter since it allows one to apply separate controls and procedures and makes the architecture of system a bit clearer:
- Deploy Vault into empty environment. Configure vault to register itself with Consul when it becomes available
- Deploy Consul across resources. Consume secrets from Vault to configure Consul
- Deploy Nomad across resources. Discover Vault from Consul DNS interface. Consume secrets from Vault to configure Nomad.
- Run Nomad jobs. Consume secrets
I chose to designate Vault as a piece of infrastructure which all other services depend on. For example, I keep the Consul secrets (ca certs, gossip key) in Vault and my deployment procedure (Ansible playbooks) consumes those secrets when deploying the datacentre. A similar approach is used with Nomad deployment. This allows a predictable, linear deployment procedure from scratch, but also allows one to isolate problems with infrastructure and reduce the blast radius of outages in one or another piece of infrastructure.
I hope these considerations help, and I’d be keen to hear from others on the point.
2 Likes