Injecting vault tokens for on-prem nomad and consul-template

How do we inject vault token(s) into consul-template (for tls) and nomad (vault integration)?
Secure introduction says approle is prefered (along with vault agent), but there are some issues:

  • Consul-template (and nomad?) only work with tokens and hence need an intermediary like vault agent to convert approle to token.
  • For a bare metal user (say with ansible as config mgmt tool), how do I get role-id and secret-id into the machine with different channels ? (For cloud this could be terraform + ansible or cloud-init + terraform).
  • Since, I am anyways trusting ansible with my config mgmt, should I stop over-engineering and let ansible create and inject tokens into consul-template and nomad ?

Bootstrapping hashistack is probably the hard hitting pain point atm.
Thanks in advance !

Hi @blmhemu,

DISCLAIMER: I’m not a Nomad or Ansible user so take any advice here with a grain of salt.

Generally speaking, one would want to have vault-agent running on any instance where Vault secrets are needed (especially so on bare metal ones where the lifecycles are traditionally longer than those of e.g. Kubernetes Pods). If not that, some other agent capable of authenticating against Vault and fetching, renewing and storing secrets locally. After that, consul-template and nomad should not have issues to find those local tokens. You would still need to pass the role-id and secret-id to the agent, perhaps using Ansible’s own vault system (which I’m not very familiar with, so can’t offer any further advice about).

Another way would be to generate indefinitely renewable tokens for those applications and let them manage the renewal themselves without any out-of-band secret management agent. In such case, you could use community.hashi_vault.hashi_vault lookup – Retrieve secrets from HashiCorp’s Vault — Ansible Documentation to get the initial tokens and configure both consul-template and nomad with them (as you suggested in your last point). In that scenario, Ansible itself would assume the role of Vault’s AppRole and not the individual instances that are being configured by it.

Some last thoughts:

  • For the moment, you may only need those two applications to have access to secrets but later on, other apps may need to access different types of secrets such as TLS certificates, key-value pairs and database credentials, so having a deployed agent may pay off in the end. (as an after-thought though, if you’re using Nomad exclusively to deploy your workloads and it is capable of fetching secrets from Vault, then you may not need the agent after all).
  • If your only purpose is to setup nomad with access to Vault secrets, I believe you don’t need to setup consul-template separately, since nomad should be able to run it under the hood.

Hi @macmiranda ! Thanks for the detailed reply.

While I agree with your assessment, currently we need to run multiple vault-agents for each consumer (vault-agent attaches same policies to all sinks, so to isolate tokens, we need one for each consumer). There are some open issues on the github for request to solve this

I am using consul-template for mTLS for consul and nomad.

The ansible lookup looks like a good trade-off though.