At this time, what is the cleanest way to deploy Nomad as a systemd service and have it be able to access the VAULT_TOKEN environment variable?
I’m attempting to remove the token from the agent configuration vault stanza and instead use the environment variable. When running the agent directly via cli (e.g nomad agent ...), Nomad is fine with this. However, as a systemd service, I get permission denied warnings from vault, as if it’s unable to read VAULT_TOKEN correctly. For those running Nomad as a systemd service, have you encountered this as well? How have you approached integrating Vault? Thanks so much!!
Yeah I’ve encountered this. I am not sure why it works that way, but I worked around it in the following way.
Step 1: Introduce VAULT_TOKEN as an environment variable in the systemd service via the EnvironmentFile directive.
Step 2: Use cloud-init to render the token when the VM instance is provisioned.
As a final note, my code example in (2) introduces the root token. I don’t advocate that, this is a hobby cluster, I’m just showing a concrete example. Instead try to use a Nomad token role set up in Vault with narrowly-scoped policies.
Just to add a new bit of information, if you install Nomad from our official Linux packages, the systemd unit is pre-configured to look for environment variables from the file /etc/nomad.d/nomad.env.
That’s a pretty odd place for environment files for a SystemD service, no? I have seen /etc/sysconfig/<app_name> OR /etc/default/<app_name>. This is new !!!
I don’t know what’s systemd idiomatic. I just read the systemd.exec documentation and found something that worked for me. There can be multiple good ways to solve a problem, and I was just offering one of them.
I don’t think there’s a standard path to place these files that set environment variables. A quick search for systemd EnvironmentFile default path returns several interesting results (like the comment section in this answer)
So at this point everyone seems to be confused about this. While it may be confusing to you, it makes sense for @AdrienneCohea’s team since that’s how they organize their files
I was just saying that I am used to /etc/default/nnn and /etc/sysconfig/nnn for key=value type of files, and would not want to mix .hcl and .env files from the same directory!