Vault Token and Nomad Agent as Systemd

Nomad
1

Hi all,

At this time, what is the cleanest way to deploy Nomad as a systemd service and have it be able to access the VAULT_TOKEN environment variable?

I’m attempting to remove the token from the agent configuration vault stanza and instead use the environment variable. When running the agent directly via cli (e.g nomad agent ...), Nomad is fine with this. However, as a systemd service, I get permission denied warnings from vault, as if it’s unable to read VAULT_TOKEN correctly. For those running Nomad as a systemd service, have you encountered this as well? How have you approached integrating Vault? Thanks so much!!

1 Like
2

Yeah I’ve encountered this. I am not sure why it works that way, but I worked around it in the following way.

Step 1: Introduce VAULT_TOKEN as an environment variable in the systemd service via the EnvironmentFile directive.

Step 2: Use cloud-init to render the token when the VM instance is provisioned.

As a final note, my code example in (2) introduces the root token. I don’t advocate that, this is a hobby cluster, I’m just showing a concrete example. Instead try to use a Nomad token role set up in Vault with narrowly-scoped policies.

1 Like
3

Hi @AdrienneCohea,

Thank you for this. I hadn’t thought about using cloud-init to render the token, this might end up fixing this chicken-egg issue I’m encountering.

1 Like
4

@AdrienneCohea answers is pretty spot-on.

Just to add a new bit of information, if you install Nomad from our official Linux packages, the systemd unit is pre-configured to look for environment variables from the file /etc/nomad.d/nomad.env.

1 Like
5

That’s a pretty odd place for environment files for a SystemD service, no? I have seen /etc/sysconfig/<app_name> OR /etc/default/<app_name>. This is new !!! :thinking:

6

@lgfa29 good to know, thank you!

7

I don’t know what’s systemd idiomatic. I just read the systemd.exec documentation and found something that worked for me. There can be multiple good ways to solve a problem, and I was just offering one of them. :heart:

1 Like
8

The concern for non-standard locations would be that users would not be expecting that.

In my opinion, mixing a typical place for .hcl files with .env files would be confusing.

I even happened to notice a dotfile for such a thing here … nomad/nomad.service at main · hashicorp/nomad · GitHub

super confusing in my opinion :slight_smile: :slight_smile:

9

Okay but it’s relatively common for system administrators to know to go into systemd and look at a unit file.

I don’t think it’s confusing to do “systemctl status myprogram” and “ls /etc/myprogram” and “ls /etc/myprogram.d”

10

I don’t think there’s a standard path to place these files that set environment variables. A quick search for systemd EnvironmentFile default path returns several interesting results (like the comment section in this answer :sweat_smile:)

So at this point everyone seems to be confused about this. While it may be confusing to you, it makes sense for @AdrienneCohea’s team since that’s how they organize their files :slightly_smiling_face:

11

Of course, anyone is free to do as they please.

I was just saying that I am used to /etc/default/nnn and /etc/sysconfig/nnn for key=value type of files, and would not want to mix .hcl and .env files from the same directory! :peace_symbol: