At this time, what is the cleanest way to deploy Nomad as a systemd service and have it be able to access the VAULT_TOKEN environment variable?
I’m attempting to remove the token from the agent configuration vault stanza and instead use the environment variable. When running the agent directly via cli (e.g nomad agent ...), Nomad is fine with this. However, as a systemd service, I get permission denied warnings from vault, as if it’s unable to read VAULT_TOKEN correctly. For those running Nomad as a systemd service, have you encountered this as well? How have you approached integrating Vault? Thanks so much!!
Yeah I’ve encountered this. I am not sure why it works that way, but I worked around it in the following way.
Step 1: Introduce VAULT_TOKEN as an environment variable in the systemd service via the EnvironmentFile directive.
Step 2: Use cloud-init to render the token when the VM instance is provisioned.
As a final note, my code example in (2) introduces the root token. I don’t advocate that, this is a hobby cluster, I’m just showing a concrete example. Instead try to use a Nomad token role set up in Vault with narrowly-scoped policies.
Just to add a new bit of information, if you install Nomad from our official Linux packages, the systemd unit is pre-configured to look for environment variables from the file /etc/nomad.d/nomad.env.
That’s a pretty odd place for environment files for a SystemD service, no? I have seen /etc/sysconfig/<app_name> OR /etc/default/<app_name>. This is new !!!
I don’t know what’s systemd idiomatic. I just read the systemd.exec documentation and found something that worked for me. There can be multiple good ways to solve a problem, and I was just offering one of them.
I don’t think there’s a standard path to place these files that set environment variables. A quick search for systemd EnvironmentFile default path returns several interesting results (like the comment section in this answer)
So at this point everyone seems to be confused about this. While it may be confusing to you, it makes sense for @AdrienneCohea’s team since that’s how they organize their files
I was just saying that I am used to /etc/default/nnn and /etc/sysconfig/nnn for key=value type of files, and would not want to mix .hcl and .env files from the same directory!
I’m working on integrating HashiCorp Vault into our application using Vault Agent for authentication. The initial setup works well, where the application reads the Vault token from a file generated by Vault Agent and uses it to authenticate with the Vault server.
However, I’m concerned about handling scenarios where the token’s max TTL is reached, and a new token is generated by Vault Agent.
Currently, our application reads the token once during initialization and uses it for subsequent operations. If the token expires and a new one is generated, the application wouldn’t automatically know about the new token, which could lead to failed operations.
To address this, I am thinking to implement a file watcher that monitors the token file for changes. When a new token is generated, the watcher reloads the token and updates the Vault client. While this seems to work in theory, I want to ensure that we’re following best practices and not missing any important considerations.
Here are the specific questions I have:
Is monitoring the token file for changes and reloading the token dynamically the recommended approach for handling token renewal with Vault Agent?
Are there any potential pitfalls or edge cases I should be aware of when implementing this solution?
Are there more efficient or reliable methods to ensure the application always has access to a valid token, especially in high-availability or production environments?
I’d appreciate any feedback or suggestions on improving this implementation.
)
