I've run out of viable methods to handle service configuration files in Nomad

Nomad
3

Hi @Uweee,

thanks a lot for your comment. Sadly, it’s not (exactly) what I was looking for, because it requires a git commit for each config change - which was one of the things I wanted to avoid. :slight_smile:

I’ve now come up with the following solution:

  • Store my config files (both, the Nomad job files and the application configs) in a S3 bucket on my local Ceph cluster
  • Mount that S3 bucket with S3FS on my developer machine
  • Use the artifact stanza’s S3 functionality to download the right config subdirectories from that same S3 bucket in the job files
  • I’m getting around defining the S3 credentials in the job files with the help of the tip in this go-getter issue. In short, I’m setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables in the Nomad client’s environment (using a systemd environment file). In addition, I’m setting AWS_METADATA_URL=dummy. Without that last variable, non-aws S3 URLs are always replaced with AWS URLs

This approach fullfills all my requirements for now.

But there is one big remaining problem: Config file updates. If the only thing which changes is one of the application config files, there is no proper way to restart a job so the new configuration is taken into account.
Because the files are downloaded during allocation creation, and then never checked again, neither changes in template files nor changes in static config files mean anything at all to Nomad.

If I change a config file or config file template and then issue nomad run /path/to/jobfile.hcl, nothing whatsoever happens - as the job file itself hasn’t changed, Nomad doesn’t see any reason to do anything. And as far as I could see, there isn’t any option anywhere that simply says: I don’t care what you think, Nomad. Completely restart all allocations of this job, including all of the prep work. Right now, please.

Initially, my solution was to always change something small in the job file alongside config file changes. My preferred approach was to just increase the memory limit in the resources stanza of one of my tasks by one Megabyte, which would induce Nomad to restart and download the newer config file.

Quick Edit: Forgot to mention: I discovered the nomad alloc stop command yesterday. This command actually does (part of!) what I need: It restarts the given allocation, including artifact dowloads and templating. But of course, it only works for a single allocation, so it’s a lot more work if you change e.g. the configs of a system job. You would first have to find all allocations of the job, and then “stop” them one by one.

I’m getting stronger and stronger vibes that I’m holding this wrong.