UID/GID ownership of template output in NOMAD_SECRETS_DIR for Postgres

[hntrmrrs]

I am running the “offical” Postgres container on Nomad (using the Docker driver) and provisioning a PKI keypair from Vault via a template stanza. The issue I have is that the Postgres container runs as a specified user with uid 999/gid 999 and it isn’t in the root group. Postgres requires file mode 0600 or 0640 for its SSL key. I’d like to have it in NOMAD_SECRETS_DIR and somehow change the ownership.

Here’s a representative example of the template for generating the key:

template {
        destination = local.postgres_key_path
        perms       = "0600"  # -rw------- 1 root root
        data        = <<EOF
{{- with secret "pki/issue/postgres" "common_name=postgres.service.consul" "format=pem" -}}
{{ .Data.private_key }}
{{- end -}}
EOF
}

I’d prefer not to run Postgres as root or run a privileged container. Is it possible to change the ownership of a secret to make Postgres happy? Or is there some other solution I’m missing?

[eikenb]

Hey @hntrmrrs,

I don’t think changing ownership of the rendered template is supported by Nomad at the moment. I’m not 100% as I maintain consul-template and don’t work directly on Nomad, but consul-template just recently merged a PR with that feature (setting the user/group of the template’s destination file) and it didn’t have it before that.

Those features will eventually make it into Nomad, but that will be probably be a ways out as they’ll need to update their dependency to use the latest consul-template.

The PRs related to adding that feature in case you’re curious.

• Allow setting user/group ownership of template output by deblasis · Pull Request #1531 · hashicorp/consul-template · GitHub
• Allow setting ownership by name by lukas-w · Pull Request #1541 · hashicorp/consul-template · GitHub
• Backward compatibility for chown, broken in 0.28.0 -> 0.29.0 by deblasis · Pull Request #1551 · hashicorp/consul-template · GitHub

Sorry I couldn’t be of more help.

[harningt]

Quick note: this feature is in (at least as of 1.3.5) however it has a nasty caveat that I just discovered.

If you set the uid of the cert/key, then the file is re-rendered every ~3-5 minutes … sometimes multiple times in a row. The final content is the same - but it is a tad annoying to have signals send so frequently to reload configuration changes.

Also with postgres - if you are considering using the pkiCert utility - it will not work out unless you post-process the output because like most tools consuming certificates the key is expected to be separately encoded from the certificate… but pkiCert will render two distinct certificates/keys.

Ideal scenario that does not work:

• Two templates:
  • pkiCert that renders the private key w/ uid = postgres
  • pkiCert that renders the certificates w/ uid = postgres
• somehow those two pkiCert renders know that they are related just like the secret mechanism but is able to find that one of the template outputs has a certificate and uses that to avoid re-fetching
  Failures in this case:
• setting uid seems to force re-rendering
• two pkiCert entries cause two independent pki entries

Ok scenario for nomad (because existing cert isn’t expected on start):

• Two templates:
  • secret resulting in private key rendering w/ uid = postgres
  • secret resulting in certificate rendering w/ uid = postgres
    Failure in this case: setting uid seems to force re-rendering

Scenario that seems to work okay:

• use two templates w/ secret to render the private key and public key
• change script configured to create a copy w/ changed ownership and send the reload signal (probably only have one of the templates signal … to avoid duplicate signals)

If pkiCert is desired (seems more relevant if using vault-template or consul-template) we have two scenarios

Complicated one - but works ok for nomad:

• Render a server.bundle that has both private and pem w/o setting UID/GID - attach a load-cert script … either using pkiCert or secret
• Have an additional boot script that does the certificate processing logic w/o config reload part

Load cert script:

• Split the key and pem into the appropriate file and change the ownership to the postgres user (openssl storeutl -certs -out <cert-file>.tmp <bundle>; openssl storeutl -keys -out <key-file>.tmp <bundle>; chmod 0600 $KEY_FILE.tmp; chmod 644 $CERT_FILE.tmp; chown postgres $KEY_FILE.tmp $CERT_FILE.tmp; mv $KEY_FILE.tmp $KEY_FILE; mv $CERT_FILE.tmp $CERT_FILE)
• Send appropriate signal to postgres (ex: pg_ctl reload -D /var/lib/postgres)
  attach a change script that uses openssl to split out the key and certificate and put them in the right place

pkiCert template that works ok for non-nomad:

• Single template that writes out the certificate and has a writeFile to key
  … does not work well for nomad because it requires unlocking file writing capabilities
  … also still requires the script to change permissions

additional note: have tries Nomad 1.4.0-beta1 to see if it somehow solved the issue - nope

[harningt]

And while trying to finish implementing the “secure” version of the rendering a server.bundle w/o messing with UID/GID and letting a script handle it… I ran into the issue with the fact that I run Docker using user-namespaces - so to properly secure the generated certificate I had to set its mode to “0600” … but with user-namespaces your container’s root != your root and the file is never readable (kind of the pt with user-namespaces, really now that I think of it).

So - setting the certificate to be root owned but accessible in the environment turned out impossible.

An idea that came in my head was to try setting the group ownership to a user in the container… it worked permission-wise. However … the re-rendering bug stops it from being an ideal solution.

So - given that a script would have been needed anyways with splitting and adjusting permissions from root… the best solution I can see now (and still use nomad templates) is:

• Setup server.bundle file as output - owned by the postgres user inside
• Setup a script that checks to see if the server.bundle has changed and ignore any attempts to update it

An alternative would be to take the VAULT_TOKEN passed in and use it in something like vault-pki-agent · PyPI to manage certificates… not ideal from the additional python dependencies and from side-stepping central configuration management in Nomad.

[eikenb]

@harningt

If you set the uid of the cert/key, then the file is re-rendered every ~3-5 minutes … sometimes multiple times in a row. The final content is the same - but it is a tad annoying to have signals send so frequently to reload configuration changes.

Ownership (UID/GID) is one of the things checked for at render time and it will trigger a re-render. It wants the files generated to have the same UID/GID as the running process. For consul-template this makes sense as you’d be running consul-template as the postgres user. Not sure how this is supposed to work with nomad.

Note that the writeToFile file destination does not have this check. If you write the template such that the writeToFile files are the ones that need the ownership changes it should handle it better.

@harningt

Scenario that seems to work okay:

use two templates w/ secret to render the private key and public key

Note that this will also result in getting a new PKI cert/key whenever you restart the (nomad?) process as secret works only with the data from vault which is keeps in memory to know its state. It cannot read data back from the local files as pkiCert does.

@harningt

pkiCert template that works ok for non-nomad:

• Single template that writes out the certificate and has a writeFile to key
  … does not work well for nomad because it requires unlocking file writing capabilities
  … also still requires the script to change permissions

This is consul-template’s current recommended way to work with pkiCert in cases where you need to split them out into separate files.

Sorry I’m not sure if any of that helps that much.

I’ve been thinking about other ways of addressing this but haven’t come up with much. You need both the PKI Key and the Cert as they cannot be re-fetched without generating a new one. But there is no place to store them except for the target file and there is only 1 of those. Cases like this are one of the reasons we added writeToFile.

Good luck and if you have more questions please feel free to ask and I’ll do what I can. Thanks.

[harningt]

@eikenb

@harningt

If you set the uid of the cert/key, then the file is re-rendered every ~3-5 minutes … sometimes multiple times in a row. The final content is the same - but it is a tad annoying to have signals send so frequently to reload configuration changes.

Ownership (UID/GID) is one of the things checked for at render time and it will trigger a re-render. It wants the files generated to have the same UID/GID as the running process. For consul-template this makes sense as you’d be running consul-template as the postgres user. Not sure how this is supposed to work with nomad.
…

Looking at the code - it looks like it is supposed to be doing a check on whether or not the UID and GID match up with what is configured for both Nomad and consul-template. The whole render-path setup for Nomad looks to be directly through consul-template … which checks to see if the recorded render time is >= the stored render time.

@eikenb

@harningt

Scenario that seems to work okay:

use two templates w/ secret to render the private key and public key

Note that this will also result in getting a new PKI cert/key whenever you restart the (nomad?) process as secret works only with the data from vault which is keeps in memory to know its state. It cannot read data back from the local files as pkiCert does.

@harningt

pkiCert template that works ok for non-nomad:

• Single template that writes out the certificate and has a writeFile to key
  … does not work well for nomad because it requires unlocking file writing capabilities
  … also still requires the script to change permissions

This is consul-template’s current recommended way to work with pkiCert in cases where you need to split them out into separate files.

Got it - yeah - it feels like it’d add a ton of complexity to handle it any other way.

@eikenb

Sorry I’m not sure if any of that helps that much.

I’ve been thinking about other ways of addressing this but haven’t come up with much. You need both the PKI Key and the Cert as they cannot be re-fetched without generating a new one. But there is no place to store them except for the target file and there is only 1 of those. Cases like this are one of the reasons we added writeToFile.

For writeToFile the listed concern is that you have control to write freely on the filesystem. Would it be reasonable to add a restriction on writeToFile to honor the path sandbox that file reading does? Looking at things it seems like it’s a simple operation since the majority of the work is dealt with by the file utility.

I think the writeToFile solution will be what I adopt - I’ve used it for writing out PKI when I don’t have Nomad helping out (but the process owner being an unprivileged user and writeToFile not restricted). In the short-term, I’ll just live with the unlikely chance that something calls writeToFile in a way that breaks the sandbox.

So the plan I think will work right:

• Set a template output to a file that has everything (keys, cert, ca…) w/ restrictive owner and not changing UID
• Add writeToFile items in the template that write out the separate certificate and key files using the stored template data and sets the appropriate owner
• Unlock the writeToFile capability in Nomad.
• See about writing an pull-request for writeToFile honoring the sandbox.