Consuming Vault Secret On Nomad Job Task

Nomad
1

Hello everyone,

I have built a test environment for a Proof of Concept for the company I work for with Consul, Nomad and Vault (Hashistack).

I have everything working, but my main issue is how to consume kv1 secrets that I have created on Vault on Nomad job files. I’ve tried using nomad job description templates with consul-template syntax (https://github.com/hashicorp/consul-template/blob/master/docs/templating-language.md#secret), and tested pretty much all the examples that we can find in the Nomad docs/tutorials pages.

I am trying to run the docker container (using driver docker) from a private repo. So I’m trying to use Vault to manage the password as a secret. I’ve already tested run the job hardcoding the credentials and it works, so the problem is consuming the secrets from Vault.

Also, I can read the secrets using vault command (eg: vault kv get) and with the API using curl.

  1. Does anyone have a working example of a job description file consuming Vault secrets?

  2. Is there a way to check if Nomad is correctly requesting secrets from Vault? I’ve tried to get logs from the allocations but it is never descriptive.

2

I have an example job consuming vault secrets as environment variables:

template {
  data = <<EOF
    {{ with secret "kv/hello-vault" }}
    VAULT_SECRET_URL="{{ .Data.url }}"
    VAULT_SECRET_USERNAME="{{ .Data.username }}"
    VAULT_SECRET_PASSWORD="{{ .Data.password }}"
    {{ end }}
  EOF

  destination = "secrets/vault.env"
  env         = true
}

vault {
  policies      = ["hello-vault"]
  change_mode   = "signal"
  change_signal = "SIGHUP"
}

Full job file • Vault policy

It’s part of a little demo setup I build for running Nomad, maybe it can give you some pointers (also feedback is appreciated):

3

Hello, Thanks a lot for trying to help.

I’ve tried your code but, I can’t understand where you are using the environment variables you have created ( VAULT_SECRET_URL, VAULT_SECRET_USERNAME, VAULT_SECRET_PASSWORD).

I will post my code below. I’m trying to get secret from the vault and inject on the docker auth.

task "docker" {
      driver = "docker"

      template {
        data = <<EOF
{{ with secret "kv/aisc/DOCKERHUB_PASS" }}
DOCKERHUB="{{ .Data.key }}"
{{ end }}
EOF

        destination = "secrets/vault.env"
        env         = true
      }

      vault {
        policies      = ["nomad-server"]
        change_mode   = "signal"
        change_signal = "SIGHUP"
      }
      
      config {
        image = "<<pathToPrivateRepo>>"

        auth {
          username = "<<user>>"
          password = "${DOCKERHUB}"
        }
      }
    }

The secret was stored in the vault with the command
vault kv put aisc/DOCKERHUB_PASS key=""secret"

4

I think this topic can be closed. I just figured what was going on.

Turns out that I didn’t have stated in my policy created for the nomad server the capability of reading from that path I have stored the secret.

Thanks @fhemberger for the help anyways.

2 Likes
5

Sorry, I’m reading those three environment variables inside my app.

Glad it worked for you in the end!

1 Like