Confusion with Nomad + Vault

Nomad
1

I’m new to both Nomad and Vault and after setting up both agents, I ran nomad setup vault. However, after reading some docs, I am confused on how to use them.

I take it that jobs can only access secrets in secret/data/<job namespace>/<job name> ? if so, how would you share credentials between 2 different jobs? For example, how would you instanciate a postgres db and some web api that uses the db (if they are separate jobs)?

2

The model from my understanding of Vault is that you provide each job with a token that provides the necessary access rights to get to the secret you need, rather than sharing passwords between the services. This is a rather general answer, and in my experience, you will need to look at vault as a framework on top of which you will build your security implementation, rather than a cookie cutter finished solution.

What I started out with was playing around with figuring out how I would log in to vault first, and then, once I had a way to interact with it, I started looking at how to get a token, and lastly, how to get the right token to the client. Once I had figured out those parts, I was able to build a chain that made sure that the client was able to access a token to get the relevant information from vault, and then improve the flow using wrapped secrets. It’s been a lot of experimentation, but in the end, it’s been worthwhile. Good luck!

3

(So, the cheap solution is to store credentials as Nomad variables. But we want to use vault. So anyway:)

First store the password in vault in some key-value store.

Then use a template that uses consul-template to generate the password from vault and export it as an environment variable for the task. template Block - Job Specification | Nomad | HashiCorp Developer

template {
  data = <<EOH
POSTGRES_PASSWORD="{{with secret "secret/data/postgres_password"}}{{.Data.data.postgres_password}}{{end}}"
EOH
  destination   = "local/env.env"
  env = true
}
4

Thanks, @Kamilcuk . That is what I did initially. However, I got errors regarding permissions, which is expected because I hadn’t setup any further policies/roles except for the ones created during nomad setup vault. I guess my questions should’ve really been “How do you setup policies and roles?”

5

What errors exactly? Please post the exact full verbatim error messages. Check your server and agent logs.

There are levels of ACL tokens - nomad server connects to vault to check, then nomad client separately connects to vault, and they both have their own permissions, and also you have VAULT_TOKEN in your shell with different permissions. Set up a diving suite and dive in Vault ACL | Nomad | HashiCorp Developer . It’s better to do without ACL for the first time to deal with ACL later.

Here is a full work through that I used instead of nomad config: Vault integration and retrieving dynamic secrets | Nomad | HashiCorp Developer . It also has all the “vault policy” and “nomad policy” odd setup.

6

For anyone like me who might visit this topic:

First, create some kv’s at auth/db (or whatever path really) using the vault cli.

And then create a policy like so:

# vault policy write auth-service auth-service.policy.hcl

path "secret/data/auth/db" {
    capabilities = ["read"]
}

Then bind that policy to a role like so:

# vault write auth/jwt-nomad/role/auth-db-access '@auth-db-access.role.hcl'

{
  "role_type": "jwt",
  "bound_audiences": ["vault.io"],
  "user_claim": "/nomad_job_id",
  "user_claim_json_pointer": true,
  "claim_mappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task"
  },
  "token_type": "service",
  "token_policies": ["auth-service"],
  "token_period": "24h",
  "token_explicit_max_ttl": 0
}

and finally, use it in your jobs:

# nomad job run postgres-auth.hcl

job "postgres-auth" {
  type = "service"
  # some stuff omitted for brevity ...

  group "postgres-auth" {
    # ...

    task "postgres-auth-server" {
      # ...

      vault { 
        role = "auth-db-access"
      }

      template  {
          data = <<EOF
              POSTGRES_USER = "{{with secret "secret/data/auth/db"}}{{.Data.data.username}}{{end}}"
              POSTGRES_PASSWORD = "{{with secret "secret/data/auth/db"}}{{.Data.data.password}}{{end}}"
          EOF
          destination = "secrets/env.env"
          env = true
      }
    }
  }
}

example of another file using the same secrets

# nomad job run api-auth.hcl

job "api-auth" {
  type = "service"
  # ...
  group "api-auth" {
    # ...

    task "api-task" {
      # ...
      vault { 
        role = "auth-db-access"
      }

      template {
          data = <<EOH
            DATABASE_URL="postgres://{{with secret "secret/data/auth/db"}}{{.Data.data.username}}:{{.Data.data.password}}{{end}}@{{ range nomadService "postgres-auth" }}{{ .Address }}:{{ .Port }}{{end}}/auth_db"
          EOH
          destination = "secrets/env.env"
          env = true
      }
    }
  }
}
7

I was only trying it out in dev mode, and I think the logs are long gone probably. Do you know where I might find any log files? FWIW, I believe the error looked something along the lines of failed to read(secrets/data/auth/db) or something.