Nomad unable to get vault token

Nomad
1

I attempting to put vault secrets in my nomad job files according to these instructions.

Running Nomad 0.9.4 and Vault 1.1.0

I’m using token role based integration, not root token integration

So far I have

  1. Created vault policy
path "auth/token/create/nomad-cluster" {
  capabilities = ["update"]
}
path "auth/token/roles/nomad-cluster" {
  capabilities = ["read"]
}
path "auth/token/lookup-self" {
  capabilities = ["read"]
}
path "auth/token/lookup" {
  capabilities = ["update"]
}
path "auth/token/revoke-accessor" {
  capabilities = ["update"]
}
path "sys/capabilities-self" {
  capabilities = ["update"]
}
path "auth/token/renew-self" {
  capabilities = ["update"]
}
  1. Applied the policy
vault policy write nomad-server nomad-server-policy.hcl
vault policy read nomad-server
  1. Generated the vault token role using the blacklist (disallowed policies) strategy.
{
  "disallowed_policies": "nomad-server",
  "explicit_max_ttl": 0,
  "name": "nomad-cluster",
  "orphan": true,
  "period": 259200,
  "renewable": true
}

vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
vault read /auth/token/roles/nomad-cluster
  1. Retrieve the token role based token
vault token create -policy nomad-server -period 72h -orphan

Key                  Value
---                  -----
token                s.xxxxxxxxx
token_accessor       yyyyyyyy
token_duration       72h
token_renewable      true
token_policies       ["default" "nomad-server"]
identity_policies    []
policies             ["default" "nomad-server"]
  1. Add the token to my nomad config and restart nomad
    (This will be moved to an environment variable for security)
vault {
  enabled = true
  address = 'https://vault.example.com:8200'
  create_from_role = nomad-cluster
  token = 's.xxxxxxxxx'
}

Then on the nomad agents I enable vault

vault {
  enabled = true
  address = 'https://vault.example.com:8200'
}
  1. Add a secret to vault
vault write secret/foo bar=42
vault read secret/foo

However when I attempt to submit a job that uses the vault secret, I get this error

I’ve tried several ways to get the secret

template {
        data = <<EOH
        {{ with secret "secret/foo" }}
        bar = {{ .Data.bar}}
        {{end}}
        EOH
        destination = "${NOMAD_SECRETS_DIR}/foo.txt"
      }

template {
        data = <<EOH
        {{ with secret "foo" }}
        bar = {{ .Data.bar}}
        {{end}}
        EOH
        destination = "${NOMAD_SECRETS_DIR}/foo.txt"
      }

template {
        data = <<EOH
        {{ with secret "foo" }}
        bar = {{ .Data.data.bar}}
        {{end}}
        EOH
        destination = "${NOMAD_SECRETS_DIR}/foo.txt"
      }

But I always get the error

	Template	Missing: vault.read(foo)

Do I need to make a new policy for every secret?

  vault {
    policies = ["foobar"]
  }
1 Like
2

I got it to work by creating a role and policy for every app that needs to store secrets. I’m not positive if this is the best workflow.

path "secret/foobar" {
  capabilities = ["read"]
}

resource "vault_token_auth_backend_role" "foobar" {
  role_name = "foobar"
  allowed_policies = ["foobar"]
  explicit_max_ttl = 0
  orphan = true
  period = 259200
  renewable = true
}
3

Hello, I’m looking at this guide, too - I guess for the blacklist approach you’d just need to have any policy that’s not blacklisted that allows read on that path.
So you should not need to create that "foobar" role, but the policy has to be created if no other policy exists that gives read capability to "secret/foobar"

That’s how I understood it, please someone correct me if I’m wrong.

4

I ended up switching to the blacklist approach

vault read auth/token/roles/nomad-cluster
Key                       Value
---                       -----
allowed_entity_aliases    <nil>
allowed_policies          []
disallowed_policies       [nomad-server]
explicit_max_ttl          0s
name                      nomad-cluster
orphan                    true
path_suffix               n/a
period                    72h
renewable                 true
token_explicit_max_ttl    0s
token_period              72h
token_type                default-service

I’m not quite positive what change fixed it, but I suspect it was this change the the vault policy

Before

path "secret/data/foo/*" {
  capabilities = ["read"]
}

After

path "secret/data/foo" {
  capabilities = ["read"]
}
5

Your “before” policy is for every secret in the path of foo (so foo is like a directory). Your “after” policy is for the secret foo (like a file).
The default policy is to deny, so if you can read everything under foo you cannot read the secret foo at all.

Confusing explanation. :joy: