Problem getting vault secrets in a Nomad job

Vault
,
1

I don’t know if this is a Vault question or Nomad question, but I’m trying here. Admins, feel free to move this over. :slight_smile:

Based on the docs here: Vault Integration and Retrieving Dynamic Secrets | Nomad - HashiCorp Learn I think I’m doing things correctly…but could be way off.

I have a vault policy like so:

path "kv-v2/db" {
  capabilities = ["read"]
}

I add that to vault like so:

vault policy write db /ops/shared/config/db-policy.hcl

My Nomad servers have a valid vault token which was created using the steps in the doc link above. The token was put in the environment, and then Nomad was started.

My jobs have this stanza in the top level:

vault {
  policies = ["db"]
  change_mode   = "signal"
  change_signal = "SIGUSR1"
}

My template in the job is this:

      template {
        data = <<EOT
          {{ with secret "kv-v2/db" }}
POSTGRES_USER="{{ .Data.data.user }}"
POSTGRES_PASSWORD="{{ .Data.data.pass | toJSON }}"
          {{ end }}
EOT
        destination = "db.env"
        env         = true
      }

From the docs, it looks like I’m doing everything right, but when I run my jobs, they fail to run, and I get this error in the Nomad console:

Missing: vault.read(kv-v2/db)

I created the ‘db’ policy, which allows reads to kv-v2/db, and I put the ‘db’ policy in the vault stanza for the job…but it’s not working.

Any hints? Please feel free to point me to the paragraph(s) in the documentation I have missed. :slight_smile:

Thank you!

2

I found this post: Nomad unable to get vault token but from what I can see, I did everything the poster did in their “working” setup. I’ve modified my setup slightly, but not really functionally.

I moved the vault stanza to the task section

I renamed the mount point ‘secret,’ so my secret input is now:

vault kv put secret/db/config user=blah pass=blah

My policy, which is added to vault using the name ‘db’ is:

path "secret/db/*" {
  capabilities = ["read"]
}

The vault stanza in my task is:

      vault {
        policies = ["db"]
        change_mode   = "signal"
        change_signal = "SIGUSR1"
      }

I would be very appreciative of any tips. This is my last hurdle to get my Proof of Concept cluster up and going. :slight_smile:

3

OK, digging around on the client syslogs, I found this:

Apr  1 22:09:09 ip-172-31-27-91 nomad[1815]:     2021/04/01 22:09:09.197562 [WARN] (view) vault.read(secret/db/config): vault.read(secret/db/config): Error making API request.
Apr  1 22:09:09 ip-172-31-27-91 nomad[1815]: URL: GET http://active.vault.service.consul:8200/v1/secret/data/db/config
Apr  1 22:09:09 ip-172-31-27-91 nomad[1815]: Code: 403. Errors:
Apr  1 22:09:09 ip-172-31-27-91 nomad[1815]: * 1 error occurred:
Apr  1 22:09:09 ip-172-31-27-91 nomad[1815]: #011* permission denied

So, I could have surmised that. :slight_smile: It still doesn’t tell me why, of course. How do I go about debugging this? How do I see what token the server is trying to use? What token the client is trying to use?

Thanks!

4

Logging in with the nomad server token gives me a view where I can’t see secret/

So, that gives me something. Hmm…nothing in the bring-up docs talks about giving permissions to the nomad server to read the tree.

I’m guessing read permissions to secret/* need to be added to the nomad-server-policy.hcl. presented in the vault/nomad integration doc referenced in my first post.

But that doesn’t make sense either. What purpose does the task’s policy serve if Nomad can read the entire tree anyway?

OR…does policy needed to make the template != policy/role given to an app once it’s started?

Any insight on all this would be appreciated. :slight_smile:

7

OK…I’m out of ideas. I added this to the nomad-server policy.

path "secret/" {
  capabilities = ["read"]
}

path "secret/*" {
  capabilities = ["read"]
}

Now, when I log in to Vault using the nomad token, I can see secret/ in the top-level list, but when I lick on “secret/” I am told: “You don’t have access to secret/.”

Help.

8

The saga continues. I changed the nomad policy to

path "secret/" {
  capabilities = ["list", "read"]
}

path "secret/*" {
  capabilities = ["list", "read"]
}

I can now list everything in the vault UI when logging in with the nomad token.

Still getting this when the app starts: Missing: vault.read(secret/db/config)

Next try: add “list” to the db policy the app uses.

9

I edited my db policy to

path "secret/db/*" {
  capabilities = ["list", "read"]
}

Still being told “Missing vault.read”

I’m lost now…

10

Hey, I’ve got Vault working with Nomad. KVv2 stores need an extra item in the “with secret” section.

This:

{{ with secret "kv-v2/db" }}
POSTGRES_USER="{{ .Data.data.user }}"

Should be like:

{{ with secret "kv-v2/data/db" }}
POSTGRES_USER="{{ .Data.data.user }}"

Notice the “/data” between the mount point and the path.

Here’s the documentation on V1 vs V2 KV engines in the template docs: template Stanza - Job Specification | Nomad by HashiCorp

As noted, this also impacts the vault policies within Nomad.

4 Likes
11

Thank you for pointing that out! I’m not sure how I missed that. I think that needs to be called out in more places and much louder! :slight_smile:

2 Likes
12

I agree 100%. On more than one occasion I felt that some lines in the docs should have the old (now deprecated) “marquee” or “blink” tag. Like, “hey reader, you really really need to know this bit here” :grinning_face_with_smiling_eyes:

13

Update: it all works now! Wow…that took WAAAAAY too long to solve…

2 Likes
14

The best part is the lack of error messages in either Vault or Nomad, at least that I saw. I just happened to have come across that during my setup.

Glad it’s working for you now!

15

The only “error” I got was “Missing vault.read,” but yeah, that was annoying.

It may have to do with the fact that the policies can be created before the mount or path is created, so it can’t check path validity at mount time. But a better error would certainly help. However, that would be hard too…because it’s a case of “the path doesn’t match,” which, technically, isn’t an error…it’s just “no privs because no path match.” Maybe the policy shouldn’t have been tied to the underlying api. Hmm…

16

I think a solution might be for vault to return an error that the mandatory /data for a KVv2 engine wasn’t included in the request.

Wouldn’t help on the client side, but the vault server should know the engine type and that that is a requirement from the API.

1 Like
17

Hello

I face the same issue and I spent 2 hours to find out where I did something wrong. I don’t get it.

I followed this and this tutorial. (by the way, why do we have two different tutorials for this?!)

server.hcl (starts both server and client)

server {
  enabled          = true
  bootstrap_expect = 1
}

client {
  enabled = true
}

vault {
  enabled = true
  address = "https://vault.######"
  task_token_ttl = "1h"
  create_from_role = "nomad-cluster"
  token = "s.###########"
}

read-kv policy

path "kv/data/secrets/*" {
  capabilities = ["read"]
}

nomad-cluster role

{
  "allowed_policies": "read-kv",
  "token_explicit_max_ttl": 0,
  "name": "nomad-cluster",
  "orphan": true,
  "token_period": 259200,
  "renewable": true
}

job.nomad

...
template {
        data = <<EOF
AZURE_STORAGE_ACCOUNT={{with secret "kv/data/secrets/azure"}}{{.Data.data.AZURE_STORAGE_ACCOUNT}}{{end}}
EOF
        destination = "secrets/file.env"
        env = true
      }
...

When I run the job, it fails with

Missing: vault.read(kv/data/secrets/azure)

When I create a token from the nomad-cluster role with:

vault token create -role nomad-cluster

and use vault login with that token and read the secrets with vault kv get kv/secrets/azure, I can read the secrets just fine. Only nomad is somehow complaining.

Anyone knows what I’m doing wrong? Can I debug the token that the nomad client uses somehow? That would help a lot…

18

@maaft You can see my working code here: GitHub - jkugler/nomad_poc: A Hashicorp Nomad proof-of-concept using Nomad, Consul, Vault, and Fabio Take a look at the files in shared/config as well as init_cluster.sh

Warning: that’s using an older version of Nomad, so it might not be working out of the box, but all the syntax and permissions was working for me.

I hope that helps!

19

I am seeing the same issue as @maaft. I made it work by omitting the /data part in the with secret call. So that it looks like:

{{with secret "kv/my_secret"}}{{.Data.data.password}}{{end}}

This is on vault v1.9.3. I am not sure why this works different from what is in the current docs.

20

AFAIK, the extra data has to do with whether the Vault KV store is KV v1 or KV v2.

21

maybe this could help …