Vault Workload identity + static ACLs?

Nomad
1

hi,

I’ve got the Nomad Workload Identity Vault tutorial working with the example mongo job, but I was not able to get my production job running. It was all the time failing, with permission denied on Vault logs.

  • Nomad 1.9.1
  • Vault 1.21.0

I’ve tried to extend the nomad-workloads ACL:

# Old style:

# List, create, update, and delete key/value secrets
path "kv/data/docker/nomad/*"
{
  capabilities = ["read", "list"]
}

path "kv/metadata/data/docker/nomad/"
{
  capabilities = ["list"]
}

# List, create, update, and delete key/value secrets
path "kv/data/acme_certs/*"
{
  capabilities = ["read", "list"]
}

path "kv/metadata/data/acme_certs/*"
{
  capabilities = ["read","list"]
}

# New Style:

path "kv/data/docker/nomad/{{identity.entity.aliases.auth_jwt_xxxxx.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_f6c4ee50.metadata.nomad_job_id}}/*" {
  capabilities = ["read"]
}

path "kv/data/docker/nomad/{{identity.entity.aliases.auth_jwt_xxxxx.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_f6c4ee50.metadata.nomad_job_id}}/*" {
  capabilities = ["read"]
}

path "kv/metadata/data/docker/nomad/{{identity.entity.aliases.auth_xxxxx.metadata.nomad_namespace}}/*" {
  capabilities = ["list"]
}

path "kv/metadata/*" {
  capabilities = ["list"]
}

My Job looks like this:

...
      template {
        data        = <<EOH
          {{with secret "kv/docker/nomad/service_ui/secrets"}}
          {{range $key, $value := .Data.data}}
          {{$key}}={{$value}}{{end}}
          {{end}}
          EOH
        destination = "secrets/file.env"
        env         = true
      }
...
      template {
        data        = "{{ with secret \"kv/acme_certs/example\" }}{{ .Data.data.key }}{{ end }}"
        destination = "certs/server.key"
        change_mode = "restart"
        perms       = "0644"
        splay       = "1m"
      }

      template {
        data        = "{{ with secret \"kv/acme_certs/example\" }}{{ .Data.data.crt }}{{ end }}"
        destination = "certs/server.crt"
        change_mode = "restart"
        perms       = "0644"
        splay       = "1m"
      }

      template {
        data        = "{{ with secret \"kv/acme_certs/example\" }}{{ .Data.data.dh_pem }}{{ end }}"
        destination = "certs/dh.pem"
        change_mode = "restart"
        perms       = "0444"
        splay       = "1m"
      }

The mongo example looks like this:

      template {
        data        = <<EOF
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD={{with secret "kv/data/docker/nomad/corp-services/mongo/config"}}{{.Data.data.root_password}}{{end}}
EOF
        destination = "secrets/env"
        env         = true
      }

and I see, that I do not have the extra data (kv/data//…) . I tried different combinations .. but no luck.

What do I miss?

cu denny

2

Is this what you are following:

?

Can you also please confirm your KV secrets engine config? I think from the example policy its:

KV v2 mounted at path kv
Secret stored in the KV v2 secrets engine named docker/nomad/ with possibly multiple secrets (based on *)?

3

yes, this howto ..

and I have from the example above:


MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD={{with secret "kv/data/docker/nomad/corp-services/mongo/config"}}{{.Data.data.root_password}}{{end}}
EOF

$ vault kv list kv/ |grep docker
docker/

Question is for me, how I can add static ACLs .. or why it doesn’t work.. I see no relations besides the role and the ACL policy.

cu denny