Secrets from Vault (Missing: vault.read(secrets/xxxxx))

Nomad
1

Hello all:

My environment:
Nomad v1.9.0
Vault v1.18.0

I recently configured Nomad’s Vault integration to get secrets from Vault using this Hashicorp tutorial.
In my proof of concept, I can successfully retrieve a secret in Nomad.

I am attempting to configure a “real” Nomad task. This new Nomad job is having issues retrieving Vault secrets, and I have no idea what the problem could be.

First, here is the Vault policy that I have dedicated to Nomad:

$ vault policy read nomad-workloads
path "secrets/data/{{identity.entity.aliases.auth_jwt_edxxx.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_edxxx.metadata.nomad_job_id}}/*" {
  capabilities = ["read"]
}

path "secrets/data/{{identity.entity.aliases.auth_jwt_edxxx.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_edxxx.metadata.nomad_job_id}}" {
  capabilities = ["read"]
}

path "secrets/metadata/{{identity.entity.aliases.auth_jwt_edxxx.metadata.nomad_namespace}}/*" {
  capabilities = ["list"]
}

path "secrets/metadata/*" {
  capabilities = ["list"]
}

I am using a version 2 kv secrets engine:

$ vault secrets list -format=json | jq '.["secrets/"].options.version'
"2"

Here is the job where I can successfully retrieve a Vault secret

job "mongo" {
  namespace = "default"

  group "db" {
    network {
      port "db" {
        static = 27017
      }
    }

    service {
      provider = "nomad"
      name     = "mongo"
      port     = "db"
    }

    task "mongo" {
      driver = "podman"

      config {
        image = "docker.io/mongo:8"
        ports = ["db"]
      }

      vault {}

      template {
        data        = <<EOF
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD={{with secret "secrets/data/default/mongo/config"}}{{.Data.data.root_password}}{{end}}
EOF
        destination = "secrets/env"
        env         = true
      }
    }
  }
}

Here is the job (trimmed down for brevity) that I having a problem with:

job "influxdb" {
  datacenters = ["homelab"]
  type        = "service"

  group "influxdb" {
    count = 1

    network {
      port "influxdb_ui" {
        to = 8086
      }
    }

    task "influxdb" {
      driver = "podman"
      config {
        image = "docker.io/influxdb:2-alpine"
        ports = ["influxdb_ui"]
      }

      template {
        data = <<EOF
DOCKER_INFLUXDB_INIT_MODE=setup
DOCKER_INFLUXDB_INIT_USERNAME={{with secret "secrets/data/infra/influxdb"}}{{.Data.data.admin_username}}{{end}}
DOCKER_INFLUXDB_INIT_PASSWORD={{with secret "secrets/data/infra/influxdb"}}{{.Data.data.admin_password}}{{end}}
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN={{with secret "secrets/data/infra/influxdb"}}{{.Data.data.admin_token}}{{end}}
DOCKER_INFLUXDB_INIT_ORG=homelab
DOCKER_INFLUXDB_INIT_BUCKET=homelab-bucket
EOF
        destination = "secrets/env"
        env         = true
      }

      vault {}
    }
  }
}

For the InfluxDB job, I get this error:
Template: Missing: vault.read(secrets/data/infra/influxdb)

Does anyone see anything inherently wrong with the InfluxDB job?
Please let me know if you require anything else to help troubleshoot this.

Thank you

2

I figured it out.

Re-reading the setup documentation showed that because the policy is scoped by
secrets/data/<nomad_namespace>/<job_name>/*, my secrets were stored in the wrong part of the secrets engine kv tree.

Recreating the secrets, for example, secrets/default/influxdb/config, and adjusting the template to suit the updated secrets path led to a successful deployment.

Thanks