Permission denied except example mongo job

Vault
1

Bonjour, :wave:

Today is my first attempt to use new workload identity on VPS. And… :exploding_head:

After deploying Hashistack, I try mongo job example from doc ( Vault ACL with Nomad Workload Identities | Nomad | HashiCorp Developer )

Job runs without problem, It finds secret in Vault, I just modified secret path to:

MONGO_INITDB_ROOT_PASSWORD={{with secret "passwords/data/default/mongo/config"}}{{.Data.data.root_password}}{{end}}

passwords instead of kv

I plan/apply a first periodic job with secret and force launch
:drum:

 Mar 08, '24 11:42:11 +0100 	Template 	Missing: vault.read(passwords/data/host01/backup)

WHY?! :sob:

This is my job definition:

job "host01-backup" {
  region = "global"
  datacenters = ["dc1"]
  type = "batch"

  node_pool = "default"
  priority = 50
  all_at_once = false
  namespace = "default"


  periodic {
    cron             = "0 1 * * *"
    prohibit_overlap = true
    time_zone = "Europe/Paris"
  }

  group "backup" {

    count = 1

    restart {
      attempts = 10
      interval = "5m"
      delay = "10s"
      mode = "delay"
    }

    network {
      mode = "bridge"

    }

    task "backup" {
      driver = "docker"

      config {
        image = "alpine:latest"


        mount {
          type = "bind"
          target = "/data/host01"
          source = "/data/host01"
          readonly = false
          bind_options {
            propagation = "rshared"
          }
        }

      }

      vault {}      
      
        template {
          change_mode = "noop"
          destination = "secrets/playbook.yml"
          perms = "644"
          data = <<EOH

[... hide ...]


EOH
        }


      template {
        data = <<EOH
ANSIBLE_FORCE_COLOR=TRUE

{{ with secret "passwords/data/host01/backup" }}
AWS_ACCESS_KEY_ID={{ .Data.data.s3_access_key_id }}
AWS_SECRET_ACCESS_KEY={{ .Data.data.s3_secret_access_key }}
RESTIC_PASSWORD={{ .Data.data.restic_password }}
RESTIC_REPOSITORY={{ .Data.data.restic_repository }}
{{ end }}


EOH
        destination = "secrets/file_group_01.env"
        env         = true
	      change_mode = "restart"
      }

      resources {
        cpu    = 512
        memory = 512
      }
    }
  }
}

In syslog I see: 403 permission denied.

Mar  8 11:39:02 pc03-quanticware-com nomad[8397]: URL: GET https://127.0.0.1:8200/v1/passwords/data/host01/backup
Mar  8 11:39:02 host01 nomad[8397]: Code: 403. Errors:
Mar  8 11:39:02 host01 nomad[8397]: * 1 error occurred:
Mar  8 11:39:02 host01 nomad[8397]: #011* permission denied
Mar  8 11:39:02 host01 nomad[8397]:  (retry attempt 4 after "2s")

Why for demo job it works and not for my job?!

This job works find on other host who use old token vault access.

I don’t understand.

And another thing when I click on RUN on Nomad UI:

2 warnings:

* Group "backup" has warnings: 1 error occurred:
    * Task "backup" has warnings: 1 error occurred:
    * 1 error occurred:
    * Identity[vault_default] using env=true without change_mode="restart" may result in task not getting updated identity
* cron is deprecated and may be removed in a future release. Use crons instead

This is part of my Nomad and Vault configuration:

vault {
  enabled = true

  create_from_role = "nomad-workloads"


  default_identity {
    aud = ["vault.io"]
    env  = true
    file = false
    ttl = "1h"
    change_mode = "restart"
  }

  address = "https://127.0.0.1:8200"

  jwt_auth_backend_path = "jwt-nomad"

...

Does it come from the type of job? I try Periodic batch and Mongo is a Service

Can you help me and explain? Thanks :ring_buoy:

2

Solved.