Nomad fails to handle tempfs for /secrets and they stay empty

Nomad
1

Hi!

We are running nomad (latest version) as priviliged container using podman on CoreOS (latest version) and want to write secrets as templates.

"Templates":[
                               {
                                  "EmbeddedTmpl":"my data",
                                  "DestPath":"${NOMAD_SECRETS_DIR}/config/config.json",
                                  "Perms":"400",
                                  "ChangeMode":"noop"
                               }

nomad executed as:

ExecStart=/bin/podman run --name %N --conmon-pidfile %t/%N.pid --cgroups disabled --log-driver journald --pull never --image-volume tmpfs -d -q --net=host --cgroupns=host --privileged -v /tmp:/tmp -v /run/podman/podman.sock:/run/podman/podman.sock -v /srv/containers/nomad:/srv/containers/nomad -v /etc/nomad:/nomad/config -v /root/.docker/config.json:/nomad/.docker/config.json -v /etc/pki:/etc/pki -v /etc/os-release:/etc/os-release -v /lib/modules:/lib/modules noenv/nomad agent -config=/nomad/config -node=${AFTERBURN_AWS_INSTANCE_ID}

if i register / restart / update the service there is a allocation directory on the host system being created containing local private and secrets all directories being empty.

nomad log shows that the templates are being rendered.

the allocation container shows that it mounts the secrets from the hosts filesystem equally to local

inside the container the secrets directory is also empty.

if i systemctl restart nomad the secrets are being written to the host filesystem and then they are visible inside the container

if i switch from secretsto local for the template everything works as expected.

i already tried

ProtectSystem=false

ProtectHome=false

sudo setenforce 0

2

i can now also reproduce this issue locally.

/etc/nomad/server.hcl

data_dir = "/srv/containers/nomad"
plugin_dir = "/nomad/data/plugins"
disable_update_check = true
log_level = "INFO"
leave_on_terminate = true
leave_on_interrupt = true
bind_addr = "0.0.0.0"

server {
  enabled = true
  bootstrap_expect = 1
  rejoin_after_leave = true
  default_scheduler_config {
    memory_oversubscription_enabled = true
    scheduler_algorithm = "spread"
  }
  server_join {
    retry_interval = "10s"
    retry_max = 0
  }
}
client {
  enabled = true
  reserved {
    memory = 200
  }
  options {
    "driver.allowlist" = "podman,exec"
    "fingerprint.denylist" = "env_aws,env_gce,env_azure"
  }
  server_join {
    retry_interval = "10s"
    retry_max = 0
  }
}
plugin "nomad-driver-podman" {
  config {
    socket_path = "unix://run/podman/podman.sock"
    client_http_timeout = "5m"
    volumes {
      enabled = true
      selinuxlabel = "z"
    }
  }
}

/srv/containers/nomad# ls -lisah
total 0
61185425 0 drwxr-xr-x. 1 root root 34 15. Sep 15:30 .
61185424 0 drwxr-xr-x. 1 root root 10 15. Sep 15:07 ..
61197615 0 drwx--x--x. 1 root root 72 15. Sep 16:51 alloc
61197613 0 drwx------. 1 root root  0 15. Sep 15:30 client
61197611 0 drwx------. 1 root root 16 15. Sep 15:30 server


the container `noenv/nomad` has the podman plugin already installed

sudo su -
podman system service -t 0 &
podman run -d --rm --name nomad \
  --network host \
  --cgroupns=host \
  --privileged \
  --log-driver journald \
  -v /srv/containers/nomad:/srv/containers/nomad -v /etc/nomad:/nomad/config \
  -v /tmp:/tmp \
  -v /run/podman/podman.sock:/run/podman/podman.sock \
  -e NOMAD_ADDR=http://localhost:4646 \
  noenv/nomad \
  agent -dev -config=/nomad/config

register job

job "podman-job" {
  datacenters = ["dc1"]
  type = "service"
  group "example" {
    task "my-task" {
      driver = "podman"
      config {
        image = "busybox:latest"
        command = "sh"
        args = ["-c", "cat secrets/test.txt; sleep infinity"]
      }
      template {
        destination = "secrets/test.txt"
        data = "hello world"
      }
    }
  }
}

log output:

2025-09-15T16:51:45.951503902+02:00 stderr F cat: can't open 'secrets/test.txt': No such file or directory

if using execit works:

job "exec-secrets-job" {
  group "esj-g" {
    task "mytask" {
      driver = "exec"
      config {
        command = "/bin/bash"
        args = ["-c", "cat secrets/yourfile.conf; sleep infinity"]
      }
      template {
        destination = "secrets/yourfile.conf"
        data = "test content"
      }
    }
  }
}

log output:

test content