Nomad secrets folder empty in Docker container

Nomad
1

I’m facing an issue with Nomad’s NOMAD_SECRETS_DIR folder allocation for a Docker container.

I’ve got the following template stanza in my job description:

template {
    destination = "/local/pki/certs/my_certificate.crt"
    perms = "440"
    data = <<EOF
[ CERTIFICATE CONTENT ]
EOF
}

template {
    destination = "/secrets/pki/private/my_key.key"
    perms = "400"
    data = <<EOF
[ KEY CONTENT ]
EOF
}

With that, from the UI, when I look at the files allocated to the task, I can see that the file for the secret key is created.

nomad secrets files

However, the secrets folder is completely empty in the running Docker container:

# nomad alloc exec -i -t -task lb0 52a13d48 /bin/sh
/ # ls -al /secrets
total 0
drwxr-xr-x. 2 root   root    6 Feb 11 11:31 .
drwxr-xr-x. 1 root   root   70 Feb 11 11:31 ..

And if I look at the allocation folder on the system itself, the folder is also empty:

# ls -al /srv/nomad/alloc/52a13d48-b073-39b4-4b79-10225aece50e/lb0/secrets/
total 0
drwxr-xr-x. 2 root   root    6 Feb 11 12:31 .
drwxrwxrwx. 5 nobody nobody 45 Feb 11 12:31 ..

Where is Nomad finding the files in the secrets folder? Why is it empty in the container?

2

Hi @spack971,

Do you happen to have any Docker volume or a task volume_mount defined that could be getting mounted on top of your alloc’s secrets dir?

I tried a minimal example based on the templates you provided and it seems to work:

job "example" {
  datacenters = ["dc1"]

  group "example" {
    task "secrets" {
      driver = "docker"

      config {
        image   = "alpine:3"
        command = "/bin/ash"
        args    = ["-c", "while true; do find /secrets /local -type f -exec cat {} +; sleep 1; done"]
      }

      resources {
        cpu    = 10
        memory = 30
      }

      template {
        destination = "/local/pki/certs/my_certificate.crt"
        perms       = "440"
        data        = <<EOF
[ CERTIFICATE CONTENT ]
EOF
      }

      template {
        destination = "/secrets/pki/private/my_key.key"
        perms       = "400"
        data        = <<EOF
[ KEY CONTENT ]
EOF
      }
    }
  }
}

$ nomad alloc logs 275a2cc2
[ KEY CONTENT ]
[ CERTIFICATE CONTENT ]
[ KEY CONTENT ]
[ CERTIFICATE CONTENT ]
[ KEY CONTENT ]
[ CERTIFICATE CONTENT ]
[ KEY CONTENT ]
[ CERTIFICATE CONTENT ]
[ KEY CONTENT ]
[ CERTIFICATE CONTENT ]
[ KEY CONTENT ]
[ CERTIFICATE CONTENT ]
...
3

Hi @lgfa29,

Indeed to made me remember that it was working before. Strangely, Nomad does not really like when I mount the filesystem read-only. :smiley:

I had put the following into the systemd unit file:

ProtectSystem=full
ProtectHome=read-only

Which remount some system folders read-only before launching the process. This was interfering with Nomad in a way.

1 Like