Hi,
I’ve noticed that if I mount a host volume (exec task type), let’s say on a path /volume1, and I run a task consisting of a bash -c command that lists the contents of /volume1, I can see all the files on the volume.
However, when I put the “bash” executable in /volume1/bash, and I run a task consisting of a /volume1/bash -c command, nomad fails to run it as it’s only looking for the binary in the chroot environment.
Is there any way to run a task which has an executable on the mounted volume?
Interestingly I’ve found this GH issue that explains the setup a bit more:
opened 08:34PM - 09 Jul 20 UTC
type/enhancement
stage/needs-discussion
### Nomad version
```
Nomad v0.10.5
```
### Operating system and Environme… nt details
```
NixOS 20.03
```
### Issue
I want to run nomad jobs whose payloads are installed with the [Nix package manager](https://nixos.org/) which is a bit of an odd-ball in that
it doesn't really adhere to FHS. this means that nomad's strategy of copying
files (or hardlinkin) from `/lib` and `/bin` doesn't work
All the binaries on the system are in a content-adressable read-only path called
the nix store; and the dependencies of those binaries are also in the nix store.
e.g. the `env` binary lives in the `coreutils` package, and refers to the `librt`, `libacl`, `libattr`, `libpthread` and `libc` packages in a content-addressable manner:
```
$ ls -tal /usr/bin/env
lrwxrwxrwx 1 root root 66 Jul 9 20:27 /usr/bin/env -> /nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/env
$ ldd /usr/bin/env
linux-vdso.so.1 (0x00007ffc91799000)
librt.so.1 => /nix/store/xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30/lib/librt.so.1 (0x00007feceb533000)
libacl.so.1 => /nix/store/rq9vqzjqay8fz8qdidmbgs3lqpq0y6zb-acl-2.2.53/lib/libacl.so.1 (0x00007feceb528000)
libattr.so.1 => /nix/store/b3yikpnxly8vgr2c0sspwqckx44hb474-attr-2.4.48/lib/libattr.so.1 (0x00007feceb520000)
libpthread.so.0 => /nix/store/xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30/lib/libpthread.so.0 (0x00007feceb4ff000)
libc.so.6 => /nix/store/xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30/lib/libc.so.6 (0x00007feceb340000)
/nix/store/xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007feceb53f000)
```
If I would want to run a nix-managed binary on nomad, it means
that all the dependencies of the binary should be in the `chroot`. These can be queried in nix using the `--query` command:
```
$ nix-store --query --references /nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/env
/nix/store/xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30
/nix/store/b3yikpnxly8vgr2c0sspwqckx44hb474-attr-2.4.48
/nix/store/rq9vqzjqay8fz8qdidmbgs3lqpq0y6zb-acl-2.2.53
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31
```
I could put those in the `chroot_env` setting in the nomad client config; however the paths on which
each binary depends is obviosuly different and thus dynamic. Using `chroot_env` which is static is thus not an option.
To solve this, I decided to `mount` the `/nix/store` folder into the container that nomad is running using nomad's
`volume` and `volume_mount` support. this is fine to do; as the `/nix/store` is a read-only file-system, containing only public files.
However, with `driver = "exec"` I get an error message:
```
rpc error: code = Unknown desc = file /nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/env not found under path /tmp/NomadClient003956682/ac65b7db-dc46-aa5b-46ba-8aed0bfa55d9/example"
```
which seems to suggest that nomad checks whether the `command` exists _before_ paths in `volume_mounts` are mounted.
If I change to `driver = "docker"` (and not use any executables from the docker container, but again refer to an executable that is in the `volume_mount`) then it works just fine! In the docker driver the check whether the `command` exists seems to happen
_after_ the `volume_mount`s are mounted.
I would really like that `exec` and `docker` would behave the same here. Currently I'm using `docker` driver purely as a workaround, so that I can run bind-mounted executables, but I would rather use the `exec` driver directly instead.
### Proposed fix
change the `exec` driver behaviour to check for the existence of the `command` **after** all paths have been mounted.
so that I do not have to use the `docker` driver just to trick nomad to execute a binary in a mounted path
### Reproduction steps
1. [Install nix pakage manager](https://nixos.org/download.html)
2. Install coreutils using nix:
```
$ nix-build '<nixpkgs>' -A coreutils --no-out-link
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31
```
3. If needed, replace the path to `coreutils` in the `.nomad` files below if they're different. As they're content-addressed; they might vary across OS (Mac vs Linux) and version of the nix package set
4. `sudo nomad agent -dev ./config.hcl`
5. `nomad run example3.hcl` and observe that using `docker` driver `env` gets executed just fine
```
$ nomad run example3.nomad
$ nomad alloc logs 8fd475a2
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=618d19b12602
NOMAD_ALLOC_DIR=/alloc
NOMAD_ALLOC_ID=8fd475a2-0074-8df4-6c74-cf6bb41fc6f4
NOMAD_ALLOC_INDEX=0
NOMAD_ALLOC_NAME=example3.example[0]
NOMAD_CPU_LIMIT=100
NOMAD_DC=dc1
NOMAD_GROUP_NAME=example
NOMAD_JOB_NAME=example3
NOMAD_MEMORY_LIMIT=300
NOMAD_NAMESPACE=default
NOMAD_REGION=global
NOMAD_SECRETS_DIR=/secrets
NOMAD_TASK_DIR=/local
NOMAD_TASK_NAME=example
HOME=/root
```
6. `nomad run example2.hcl` and observer that using `exec` driver it says the `env` executable cannot be found
```
2020-07-09T22:11:28.132+0200 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=ac65b7db-dc46-aa5b-46ba-8aed0bfa55d9 task=example error="failed to launch command with executor: rpc error: code = Unknown desc = file /nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/env not found under path /tmp/NomadClient003956682/ac65b7db-dc46-aa5b-46ba-8aed0bfa55d9/example"
```
### Job file (if appropriate)
config.hcl:
```
client {
host_volume "nix-store" {
path = "/nix/store"
read_only = true
}
}
```
example3.nomad:
```
job "example3" {
datacenters = ["dc1"]
type = "batch"
group "example" {
count = 1
volume "nix-store" {
type = "host"
source = "nix-store"
read_only = true
}
task "example" {
driver = "docker"
volume_mount {
volume = "nix-store"
destination = "/nix/store"
}
config {
# NOTE: This could even be an empty image! like 'scratch' but docker
# needs a non-empty image to start. Nothing from the container itself
# is used
image = "alpine"
command = "/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/env"
}
}
}
}
```
example2.nomad
```
job "example2" {
datacenters = ["dc1"]
type = "batch"
group "example" {
volume "nix-store" {
type = "host"
source = "nix-store"
read_only = true
}
count = 1
task "example" {
driver = "exec"
volume_mount {
volume = "nix-store"
destination = "/nix/store"
}
config {
command = "/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/env"
}
}
}
}
```
### Nomad Client logs (if appropriate)
```
2020-07-09T22:11:28.132+0200 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=ac65b7db-dc46-aa5b-46ba-8aed0bfa55d9 task=example error="failed to launch command with executor: rpc error: code = Unknown desc = file /nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin/env not found under path /tmp/NomadClient003956682/ac65b7db-dc46-aa5b-46ba-8aed0bfa55d9/example"
```
For anyone interested:
you can setup a PATH env variable in the task and reference the command without an absolute path
or you can use bash -c “/volume1/your_command”
Weird workarounds around the nomad exec checker, but both work
