Chroot Exec on Ubuntu 16.04

Nomad
1

Hey all, attempting to run a simple binary on a few linux hosts a thousand times, and when I attempt to use the exec driver instead of docker, I’m getting the following error:

failed to launch command with executor: rpc error: code = Unknown desc = container_linux.go:346: starting container process caused “setup user: unable to find user nobody: no matching entries in passwd file”

Job file is like this :
job “nondocker” {
constraint {
attribute = “${attr.os.version}”
value = “16.04”
}
type = “batch”
datacenters = [“dc1”]

task “run uuid chroot exec” {
driver = “exec”
config {
command = “/opt/programname_2020.0/bin/program.bin”
args = ["-datadir /Example_Data"]
}
}
}

2

Hi @tehrandom42! That’s interesting… are you running Nomad as root?

3

Yes. And the docker jobs run properly.

4

Ok, I just tried this myself on an Ubuntu 16.04 machine and didn’t get the same error. I’ve got a dummy workload (source here) and I’m running the following job based on what you’ve provided:

job "nondocker" {
  constraint {
    attribute = "${attr.os.version}"
    value     = "16.04"
  }

  type        = "batch"
  datacenters = ["dc1"]

  task "run uuid chroot exec" {
    driver = "exec"

    config {
      command = "/bin/primes"
      args    = ["1000"]
    }
  }
}

This test was on a recent Vagrant box for Ubuntu 16.04 with just the dev agent. Maybe there’s something about the agent/client configuration that’s causing this problem. Can you share your client HCL config?

5

data_dir = “/var/nomad/data”
bind_addr = “0.0.0.0”
ports {
http = 4646
rpc = 4647
serf = 4648
}
client {
enabled = true
network_speed = 1000
servers = [“192.168.102.13:4647”]
options {
“docker.privileged.enabled” = “true”
“driver.raw_exec.enable” = “1”
}
chroot_env {
“/bin” = “/bin”
“/etc/ld.so.cache” = “/etc/ld.so.cache”
“/etc/ld.so.conf” = “/etc/ld.so.conf”
“/etc/ld.so.conf.d” = “/etc/ld.so.conf.d”
“/lib” = “/lib”
“/lib64” = “/lib64”
“/run/resolvconf” = “/run/resolvconf”
“/sbin” = “/sbin”
“/usr” = “/usr”
“/opt” = “/opt”
“/Example_Data” = “/Example_Data”
}
}
consul {
address = “192.168.1.62:8500”
}

6

I don’t know much about programming in general, as I’m an IT guy, but when attempting to build your primes.go example project from github, it’s complaining about import cycle not allowed. I was trying to use your exact job syntax to see if it’s something I screwed up in the job.

edit: using go version go1.6.2 linux/amd64

7

I think I see the issue. When the chroot environment is being created, we bind-mount a default set of paths. This includes /etc/passwd, which defines the Unix user nobody that we run the process under. If I leave that out, I can replicate what you’re seeing:

    2019-11-22T19:07:17.032Z [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=75455c34-6904-3627-88cc-83b4270adbf2 task="run uuid chroot exec" error="failed to launch command with executor: rpc error: code = Unknown desc = container_linux.go:346: starting container process caused "setup user: unable to find user nobody: no matching entries in passwd file""
    2019-11-22T19:07:17.032Z [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=75455c34-6904-3627-88cc-83b4270adbf2 task="run uuid chroot exec" reason="Error was unrecoverable"

You’ll need to add /etc/passwd to the chroot env. This is definitely something we should clearly call out in the docs, so I’ll fix that!

using go version go1.6.2 linux/amd64

That’s a pretty old version of go. But if you don’t feel like standing up a whole golang toolchain (certainly understandable!) and you have Docker handy you could drop that as primes.go in your home directory and then run:

docker run -it --rm -v $(pwd):/tmp golang:1.12 go build -o /tmp/primes /tmp/primes.go

That’ll drop a Linux executable named primes in the home dir, which you could place on the client.

8

Thank you, I will export /etc to the chroot. I was basing my chroot jail on

I just wanted to add /opt and /Example_Data to the default as that’s where my app and data reside. I mistakenly thought that the folders described in that were the full list of default includes.

9

Thank you for taking the time to help me out. I have 2 last questions if you have the time.

  1. When I use a command with args defined, I always get an error in the log saying “unrecognised option” with my arg in ‘single quotes’. Is it passing in the args with ‘’ surrounding them, if so, can I tell it not to? or is there something else I’m missing? I know that running “my_binary -datadir /Path” works properly, but in the above job I get:
    Error: unrecognised option ‘-datadir /Example_Data’
    in the log as it keeps re-starting the job.

  2. Are you aware of a hashicorp/nomad IRC Channel, Slack, Discord, or any other space that users hang out and help each other with questions other than this web forum?

10

When I use a command with args defined…

Usually you’ll want to break the args up, so that it’s ["-datadir", "/Path"]. You can escape quotes with a backslash, ex. ["-datadir", "/\"Path With Some Spaces\""]

Are you aware of a hashicorp/nomad IRC Channel, Slack, Discord, or any other space that users hang out and help each other with questions other than this web forum?

We’re encouraging folks to join this forum so that we have a moderated place for the Hashi community, and this is the best place to find folks from Hashi who work on the products. I think I’ve heard the HangOps Slack has a Nomad channel where you might find some other users, but I couldn’t really say how active it is as I don’t participate there.

11

Breaking up the args did the trick, thank you again for taking up the time.

While I understand Hashicorp wanting to moderate a public facing resource such as this, I was thinking a more informal place where regulars could answer the easy questions to free up you smart guys for the hard questions :wink:

1 Like