# Nomad job fails with "prestart hook \\"task\_dir\\" failed: mount: operation not permitted" **URL:** \ **Category:** Nomad\ **Created:** [February 27, 2023, 12:00am UTC](https://discuss.hashicorp.com/t/nomad-job-fails-with-prestart-hook-task-dir-failed-mount-operation-not-permitted/50694 "2023-02-27T00:00:15Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sammy676776](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@sammy676776](https://discuss.hashicorp.com/u/sammy676776)\ **Post date:** [February 27, 2023, 12:00am UTC](https://discuss.hashicorp.com/t/nomad-job-fails-with-prestart-hook-task-dir-failed-mount-operation-not-permitted/50694/1 "2023-02-27T00:00:15Z")
I have created a brand new cluster in linux and all the agents and servers are communicating well and showing their status in UI as well . Consul is also running fine . However when I submit a job I get the following errors. Please note I am running as ROOT so there should not be any permission issue as such . ```auto > 023-02-26T18:32:46.349-0500 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=6a633002-88b2-3f74-f8f1-ae13592072fd task=web error="prestart hook \"task_dir\" failed: mount: operation not permitted" > 2023-02-26T18:32:46.349-0500 [INFO] client.alloc_runner.task_runner: not restarting task: alloc_id=6a633002-88b2-3f74-f8f1-ae13592072fd task=web reason="Error was unrecoverable" > 2023-02-26T18:32:46.414-0500 [INFO] client.gc: marking allocation for GC: alloc_id=6a633002-88b2-3f74-f8f1-ae13592072fd ``` I have tried diff kind of jobs including Java and simple shell script or just sleep command like here and they all give same errors ```auto job "test" { datacenters = ["dc1"] group "group1" { task "sleep" { driver = "raw_exec" config { command = "sleep" args = ["infinity"] } resources { cpu = 10 memory = 10 } } } } ``` When I look at “nomad alloc status 831f9a6b” it shows the following ```auto ID = 831f9a6b-61ba-9c5a-8b5e-b8d9a172c61f Eval ID = a10d4f4d Name = test.group1[0] Node ID = 15ceea08 Node Name = Job ID = test Job Version = 6 Client Status = failed Client Description = Failed tasks Desired Status = run Desired Description = Created = 1m28s ago Modified = 1m27s ago Deployment ID = 6b2df708 Deployment Health = unhealthy Reschedule Eligibility = 32s from now Task "sleep" is "dead" Task Resources CPU Memory Disk Addresses 10 MHz 10 MiB 300 MiB Task Events: Started At = N/A Finished At = 2023-02-26T23:56:34Z Total Restarts = 0 Last Restart = N/A Recent Events: Time Type Description 2023-02-26T18:56:34-05:00 Not Restarting Error was unrecoverable 2023-02-26T18:56:34-05:00 Task hook failed task_dir: mount: operation not permitted 2023-02-26T18:56:34-05:00 Task Setup Building Task Directory 2023-02-26T18:56:34-05:00 Received Task received by client ``` ---
**Author:** ![seth.hoenig](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/seth.hoenig/32/27213_2.png) [@seth.hoenig](https://discuss.hashicorp.com/u/seth.hoenig)\ **Post date:** [February 27, 2023, 5:41pm UTC](https://discuss.hashicorp.com/t/nomad-job-fails-with-prestart-hook-task-dir-failed-mount-operation-not-permitted/50694/2 "2023-02-27T17:41:52Z")
Hi @sammy676776 can you describe what host OS you are running Nomad on? And can you post the output of `stat ` for the path you have set for `client.alloc_dir`? > **[client Block - Agent Configuration | Nomad | HashiCorp Developer](https://developer.hashicorp.com/nomad/docs/configuration/client#alloc_dir)** > > The "client" block configures the Nomad agent to accept jobs as assigned by > the Nomad server, join the cluster, and specify driver-specific configuration. ---
**Author:** ![sammy676776](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@sammy676776](https://discuss.hashicorp.com/u/sammy676776)\ **Post date:** [February 27, 2023, 6:47pm UTC](https://discuss.hashicorp.com/t/nomad-job-fails-with-prestart-hook-task-dir-failed-mount-operation-not-permitted/50694/3 "2023-02-27T18:47:31Z")
I am running on linux I have only setup data\_dir in the Nomad Nodes …and I do see an alloc dir inside . Is this supposed to be set on servers or clients ? ```auto stat /xxx/xxx/nomad_hostname/alloc File: ‘ /xxx/xxx/nomad_hostname/alloc’ Size: 10 Blocks: 0 IO Block: 4096 directory Device: 26h/38d Inode: 27917335686 Links: 2 Access: (0711/drwx--x--x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-02-27 03:42:03.464494997 -0500 Modify: 2023-02-26 20:03:50.233770092 -0500 Change: 2023-02-26 20:03:50.233770092 -0500 ``` ---
**Author:** ![seth.hoenig](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/seth.hoenig/32/27213_2.png) [@seth.hoenig](https://discuss.hashicorp.com/u/seth.hoenig)\ **Post date:** [February 28, 2023, 3:20pm UTC](https://discuss.hashicorp.com/t/nomad-job-fails-with-prestart-hook-task-dir-failed-mount-operation-not-permitted/50694/4 "2023-02-28T15:20:16Z")
The `alloc_dir` only exists on Client nodes - it’ll inherit the path from `data_dir` if not explicitly set. From the output of your `stat` command that directory looks fine. Are you _sure_ your Client agents are running as `root`? Can check with something like ```auto ps -ef | grep 'nomad agent' ``` I _suspect_ this is the underlying call to `mount` that is failing - the `raw_exec` driver otherwise doesn’t do much else with regard to the filesystem. > I created [overhaul error handling in fs\_linux.go · Issue #16275 · hashicorp/nomad · GitHub](https://github.com/hashicorp/nomad/issues/16275) to help debug issues like this one in the future, but at the moment I’m not even sure what else to check for - different filesystems will fail the `mount` syscall for any number of various reasons. ---
**Author:** ![sammy676776](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@sammy676776](https://discuss.hashicorp.com/u/sammy676776)\ **Post date:** [March 3, 2023, 7:00pm UTC](https://discuss.hashicorp.com/t/nomad-job-fails-with-prestart-hook-task-dir-failed-mount-operation-not-permitted/50694/5 "2023-03-03T19:00:28Z")
Thanks @seth.hoenig It appears running as “root” was the problem .My co worker had installed it with a diff user and had couple directories as that user but I thought root should have access to all of it but then looks like “nobody” is default user and perhaps had trouble writing …although most of directories underst there were all rwx …some weird issue but is resolved now running as local user