Nomad1.5.0-rc1- failed to download artifact: Error permission denied

Mar 01 08:47:46 ai-149 nomad[3775]: 2023-03-01T08:47:46.674+0800 [ERROR] client.artifact: sub-process: OUTPUT=“failed to download artifact: Error creating temporary directory for archive: mkdir /data/nomad/data/alloc/47d107ae-4210-f145-d56c-7a47475a71dc/app/tmp/getter291344646: permission denied”
Mar 01 08:47:46 ai-149 nomad[3775]: 2023-03-01T08:47:46.674+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=47d107ae-4210-f145-d56c-7a47475a71dc task=app type=“Failed Artifact Download” msg=“failed to download artifact "s3::http://xxxx/ptl/agent-0.29.0_linux_amd64.tar.gz\”: getter subprocess failed: exit status 1" failed=false
Mar 01 08:47:46 ai-149 nomad[3775]: 2023-03-01T08:47:46.675+0800 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=47d107ae-4210-f145-d56c-7a47475a71dc task=app error=“prestart hook "artifacts" failed: failed to download artifact "s3::http://xxxxx/ptl/agent-0.29.0_linux_amd64.tar.gz\”: getter subprocess failed: exit status 1"
Mar 0

Problem resolved due to a local parent directory permission configuration problem

Glad you got it working @chenjpu1!

Just FYI (to any other readers), Nomad 1.5 is significantly more aggressive about sandboxing artifact downloads. Most users shouldn’t notice a difference, but some may need to make minor tweaks to their system configuration to allow artifact downloads to continue to work. More details in the upgrade guide: nomad/upgrade-specific.mdx at v1.5.0-rc.1 · hashicorp/nomad · GitHub

1 Like

Thanks for heads up. I think I am hitting the same issue when trying to download a basic artifact over https or s3.

[ERROR] client.artifact: sub-process: OUTPUT="failed to download artifact: Error creating temporary directory for archive: mkdir /opt/nomad/alloc/<alloc uuid>/server/tmp/getter3346035947: permission denied"

We are running nomad 1.5rc1. What should we adjust to get it working? I tried clicking through docs for settings mentioned, but I get a 404.

Thanks in advance!

If nomad is root boot, need to check the folder permission settings( /opt, /opt/nomad, /opt/ nomad /alloc)

Just to follow up with what @chenjpu1 mentioned, the directory tree should look something like this,

$ pwd && sudo tree -d -p -u --metafirst
/opt/nomad
[drwxr-xr-x root    ]  .
[drwxr-xr-x nomad   ]  └── data
[drwx--x--x root    ]      ├── alloc
[drwx------ root    ]      ├── client
[drwxr-xr-x root    ]      └── server
[drwx------ root    ]          ├── keystore
[drwxr-xr-x root    ]          ├── raft
[drwxr-xr-x root    ]          │   └── snapshots
[drwxr-xr-x root    ]          └── serf

This is the default for the official .deb package, and I just double checked artifact downloading should work in this configuration.

The official 1.5 release is out, and you should be able to read the upgrade guide at