I’m trying to configure nomad to download an artifact from a private git repo on GitHub. I have the added the base64 encoded private key as a string to the sshkey field. However, every time I try to run it, I get this error: failed to download artifact "git::git@github.com:<my_org>/<repo>.git//nginx": getter subprocess failed: exit status 1.
When I check the logs using journalctl -u nomad I see the following:
client.artifact: sub-process: OUTPUT="failed to download artifact: error downloading 'ssh://git@github.com/<my_org>/<repo>.git?depth=1&ref=nomad&sshkey>
client.artifact: sub-process: OUTPUT="warning: templates not found in /usr/share/git-core/templates"
client.artifact: sub-process: OUTPUT="hostkeys_find_by_key_hostfile: hostkeys_foreach failed for /etc/ssh/ssh_known_hosts: Permission denied"
client.artifact: sub-process: OUTPUT="Host key verification failed."
client.artifact: sub-process: OUTPUT="fatal: Could not read from remote repository."
client.artifact: sub-process: OUTPUT=""
client.artifact: sub-process: OUTPUT="Please make sure you have the correct access rights"
client.artifact: sub-process: OUTPUT="and the repository exists."
client.artifact: sub-process: OUTPUT=""
I updated the permissions of the file /etc/ssh/ssh_known_hosts to 777 just to test and I still get the same thing. The file and containing folder is owned by root which I assume is correct because that’s what nomad run as.
I think nomad is complaining because it’s not trusting in GitHub.com. Nomad needs you agree to connect to GitHub somehow, while doing interactively we type yes to this question:
The authenticity of host 'github.com (192.30.253.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)?
Make sure you have a github.com entry in your known_hosts file as follow.
artifact {
# The git:: prefix forces go-getter's protocol detection to use the git ssh
# protocol. It can also automatically detect the protocol from the domain of
# some git hosting providers (such as GitHub) without the prefix.
source = "git::git@bitbucket.org:example/nomad-examples"
destination = "local/repo"
options {
# Make sure that the system known hosts file is populated:
# ssh-keyscan github.com | sudo tee -a /etc/ssh/ssh_known_hosts
# https://github.com/hashicorp/go-getter/issues/55
sshkey = "${base64encode(file("/path/to/private-key"))}"
}
}
So I guess your issue will be solved by running the following command:
ssh-keyscan github.com | sudo tee -a /etc/ssh/ssh_known_hosts
Let us know if it worked! And mark this as solved in this case. If it’s not solved, let us know as well, someone else might had this issue before.
Sorry, I forgot to mention that I had already run that command to add github.com to the host file. The error looks more like it can’t access the ssh_known_hosts file which is why authentication is failing. I’m not sure why, though, because nomad should be running as root which owns the file.
Hi @darryldaniel I think you found an actual bug. For context, Nomad 1.5+ runs the artifact download as a child process, and on Linux is sandboxed with the Landlock LSM. The problem is we don’t allow that child process to access /etc/ssh/ssh_known_hosts , as we should.
Either of these workarounds should work for you until we can ship a bug fix
Use $HOME/.ssh/known_hosts instead (where $HOME is of root)
Disable the sandboxing feature by setting artifact.disable_filesystem_isolation in the Client config.
@seth.hoenig that works with the hard-coded ssh key, thanks! I’m trying now to load an ssh key from a file but no matter what I try I get this error in the nomad logs:
failed to download artifact: error downloading 'ssh://git@github.com/<my_org>/<repo>.git?depth=1&ref=nomad&sshkey=redacted': illegal base64 data at input byte 0
Any idea what this means? does the ssh key need to be on the nomad client or does it get passed through from the machine that is running the job?
Unfortunately it currently isn’t possible to specify an ssh (private) key path for a key stored on the Client node. Under the hood Nomad is just using the go-getter library which only enables specifying the actual key itself. If a job spec is making use of the HCL2 file function to load a key, that function is running on the machine where nomad job run is being run on (e.g. your laptop).
I have some problem with downloads artifact from private git repo.
My env is debian 12 and nomad 1.6.0, nomad service run as root:bin
Nomad logs when trying download
client.artifact: sub-process: OUTPUT="failed to download artifact: error downloading 'ssh://git@gitlab.ltd/repo?sshkey=redacted': /usr/bin/git exited with 128: Cloning into '/var/lib/nomad/alloc/07db8193-4ae6-fda3-87ce-9af31a376865/backend/tmp/getter3382104804/temp'..."
client.artifact: sub-process: OUTPUT="warning: templates not found in /usr/share/git-core/templates"
client.artifact: sub-process: OUTPUT="hostfile_replace_entries: mkstemp: Permission denied"
client.artifact: sub-process: OUTPUT="update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Pe
rmission denied"
client.artifact: sub-process: OUTPUT="error: unable to get random bytes for temporary file: Permission denied"
client.artifact: sub-process: OUTPUT="error: unable to get random bytes for temporary file: Permission denied"
client.artifact: sub-process: OUTPUT="fatal: Unable to create temporary file '/var/lib/nomad/alloc/07db8193-4ae6-fda3-87
ce-9af31a376865/backend/tmp/getter3382104804/temp/.git/objects/pack/tmp_pack_XXXXXX': Permission denied"
client.artifact: sub-process: OUTPUT="fatal: fetch-pack: invalid index-pack output"
Hi @ipseeh can you file an Issue in the Nomad GitHub repo?
At first glance I’m not sure why your config wouldn’t be working; the error of trying to create the temp files shouldn’t happen since we unlock the alloc_dir and task_dir of the task with rwc permissions.
In the meantime you should be able to workaround by setting
And mark this as solved in this case. If it’s not solved, let us know as well, someone else might had this issue before.