Issues with host keys file when using git artifact

Nomad
1

Hi,

I’m trying to configure nomad to download an artifact from a private git repo on GitHub. I have the added the base64 encoded private key as a string to the sshkey field. However, every time I try to run it, I get this error: failed to download artifact "git::git@github.com:<my_org>/<repo>.git//nginx": getter subprocess failed: exit status 1.

When I check the logs using journalctl -u nomad I see the following:

client.artifact: sub-process: OUTPUT="failed to download artifact: error downloading 'ssh://git@github.com/<my_org>/<repo>.git?depth=1&ref=nomad&sshkey>
client.artifact: sub-process: OUTPUT="warning: templates not found in /usr/share/git-core/templates"
client.artifact: sub-process: OUTPUT="hostkeys_find_by_key_hostfile: hostkeys_foreach failed for /etc/ssh/ssh_known_hosts: Permission denied"
client.artifact: sub-process: OUTPUT="Host key verification failed."
client.artifact: sub-process: OUTPUT="fatal: Could not read from remote repository."
client.artifact: sub-process: OUTPUT=""
client.artifact: sub-process: OUTPUT="Please make sure you have the correct access rights"
client.artifact: sub-process: OUTPUT="and the repository exists."
client.artifact: sub-process: OUTPUT=""

I updated the permissions of the file /etc/ssh/ssh_known_hosts to 777 just to test and I still get the same thing. The file and containing folder is owned by root which I assume is correct because that’s what nomad run as.

My artifact block looks like this:

      artifact {
        source = "git::git@github.com:<my_org>/<repo>.git//nginx"
        destination = "local/nginx"
        options {
          sshkey = "<base64_encoded_ssh_key_here>"
          ref = "nomad"
          depth = 1
        }
      }

Please assist me here, thanks!

2

I think nomad is complaining because it’s not trusting in GitHub.com. Nomad needs you agree to connect to GitHub somehow, while doing interactively we type yes to this question:

The authenticity of host 'github.com (192.30.253.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)?

Make sure you have a github.com entry in your known_hosts file as follow.

artifact {
  # The git:: prefix forces go-getter's protocol detection to use the git ssh
  # protocol. It can also automatically detect the protocol from the domain of
  # some git hosting providers (such as GitHub) without the prefix.
  source      = "git::git@bitbucket.org:example/nomad-examples"
  destination = "local/repo"
  options {
    # Make sure that the system known hosts file is populated:
    # ssh-keyscan github.com | sudo tee -a /etc/ssh/ssh_known_hosts
    # https://github.com/hashicorp/go-getter/issues/55
    sshkey = "${base64encode(file("/path/to/private-key"))}"
  }
}

So I guess your issue will be solved by running the following command:

ssh-keyscan github.com | sudo tee -a /etc/ssh/ssh_known_hosts

Let us know if it worked! :grinning: And mark this as solved in this case. If it’s not solved, let us know as well, someone else might had this issue before.

3

Sorry, I forgot to mention that I had already run that command to add github.com to the host file. The error looks more like it can’t access the ssh_known_hosts file which is why authentication is failing. I’m not sure why, though, because nomad should be running as root which owns the file.

4

Hi @darryldaniel I think you found an actual bug. For context, Nomad 1.5+ runs the artifact download as a child process, and on Linux is sandboxed with the Landlock LSM. The problem is we don’t allow that child process to access /etc/ssh/ssh_known_hosts , as we should.

Either of these workarounds should work for you until we can ship a bug fix

  1. Use $HOME/.ssh/known_hosts instead (where $HOME is of root)
  2. Disable the sandboxing feature by setting artifact.disable_filesystem_isolation in the Client config.

FYI I filed artifact: sandbox should be able to read /etc/ssh/ssh_known_hosts · Issue #17086 · hashicorp/nomad · GitHub based on this post

5

Thanks for the response @seth.hoenig! I’m going to try that out, hoping that works!

6

@seth.hoenig that works with the hard-coded ssh key, thanks! I’m trying now to load an ssh key from a file but no matter what I try I get this error in the nomad logs:

failed to download artifact: error downloading 'ssh://git@github.com/<my_org>/<repo>.git?depth=1&ref=nomad&sshkey=redacted': illegal base64 data at input byte 0

Any idea what this means? does the ssh key need to be on the nomad client or does it get passed through from the machine that is running the job?

7

Unfortunately it currently isn’t possible to specify an ssh (private) key path for a key stored on the Client node. Under the hood Nomad is just using the go-getter library which only enables specifying the actual key itself. If a job spec is making use of the HCL2 file function to load a key, that function is running on the machine where nomad job run is being run on (e.g. your laptop).

[Feature] secrets in artifact stanza · Issue #3854 · hashicorp/nomad · GitHub contains some discussion about managing secret things regarding the artifact stanza; I suspect private key management would fall into that.