Can't pull images from a private repo on ECR - official documentation won't work

Nomad
1

I am getting crazy and a bit frustrated trying to run a job that uses an image from a private ECR repo. The nomad client is onprem - can’t use IAM roles.

So:

  • I installed amazon-ecr-credential-helper
  • I installed the awscli and set up the credentials in /root/.aws/credentials and the region name in /root/.aws/config. The CLI works.
  • I configured NOMAD client’s config to use a docker auth configuration file:
plugin "docker" {
   config{
     auth {
       config = "/etc/docker-auth.json"
     }
   }
 }
  • The /etc/docker-auth.json file have this contents:
{
 "credHelpers": {
   "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com": "ecr-login"
 }
}
  • My job have this section:
    task "server" {
      driver = "docker"

      config {
        image = "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<MY_IMAGE>:<MY_TAG>"
        ports = ["http"]
      }
    }

So, when I run the job, I get this error:

Failed to find docker auth for repo "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<MY_IMAGE>": docker-credential-ecr-login with input "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<MY_IMAGE>" failed with stderr:

NOTE: if I, as root, do this:

  • run aws ecr get-login-password| docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com
  • copy the auth file from /root/.docker/config.json to /etc/docker-auth.json, then the job RUNS!!!

I really can’t figure out what is not working.

I did a lot of research online and found many posts with similar issues - I tried to apply the solutions found there (e.g put AWS credentials in the /etc/nomad.d/nomad.env file) but nothing works.

Thanks for any help.
Matteo

2

This one took me a while to work out too, what happens is that $HOME isn’t set while running nomad with the systemd files from the (in my case) hashicorp provided rpms. The following blurb is from my cloud-init script to make it all work.

mkdir -p /etc/systemd/system/nomad.service.d
touch /etc/systemd/system/nomad.service.d/nomad.conf
cat > /etc/systemd/system/nomad.service.d/nomad.conf << EOL
[Service]
Environment=HOME=/root
EOL
systemctl daemon-reload
service nomad restart
3

THANK YOU SO MUCH! It did work, you’re a life saver.

I am on Debian 11, but the problem was the same. Thanks again.

Matteo

1 Like