Can't pull images from a private repo on ECR - official documentation won't work

I am getting crazy and a bit frustrated trying to run a job that uses an image from a private ECR repo. The nomad client is onprem - can’t use IAM roles.


  • I installed amazon-ecr-credential-helper
  • I installed the awscli and set up the credentials in /root/.aws/credentials and the region name in /root/.aws/config. The CLI works.
  • I configured NOMAD client’s config to use a docker auth configuration file:
plugin "docker" {
     auth {
       config = "/etc/docker-auth.json"
  • The /etc/docker-auth.json file have this contents:
 "credHelpers": {
   "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>": "ecr-login"
  • My job have this section:
    task "server" {
      driver = "docker"

      config {
        image = "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION><MY_IMAGE>:<MY_TAG>"
        ports = ["http"]

So, when I run the job, I get this error:

Failed to find docker auth for repo "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION><MY_IMAGE>": docker-credential-ecr-login with input "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION><MY_IMAGE>" failed with stderr:

NOTE: if I, as root, do this:

  • run aws ecr get-login-password| docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>
  • copy the auth file from /root/.docker/config.json to /etc/docker-auth.json, then the job RUNS!!!

I really can’t figure out what is not working.

I did a lot of research online and found many posts with similar issues - I tried to apply the solutions found there (e.g put AWS credentials in the /etc/nomad.d/nomad.env file) but nothing works.

Thanks for any help.

This one took me a while to work out too, what happens is that $HOME isn’t set while running nomad with the systemd files from the (in my case) hashicorp provided rpms. The following blurb is from my cloud-init script to make it all work.

mkdir -p /etc/systemd/system/nomad.service.d
touch /etc/systemd/system/nomad.service.d/nomad.conf
cat > /etc/systemd/system/nomad.service.d/nomad.conf << EOL
systemctl daemon-reload
service nomad restart

THANK YOU SO MUCH! It did work, you’re a life saver.

I am on Debian 11, but the problem was the same. Thanks again.


1 Like