Deploying Docker container via ECR "Failed to find docker auth for repo"

Nomad
1

I’m having issues deploying a Docker container hosted on a private ECR registry to Nomad.

The job always fails with the following message: (haven’t found any more logs on this)

Driver Failure:
Failed to find docker auth for repo “<aws_id>.dkr.ecr.<aws_region>.amazonaws.com/”: docker-credential-ecr-login with input “<aws_id>.dkr.ecr.<aws_region>.amazonaws.com/<image_name>” failed with stderr:

I’m configuring the Docker authentication via the plugin stanza in my client configs.

At first I’ve tried to use AWS configuration files (in both root & nomad user home directories) but this does not seem to work in nomad although docker-credential-ecr-login returns valid authentication credentials.

$ export HOME=/home/nomad
$ echo "<aws_id>.dkr.ecr.<aws_region>.amazonaws.com/<image_name>"|docker-credential-ecr-login get

The docker-credential-ecr-login binary is placed in /user/bin so it is available to the nomad user.

Adding the AWS authenication credentails to the nomad systemd service via environment variables (suggested in this issue) also did not change anything

[Service]
Environment=AWS_ACCESS_KEY_ID=***
Environment=AWS_SECRET_ACCESS_KEY=***

… but still works in CLI…

$ export AWS_ACCESS_KEY_ID=***
$ export AWS_SECRET_ACCESS_KEY=***
$ echo "<aws_id>.dkr.ecr.<aws_region>.amazonaws.com/<image_name>"|docker-credential-ecr-login get

Update: I’ve discovered that the nomad systemd service specifies an EnvironmentFile at /etc/nomad.d/nomad.env. Appending the AWS env vars to this file also did not change anything.

How can I debug the output of Nomads Docker authentication command since I’m not gaining any more information than failed with stderr:?

Thanks!

AWS CLI Version: aws-cli/1.18.69 Python/3.8.10 Linux/5.4.0-104-generic botocore/1.16.19
Nomad version: Nomad v1.2.6
docker-credential-ecr-login version: 0.3.1
Docker version: 20.10.13

job.hcl

job "<name>" {
  datacenters = ["dc1"]

  group "web" {
    count = 2

    task "<app>" {
      driver = "docker"

      config {
        image = "<aws_id>.dkr.ecr.<aws_region>.amazonaws.com/<image_name>:<tag>"
      }
    }
  }
}

server.hcl

datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"

advertise {
  # ...
}

server {
  enabled = true
}

client {
  enabled = true
}

plugin "docker" {
  config {
    auth {
      config = "/etc/docker/config.json"
    }
  }
}

There’s also a (deprecated) way of configuring the Docker plugin via client.options.*** which also did not work for me.

/etc/docker/config.json

{
    "credHelpers": {
        "<aws_id>.dkr.ecr.<aws_region>.amazonaws.com": "ecr-login"
    }
}
2

Hello,
It can be challenging to find out how to do it but it’s simple.
Here is an example how you can do it. Note that in this example root is the user that runs Nomad but you can change it to match your setup:

Install amazon-ecr-credential-helper package on the system.

create a /root/.aws/credentials file with your AWS env vars into.

[default]
aws_access_key_id = <aws_access_key_id>
aws_secret_access_key = <aws_secret_access_key>

Create the /root/.docker/config.json file with the following conf:

{
  "credHelpers": {
    "<url_of_the_registry>": "ecr-login"
  }
}

In clients.hcl add the docker plugin authentification:

plugin "docker" {
    config {
        auth {
            config = "/root/.docker/config.json"
        }
    }
}

Finally don’t forget to restart your nomad client :slight_smile:

Unless I forget something you should be able to pull images from your private ECR registry :+1: