Unable to pull docker images

Nomad
1

I have a nomad agent and a gitlab docker registry.

The following job is unable to pull the docker image

  group "group" {
    count = 1
    task "app" {
      driver = "docker"
      config {
        image = "gitlab.example.com:4567/foo/bar:1234"

The error the nomad server shows is:

Driver Failure	
Failed to pull `gitlab.example.com:4567/foo/bar:1234`: API error (500): Get https://gitlab.example.com:4567/v2/foo/bar/manifests/1234: denied: access forbidden

However if I ssh into the nomad agent, and run the following command, the image is pulled successfully.

docker pull gitlab.example.com:4567/foo/bar:1234

The docker pull works correctly from the agent because the credentials are stored in /root/.docker/config.json

{
	"auths": {
		"gitlab.example.com:4567": {
			"auth": "xxxxxxx="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.1 (linux)"
	}
}

Why is nomad not using /root/.docker/config.json for authentication to my private docker registry?

2

There are 2 ways to tell nomad how to access the docker config file.

  1. Add "docker.auth.config": "/root/.docker/config.json" to nomad config

This is the older, soon to be deprecated way

# Deprecated
"config": {
    "options": {
      "docker.auth.config": "/root/.docker/config.json"
    }
}

Note: Reloading nomad agent is not sufficient, you must do a full restart.

  1. Add Docker plugin to nomad config

Going forward, the non-deprecated method is to use plugins. Add the following to the config.json or config.hcl

  "plugin": [{
    "docker": [{
      "config": [{
        "auth": [{
          "config": "/root/.docker/config.json"
        }]
      }]
    }]
  }]

Note: Reloading nomad agent is not sufficient, you must do a full restart.

2 Likes
3

/root/.docker/config.json must be present in all nomad agent ??

4

Yes. Nomad agent communicates with local docker exec and does not distribute/grab credentials from server.