Pull GCR Image fails with Unauthorized

Nomad
1

I’m struggling getting nomad to pull an image from a private GCR instance. I have the gcr cred helper installed and am able to pull the image from command line, however the nomad job fails with 500 unauthorized.

agent config :
plugin “docker” {
config {
auth {
# Nomad will prepend “docker-credential-” to the helper value and call
# that script name.
helper = “gcr”
}
}
}

Task:
“Tasks”: [
{
“Name”: “deploy-container”,
“Driver”: “docker”,
“Config”: {
“auth”: {
“server_address”: “gcr.io”
},
“network_mode”: “host”,
“image”: “gcr.io/bc-corp-it/duoproxy”
}

any help would be appreciated.

2

Hi @arjunasokan-bc :wave:

You mentioned that you are able to pull the image from the command line, but can you confirm if you are able to pull when logged in as the user that Nomad is running (usually root)?

Could you also check if your docker-credential-gcr is properly configured and authenticated? You can do this by running this command (again, as root):

# echo "https://gcr.io" | docker-credential-gcr get

If everything is properly setup it will output some auth info.

On the other hand, if you see this error message:

docker-credential-gcr/helper: could not retrieve GCR's access token: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

It means that the root user can’t authenticate with the registry, so make sure you have your service key in one of the expected locations.