Nomad Consul Connect Terminating Gateway and auth_soft_fail

Hello,
I am trying to set up a terminating gateway nomad job to allow connecting to an AWS RDS instance but because we use ECR nomad can not download the Envoy image.

I’m hoping someone can tell me where I can set auth_soft_fail in a terminating gateway nomad job.
In regular connect jobs I have been setting it here and it is working:

Job {
group {
service {
connect {
sidecar_service {}
sidecar_task {
config {
auth_soft_fail = true
}
}
}
}
}

But I haven’t found a way to add that config in a terminating gateway so that it can pull images from Dockerhub after failing on ECR. I’ve tried it in lots of places but keep getting errors from nomad plan that it isn’t allowed in that section.

Reading from: gateway Stanza - Job Specification | Nomad by HashiCorp

Job {
group {
service {
connect {
gateway {
proxy { }
terminating {
service {
name = “RDS instance”
}
}
}
}
}
}
}

Any help would be greatly appreciated!

Hey @plesher

I see this is your first post, so welcome to the community!

It looks like you just need to wrap that auth_soft_fail inside of an auth stanza, like so:

plugin "docker" {
  config {
    endpoint = "unix:///var/run/docker.sock"

    auth {
      config = "/etc/docker-auth.json"
      helper = "ecr-login"
    }
   }
}

Documentation for this is located in the docker driver section, so I get how that can be a little confusing. I’d recommend reading more about the auth stanza and the auth_soft_fail arg.

Also, you said you weren’t able to get Nomad to authenticate with ECR? Maybe we can fix that issue as well so you don’t have to failover to a public envoy image. If you’re interested i’m happy to help :slight_smile:

Hey @Amier
Thank you very much for getting back to me and welcoming me to the community!

Authentication with ECR is working fine and I’m able to pull images/containers from ECR with no issue. I have the config you mentioned in the local nomad config on each docker server.

For jobs that need images/containers from dockerhub, I have to add the auth_soft_fail in the job so that when it doesn’t find the image/container in ECR, it will look in dockerhub, otherwise the job fails.

In the documentation you linked to it says:
" If you set an auth helper, it will be tried for all images, including public images. If you mix private and public images, you will need to include auth_soft_fail=true in every job using a public image."

The problem is that I can not find the correct place, in the job file, to put the auth_soft_fail=true so that the job can download envoy container from dockerhub.
Normally I put it in the job file like:

 job {
  group {
    service {
      connect {
        sidecar_service {}

        sidecar_task {
          config {
            auth_soft_fail = true
          }

But every place I try it doesn’t work.
Error with no auth_soft_fail:
Driver Failure Failed to find docker auth for repo “envoyproxy/envoy”: docker-credential-ecr-login with input “envoyproxy/envoy” failed with stderr:

It looks like I might have gotten it to work. At least the job starts envoy and registers the terminating gateway in consul.

job {
  group {
    service {
      connect {
         sidecar_task {
           config {
              auth_soft_fail = true
            }
          }
         gateway {
              proxy { }
              terminating {
                 service {
                      name = "terminating-gateway"

Sorry about the confusion, I really thought I had tried that location/setup before and it had failed.

Thanks!