Use consul connect with nomad poststop hooks

[dave.bell]

I want to have a batch job signal it has completed. I have built this using a poststop lifecycle task. However, the poststop hook needs to access the service mesh in order to signal.

This is the job file I have:

job "dbell-test" {
  datacenters = [
    "dc1",
    "us-west-2a",
    "us-west-2c",
    "us-west-2d"
  ]

  type = "batch"
  namespace = "imptesting"

  parameterized {
    payload = "required"
    meta_required = [
      "AUTH_BEARER_TOKEN",
      "EXECUTION_ID"
    ]
  }

  group "tg" {
    task "workload" {
      driver = "docker"

      config {
        image = "belljd/happy_path"
      }

      env {
        ACCESS_TOKEN = "${NOMAD_META_AUTH_BEARER_TOKEN}"
        EXECUTION_ID = "${NOMAD_META_EXECUTION_ID}"
      }

      dispatch_payload {
        file = "input.gz"
      }
    }

    task "notify" {
      driver = "docker"

      lifecycle {
        hook = "poststop"
      }

      config {
        image = "alpine/curl"
        args = [
          "--header",
          "Content-Type: application/json",
          "--header",
          "Authorization: Bearer ${NOMAD_META_AUTH_BEARER_TOKEN}",
          "--header",
          "x-s4-env: hp595",
          "--request",
          "POST",
          "http://imptesting-api-gateway.service.consul/executions/${NOMAD_META_EXECUTION_ID}/complete"
        ]
      }
    }

    service {
      name = ""

      connect {
        sidecar_service {
          tags = ["sidecar"]
          proxy {
            upstreams {
              destination_name = "imptesting-api-gateway"
              local_bind_port = 8080
            }
          }
        }
      }
    }

    network {
      mode = "bridge"
    }
  }
}

When I dispatch this batch job the main task runs as expected and the poststop starts up correctly. However, it fails to connect on localhost:8080.

In the nomad UI looking at an allocation of a dispatch I see this in the lifecycle status section:

The consul connect proxy shuts down before the poststop hook runs and thus isn’t available. How do I configure a poststop hook to be able to make use of consul connect?