Nomad, consul connect and Traefik: fails to scale with zero downtime

Nomad
1

Hello,
We have a problem concerning the restart of a task inside a task group.
We are using a task group for our nomad leader task and the sidecar proxy (consul connect) here is an example:

job "test-api" {
  meta {
    environment = "dev"
  }

  region = "eu-west-3"
  datacenters = ["eu-west-3a", "eu-west-3b", "eu-west-3c"]
  type        = "service"
  namespace   = "api"

  migrate {
    max_parallel = 1
    health_check     = "checks"
    min_healthy_time = "10s"
    healthy_deadline = "5m"
  }

  update {
    canary           = 1
    max_parallel     = 1
    stagger          = "15s"
    healthy_deadline = "2m"
    auto_promote     = true
    auto_revert      = true
  }

  group "app" {
    count = 1

    network {
      mode = "bridge"
      port "http" {
        to = 8080
      }
    }

    service {
      task = "server"
      name = "agent-service"
      port = 8080
      tags = [ "traefik.enable=true", "traefik.consulcatalog.connect=true" ]
      connect {
        sidecar_service {
          proxy {}
        }

        sidecar_task {

          config {
            auth_soft_fail = true
            image          = "envoyproxy/envoy:v${NOMAD_envoy_version}"

            args = [
              "-c",
              "${NOMAD_SECRETS_DIR}/envoy_bootstrap.json",
              "-l",
              "debug",
              "--disable-hot-restart"
            ]

          }

          resources {
            cpu    = 20
            memory = 50
          }
        }
      }

      check {
        name     = "alive"
        port     = "http"
        type     = "http"
        interval = "10s"
        timeout  = "2s"
        path = "/health"
        protocol = "http"
      }
    }

    ephemeral_disk {
      size = 110
    }

    shutdown_delay = "10s"

    task "server" {
      driver = "docker"
      leader = true

      vault {
        policies    = [ "test-api" ]
        change_mode = "restart"
      }

      config {
        image = "${env.IMAGE}:latest"
        auth_soft_fail = true
      }

      env {
        PORT                 = "8080"
        SERVICE_ENV          = "dev"
      }

      template {
        data        = "{{  with secret \"aws/creds/test-api\" }}\n      AWS_ACCESS_KEY_ID={{.Data.access_key}}\n      AWS_SECRET_ACCESS_KEY={{.Data.secret_key}}\n    {{ end }}\n"
        destination = "secrets/aws.env"
        env         = true
      }

      resources {
        cpu    = 200
        memory = 128
      }
    }
  }
}

We are using Traefik with the consul aware integration in order to route request to our services.
Services are (de)registered thanks to the consul catalog.

We are experiencing some “502 Bad Gateway” when restarting the task due to template stanza re-rendering (vault or consul changes).

Is there a way to de-register the sidecar before the leader task is restarting or restart the full allocation on template re-rendering ?

Thank you very much.