Prevent port exposure with Nomad + Connect + Docker

Nomad
,
1

I’m trying to understand and set-up Nomad connect with sidecar proxy in combination with docker and health checks. So far I haven’t been successful unless I’m exposing ports.

My current config is:

job "demo-webapp" {
  datacenters = ["dc1"]

  group "demo" {
    count = 3

    network {
      mode = "bridge"

      # No port mapping because Consul Service Mesh / Nomad Connect is used.
    }

    service {
      name = "demo-webapp-conn"
      port = "80"
      task = "server"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.http.rule=Path(`/myapp`)",
      ]

      # This health check fails!
      check {
        type     = "http"
        name     = "nginx-health"
        path     = "/"
        # No `expose = true` here either because that allows bypassing the service mesh.
        interval = "20s"
        timeout  = "10s"
      }

      connect {
        sidecar_service {}
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "nginx"

        # No port mapping here because sidecar should be able to reach the container (same namespace).
        # Sidecar probably exposes on loopback automatically (that's all Nomad/Consul internals).
      }
    }
  }
}

When running this job, I end up with 2/4 failing health checks per instance.

Consul health check screenshot showing a failed status for both the defined nginx-health check and Connect Sidecar Aliasing
Consul health check screenshot showing a failed status for both the defined nginx-health check and Connect Sidecar Aliasing2192×1276 142 KB

It seems like the host is omitted from the request (although I’m not sure if this is the actual cause or just part of how requests are internally handled in the mesh).

The textual output of the health check is as follows.

HTTP GET http://:80/: 400 Bad Request Output: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
 </body>
</html>

Note the http://:80/ part of the result.

I’m not sure if this is a bug or whether I’m misunderstanding how it should work.
Help would be greatly appreciated.

PS: While learning and configuring my cluster I stumbled upon some documentation issues (attached below).

#1 - Missing documentation:

The documentation for the job-specification check block, refers to missing documentation multiple times:

See below for details.

See Using Driver Address Mode for details on address selection.

etc.

The anchor that is being referred to is #using-driver-address-mode, but this does not exist (and after consulting wayback machine it seems like there are no records of such a section for the past few years).

#2 - Unclear documentation:

The documentation for the job-specification network block mention (multiple times):

make sure to bind your workloads to the loopback interface only.

But there’s no guidance on how to achieve this.
Assuming Docker is used;

  • Does this mean that the application running within the Docker container is responsible for binding to the loopback instead of 0.0.0.0? I hope not, because that’d be a massive headache to secure.
  • Is it possible to configure this through the job spec (and if so, how)?
  • Should/can this be done through CNI config instead?