Nomad Load Balancing with Traefik and Consul Connect on the same time

Hello there,

I’m quite new to the Nomad and Consul.

I’m trying to merge those both tutorials:

I have that nomad file:

job "countdash" {
    datacenters = [
        "fra1"]

    group "api" {
        network {
            mode = "bridge"
        }

        service {
            name = "count-api"
            port = "9001"

            connect {
                sidecar_service {}
            }
        }

        task "web" {
            driver = "docker"

            config {
                image = "hashicorpnomad/counter-api:v1"
            }
        }
    }

    group "dashboard" {
        network {
            mode = "bridge"

            port "http" {
                static = 9002
                to = 9002
            }
        }

        service {
            name = "count-dashboard"
            port = "9002"

            tags = [
                "traefik.enable=true",
                "traefik.http.routers.http.rule=PathPrefix(`/count`)",
            ]

            connect {
                sidecar_service {
                    proxy {
                        upstreams {
                            destination_name = "count-api"
                            local_bind_port = 8080
                        }
                    }
                }
            }
        }

        task "dashboard" {
            driver = "docker"

            env {
                COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
            }

            config {
                image = "hashicorpnomad/counter-dashboard:v1"
            }
        }
    }
}

The job is successfully sent and handled by Nomad which is running api and dashboard.
The both count-api and count-dashboard services are connected to Consul.
The count-dashboard can access to count-api (using the private ip adress).
The count-dashboard service is registered to Treafik.

Here’re the issues:

  • I saw two HTTP Services count-dashboard@consulcatalog and count-dashboard-sidecar-proxy@consulcatalog (can the issue be caused by the sidecar-proxy ?).
  • The rule is missing in the http router.

Is anyone have an idea how to make all of this working ?

EDIT 1: If I remove the tag traefik.http.routers.http.rule=PathPrefix:/count, I saw the two services served using their name as Host rule (e.g. for count-dashboard@consulcatalog the rule is Host:count-dashboard)…

EDIT 2 : If I disable the sidecar-proxy and set the rule traefik.http.routers.http.rule=PathPrefix('/count') the reverse-proxy is correctly set but the dashboard is no longer connected to the api…

Thanks,

1 Like

Seems not possible for today :

1 Like

Thanks for looking into this, and for this thread. After spending dozens of hours getting Nomad set up on my cluster, and the jobs working with Connect, it is quite discouraging to see that this is where the road ends :sweat_smile:

I’ll post here again if I find any nice solutions around this.

Not so fast, @AngelOnFira ! The PR @scorsi linked to is basically complete now, at this point I think the traefik team is just reviewing it. I spent some time playing around with my own docker image built from the PR, and it seems to work well. Here is a quick demo of using traefik to reverse proxy to both a connect native service and a service fronted with a sidecar proxy.

job "sample" {
  datacenters = ["dc1"]

  group "traefik" {
    network {
      mode = "bridge"

      port "http" {
        static = 9000
        to     = 9000
      }
    }

    service {
      name = "traefik"
      port = 9000

      connect {
        native = true
      }
    }

    task "traefik" {
      driver = "docker"

      config {
        image = "shoenig/traefik:connect"
        args = [
          "--entrypoints.http=true",
          "--entrypoints.http.address=:9000",

          "--providers.consulcatalog.connectaware=true",
          "--providers.consulcatalog.connectbydefault=false",
          "--providers.consulcatalog.exposedbydefault=false",

          # Nomad will automatically set environment variables for these
          # for Connect native tasks.
          # "--providers.consulcatalog.endpoint.address=<socket|address>"
          # "--providers.consulcatalog.endpoint.tls.ca=<path>"
          # "--providers.consulcatalog.endpoint.tls.cert=<path>"
          # "--providers.consulcatalog.endpoint.tls.key=<path>"
          # "--providers.consulcatalog.endpoint.token=<token>"
        ]
      }
    }
  }

  # An example destination service using a connect sidecar 
  group "kitchen-clock" {
    network {
      mode = "bridge"
    }

    service {
      name = "clock"
      port = "3333"
      tags = [
        "traefik.enable=true",
        "traefik.connect=true",
      ]
      connect {
        sidecar_service {}
      }
    }

    task "server" {
      driver = "docker"
      config {
        image = "shoenig/simple-http:v1"
        args  = ["server"]
      }
      env {
        BIND = "0.0.0.0"
        PORT = 3333
      }
    }
  }


  # An example connect native destination service
  group "uuids" {
    network {
      mode = "bridge"
      port "uid" {
        to = 8999
      }
    }

    service {
      name = "uuid-api"
      port = "uid"
      tags = [
        "traefik.enable=true",
        "traefik.connect=true",
      ]
      connect {
        native = true
      }
    }

    task "uuid-api" {
      driver = "docker"
      config {
        image = "hashicorpnomad/uuid-api:v5"
      }

      env {
        BIND = "0.0.0.0"
        PORT = 8999

        # If using Consul TLS, this is also required
        # until #10805 is fixed
        CONSUL_TLS_SERVER_NAME = "localhost"
      }
    }
  }
}

Using the default Host header routing

curl -H "Host: clock" localhost:9000
the time is 6:14PM
curl -H "Host: uuid-api" localhost:9000
a60fb27d-192d-8638-6dad-768b0681494b
1 Like

@AngelOnFira You could also check out Proxy Ingress to Consul Service Mesh (hashicorp.com) (nginx/haproxy used in examples).

I am eagerly awaiting the @shoenig PR for traefik (yay!), but in the meantime I am using traefik with mTLS ( Traefik Proxy 2.4 Adds Advanced mTLS) for accessing the connect-enabled jobs.

Using info from the first article I linked, I get the ca & certs via consul-template, and I use consul provider for configuring routing in the the nomad job service-tags. I have to use file-provider for services (automatically populates connect services via using consul-template):

traefik file-provider:

# Was unable to set serversTransport w/consul-provider
---
# Using consul-template (nomad traefik job -> task -> template)
http:
  serversTransports:
    # SSL for Consul Connect connections (envoy/consul-connect)
    consulConnect:
      #insecureSkipVerify: true
      rootCAs:
      - /secrets/consul-ca.crt
      certificates:
      - certFile: /secrets/consul.pem
        keyFile: /secrets/consul.key
      forwardingTimeouts:
        dialTimeout: "5s"
        idleConnTimeout: "5s"
  services:
    # Consul Connect Services
    # - Uses consul-template to find any traefik-enabled consul connect service
{{ $allowedTags := parseJSON `["traefik.enable=true"]` -}}
{{- range services -}}
  {{- if and (.Name | contains "sidecar" | not) (containsAny $allowedTags .Tags) }}
    {{.Name}}:
      loadBalancer:
        serversTransport: consulConnect
        servers:
    {{- range connect .Name }}
        - url: "https://{{ .Address }}:{{ .Port }}"
    {{- else }}
        - url: "127.0.0.1:65535" # force a 502 + avoids error on 0 services returned
    {{- end }}
  {{- end -}}
{{- end }}
...

Nomad job → group → service)

# Example service tags in nomad job
tags = [
  "traefik.enable=true",
  "traefik.http.routers.example.entrypoints=websecure",
  "traefik.http.routers.example.rule=Host(`some.example.com`)",
  "traefik.http.routers.example.service=example-connect-backend-service@file",
  "traefik.http.routers.example.tls=true",
  "traefik.http.routers.example.tls.certresolver=le_test",
  "traefik.http.routers.example.tls.domains[0].main=example.com",
  "traefik.http.routers.example.tls.domains[0].sans=some.example.com",
]

…probably other/better ways of doing this, and it does get a bit messy, but it works