My Proxy (Traefik) is unable to load provider config when using a tls/secure Entrypoint on a Nomad Cluster

Hi Everyone,

I’m new to Nomad and I’m trying to configure traefik as a reverse proxy on a Nomad cluster. I’m able to get traefik working properly without a secure/tls entrypoint but as soon as I try to add a tls entrypoint I run into the following issues.

The first issue is I get the following errors (repeated) in stdout

2025-04-14T17:33:34Z ERR Loading configuration, retrying in 7.999285545s error="loading configuration: Unexpected response code: 403 (Permission denied)" providerName=nomad-dev
2025-04-14T17:33:35Z ERR Loading configuration, retrying in 2.990902199s error="loading configuration: Unexpected response code: 403 (Permission denied)" providerName=nomad-default

It seems like for whatever reason Nomad is not accepting the client token from traefik, but only when traefik tries to listen on https. I’ve confirmed that this is not an issue with the client token for Nomad when tls isn’t involved. I’ve also confirmed that my traefik.yml and tls.yml files have 644 permissions on the container.

Secondly it seems that traefik may not be loading/using the provided certificate. When I try to access one of the services traefik should be proxying for I can see that traefik is using it’s default cert (not the provided self signed cert I was hoping for).

FWIW I’ve also tried starting this up in docker but run into the same errors. I realize this seems like more of an issue with traefik than nomad but I was hoping someone could look at my configs and tell me if I’m missing something or just way off.

It’s probably worth noting that I’m using the Nomad built in service discovery (not consul).

Here are my configuration files:

Nomad Client Config

client {
    enabled = true
    preferred_address_family = "ipv4"
    reserved = {
        cpu = 20
        memory = 2000
        disk = 50000
    }

    server_join = {
        retry_join = ["127.0.0.1", "192.168.30.151", "192.168.30.152"]
        retry_max      = 5
        retry_interval = "20s"
    }

    host_volume "tls-certs" {
        path      = "/var/nomad/config/tls"
        read_only = true
    }

    host_volume "traefik-config" {
        path      = "/var/nomad/config/traefik"
        read_only = false
    }
}

Nomad Job file for Traefik

variable "TRAEFIK_CLIENT_TOKEN" {
  type = string
}
job "traefik" {
  datacenters = ["ifs"]
  type        = "service"

  group "traefik" {
    count = 1

    network {
      port  "http"{
         static = 80
      }
      port "https"{
        static = 443
      }
      port  "admin"{
         static = 8080
      }
    }

    service {
      name = "traefik-https"
      provider = "nomad"
      port = "https"
    }

    volume "tls-certs" {
      type = "host"
      source = "tls-certs" # Match the name from the client config
      read_only = true
    }

    volume "traefik-config" {
      type = "host"
      source = "traefik-config" # Match the name from the client config
      read_only = true
    }

    task "server" {
      driver = "docker"
      config {
        image = "traefik:3"
        ports = ["admin", "http", "https"]
        args = [
          "--api.dashboard=true",
          "--api.insecure=true", ### For Test only, please do not use that in production
          "--configFile=/etc/traefik/traefik.yml",
          "--providers.nomad.endpoint.address=http://${attr.unique.network.ip-address}:4646", ### IP to your nomad server 
          "--log.maxbackups=5"
        ]
      }
      volume_mount {
        volume      = "tls-certs"
        destination = "/etc/ssl/traefik"
        propagation_mode = "private"
      }
      volume_mount {
        volume      = "traefik-config"
        destination = "/etc/traefik"
        read_only   = true
      }
      env {
        NOMAD_TOKEN = var.TRAEFIK_CLIENT_TOKEN
      }
    }
  }
}

traefik.yml

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "https"
          scheme: "https"
          permanent: true
  https:
    address: ":443"
    http:
      tls: {}
  traefik:
    address: ":8080"

providers:
  file:
    directory: "/etc/traefik/config"
    watch: true
  nomad:
    endpoint:
      address: "http://192.168.30.150:4646"
      token: "${NOMAD_TOKEN}"
    namespaces:
      - default
      - dev

api:
  dashboard: true
  insecure: true

log:
  level: ERROR

/etc/trafik/config/tls.yml

tls:
  certificates:
    - certFile: "/etc/ssl/traefik/my-cert.crt"
      keyFile: "/etc/ssl/traefik/my-cert.key"
  options:
    default:
      clientAuth:
        clientAuthType: RequireAndVerifyClientCert
        caFiles:
          - "/etc/ssl/traefik/my-cert-ca.crt"

The command I use to start the job on Nomad is:

nomad job run -namespace dev -var=TRAEFIK_CLIENT_TOKEN=${TRAEFIK_CLIENT_TOKEN} -verbose traefik.nomad.hcl

Any help is appreciated

Hello, did you get a chance to test this Traefik configuration using Docker Compose? Here’s an example I created using Compose:

services:
  traefik:
    image: traefik:latest
    command:
      - "--configFile=/etc/traefik/config.yml"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.docker.localhost`)"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.tls.domains[0].main=docker.localhost"
      - "traefik.http.routers.traefik.tls.domains[0].sans=*.docker.localhost"
      - "traefik.http.routers.traefik.service=api@internal"
    restart: unless-stopped
    ports:
      - "8081:8080" # Porta para o dashboard
      - "80:80" # Porta para HTTP
      - "443:443" # Porta para HTTPS
    networks:
      - net
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik/certs:/certs
      - ./traefik/config.yml:/etc/traefik/config.yml
      - ./traefik/dynamic.yml:/etc/traefik/dynamic.yml

networks:
  net:
    name: net
    driver: bridge

config.yml

# Configuração dos EntryPoints
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: ":443"
    http:
      tls: {} # TLS habilitado sem configuração adicional

# Configuração dos Providers
providers:
  file:
    filename: "/etc/traefik/dynamic.yml"
    directory: "/etc/traefik"
    watch: true
  docker:
    network: net
    endpoint: "unix:///var/run/docker.sock"
    watch: true
    exposedByDefault: false
# Configuração da API
api:
  dashboard: true
  insecure: true # Para fins de desenvolvimento; desative em produção

dynamic.yml

http:
  routers:
    traefik:
      rule: "Host(`traefik.docker.localhost`)"
      service: api@internal
      tls:
        domains:
          - main: "docker.localhost"
            sans:
              - "*.docker.localhost"

tls:
  stores:
    default:
      defaultCertificate:
        certFile: "/certs/tls.crt.pem"
        keyFile: "/certs/tls.key.pem"
  certificates:
    - certFile: "/certs/tls.crt.pem"
      keyFile: "/certs/tls.key.pem" 

To generate the certificates, use the following shell command:

openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout tls.key.pem \
  -out tls.crt.pem \
  -subj "/CN=*.docker.localhost" \
  -addext "subjectAltName=DNS:*.docker.localhost,DNS:docker.localhost"

I created the Traefik job nomad that I use in production based on this Compose file.

Here’s a job very similar to the one I use. I don’t use certificates in Traefik because I have Nginx in front of it:

job "traefik-lb" {
  datacenters = ["dc1"]
  type        = "system"

  group "traefik-lb" {
    count = 1


    update {
      max_parallel      = 1
      canary            = 1
      min_healthy_time  = "30s"
      healthy_deadline  = "4m"
      progress_deadline = "5m"
      auto_revert       = true
      auto_promote      = false
    }
    network {
      port "http" {
        static = 11080
        to     = 80
      }

      port "https" {
        static = 11443
        to     = 443
      }

      port "dashboard" {
        to     = 8080
        static = 38080
      }
    }

    service {
      name     = "traefik-lb"
      provider = "nomad"
      port     = "http"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.traefik-lb.rule=Host(`${var.SUBDOMINIO}.${var.DOMINIO}`)",
        "traefik.http.routers.traefik-lb.entrypoints=web-secure",
        "traefik.http.routers.traefik-lb.tls=true",
        "traefik.http.routers.traefik-lb.service=api@internal",
        "traefik.http.services.traefik-lb.loadbalancer.server.port=${NOMAD_HOST_PORT_http}",
        "traefik.http.routers.traefik-lb.middlewares=traefik-auth@file",
      ]
    }

    task "traefik-lb" {
      driver = "docker"

      config {
        image        = "traefik:v3.3.3"
        ports        = ["http", "https", "dashboard"]
        force_pull   = true
        network_mode = "${var.DOCKER_REDE}"

        args = [
          "--configFile=/traefik/configs/config.yml",
        ]

        volumes = [
          "/etc/localtime:/etc/localtime:ro",
          "/etc/timezone:/etc/timezone:ro",
          "/var/run/docker.sock:/var/run/docker.sock:ro",
          "local/configs/config.yml:${var.TRAEFIK_CONFIG_FILEPATH}",
          "local/configs/dynamic.yml:${var.TRAEFIK_DYNAMIC_FILEPATH}",
          "local/basic_auth/htpasswd:${var.TRAEFIK_BASICAUTH_FILEPATH}",
          "${var.TRAEFIK_VOLUME_PATH}/logs:/traefik/logs"
        ]
      }

      template {
        data = <<EOF
global:
  checkNewVersion: false
  sendAnonymousUsage: false

entryPoints:
  web: 
    address: ":{{ env "NOMAD_ALLOC_PORT_http" }}"
    proxyProtocol:
      trustedIPs: "${var.TRAEFIK_IPS_CONFIAVEIS}"
    forwardedHeaders:
      trustedIPs: "${var.TRAEFIK_IPS_CONFIAVEIS}"
    http: 
      redirections: 
        entryPoint:
          to: web-secure
          scheme: https
          permanent: true
  web-secure:
    address: ":{{ env "NOMAD_ALLOC_PORT_https" }}"
    proxyProtocol:
      trustedIPs: "${var.TRAEFIK_IPS_CONFIAVEIS}"
    forwardedHeaders:
      trustedIPs: "${var.TRAEFIK_IPS_CONFIAVEIS}"

ping:
  entryPoint: traefik

api:
  dashboard: true
  insecure: false
  debug: true

log:
  level: INFO
  filePath: "${var.TRAEFIK_LOG_FILEPATH}"

accesslog:
  filepath: "${var.TRAEFIK_ACCESSLOG_FILEPATH}"
  bufferingsize: 100
  fields:
    names:
      StartUTC: drop

providers:
  file:
    directory: ${var.TRAEFIK_PROVIDER_FILE_PATH}
    watch: true  

  docker:
    endpoint: unix:///var/run/docker.sock
    watch: true
    exposedByDefault: false

  nomad:
    endpoint:
      address: ${var.NOMAD_ADDR}
      token: ${var.TRAEFIK_TOKEN}
      tls:
        insecureSkipVerify: true
    exposedByDefault: false

serverstransport:
  insecureSkipVerify: true
EOF

        destination = "local/configs/config.yml"
      }

      template {
        data        = <<EOH
http:
  routers:
    traefik-lb:
      rule: "Host(`${var.SUBDOMINIO}.${var.DOMINIO}`)"
      service: api@internal
      middlewares:
        - traefik-auth@file

    traefik-auth:
      basicAuth:
        usersFile: "${var.TRAEFIK_BASICAUTH_FILEPATH}"
        removeHeader: true
EOH
        destination = "local/configs/dynamic.yml"
        change_mode = "restart"
      }

      template {
        data        = <<EOH
diego:$apr1$qWAQK7VX$QX4NoJnI4TkONANDMcaoT1
EOH
        destination = "local/basic_auth/htpasswd"
        change_mode = "restart"
      }

      resources {
        cpu    = 300
        memory = 256
      }
    }
  }
}

According to the documentation…

There are three different, mutually exclusive (i.e. you can use only one at the same time), ways to define static configuration options in Traefik:

In a configuration file
In the command-line arguments
As environment variables

I had read this before but apparently forgot about it and misunderstood the fact that not only are they mutually exclusive but the static config file is also unable to make use of any environment variable. So if you are going to use a static config file you cannot override or set values with cli arguments or env variables. Furthermore, it seems that the static config file is not capable of reading environment variables of any kind. This means you have to hard code tokens and any other value you wish to be dynamic. I was able to get things working by doing just that, however I don’t like the idea of my traefik.yml file containing hard coded tokens so I’m working on adapting it to command line arguments. it almost works but I am now getting a 404 not found when I try to reach my servivce over https.

@diegovitor I ended up with a similar setup but my main issue was I was trying to use more than one option for the static config settings

FWIW I was able to get past the 404 error and now everything works as expected. Here is what I ended up with (the condensed version):
Nomad job file for traefik (task section only as nothing changed anywhere else)

task "server" {
      driver = "docker"
      config {
        image = "traefik:3"
        ports = ["admin", "http", "https"]
        args = [
          "--entryPoints.web.address=:${NOMAD_PORT_http}",
          "--entrypoints.web.http.redirections.entrypoint.to=websecure",
          "--entrypoints.web.http.redirections.entrypoint.scheme=https",
          "--entrypoints.web.http.redirections.entrypoint.permanent=true",

          "--entryPoints.websecure.address=:${NOMAD_PORT_https}",
          "--entrypoints.websecure.http.tls=true",

          "--entrypoints.traefik.address=:${NOMAD_PORT_admin}",

          "--providers.file.directory=/etc/traefik/config",
          "--providers.file.watch=true",

          "--providers.nomad.endpoint.address=http://${attr.unique.network.ip-address}:4646", 
          "--providers.nomad.endpoint.token=${ var.TRAEFIK_CLIENT_TOKEN }",
          "--providers.nomad.namespaces=default,dev",

          "--api.dashboard=true",
          "--api.insecure=true",

          "--log.level=ERROR",
          "--log.maxbackups=5",
          # "--log.maxsize=100",
          # "--log.maxage=3",
          # "--log.compress=false"
        ]
      }
      volume_mount {
        volume      = "tls-certs"
        destination = "/etc/ssl/traefik"
        propagation_mode = "private"
      }
      volume_mount {
        volume      = "traefik-config"
        destination = "/etc/traefik"
        read_only   = true
      }
    }

My traefik.yml no longer exists since I can’t combine it with the command line flags. I’m not sure but I think the 404 error may have been due to the fact that I originally had --provider.file=true which I think may have caused traefik to ignore the other settings for that provider but that’s just a wild guess. In any case it’s not necessary to have that flag so I removed it.