Traefik error with consul connect

Hello,

My job:

job "test.domain.tld" {
  region = "global"
  datacenters = ["dc1"]
  type = "service"
   update {
     max_parallel     = 1
     canary           = 1
     min_healthy_time = "10s"
     healthy_deadline = "5m"
     auto_revert      = true
     auto_promote     = true
     health_check     = "checks"
     stagger          = "30s"
   }

  group "database" {
    network {
      mode = "bridge"
    }

    service {
      name = "database"
      port = "3306"

      connect {
        sidecar_service {}
      }
    }

    task "mariadb" {
        driver = "docker"
        config {
            image = "ghcr.io/linuxserver/mariadb"
            volumes = [
          "/data/test.domain.tld/mysql:/config"
          ]
        }
        env {
            MYSQL_USER = "traccar"
            MYSQL_PASSWORD = "xxx"
            MYSQL_ROOT_PASSWORD = "xxx"
            MYSQL_DATABASE = "traccar"
        }
        resources {
            memory = 300
        }
        
       }
  }

  group "app" {
    network {
      mode = "bridge"

      port "webinterface" {
        to     = 8082
      }
    }

    restart {
      attempts = 10
      interval = "5m"
      delay = "10s"
      mode = "delay"
    }

    ephemeral_disk {
        size = 300
        sticky = "true"
        migrate = true
    }
    
    service {
      name = "traccar"
      port = "webinterface"

      tags = [
                "traefik.enable=true",
                "traefik.http.routers.domaintld.tls=true",
                "traefik.http.routers.domaintld.tls.certresolver=myresolver",
                "traefik.http.routers.domaintld.tls.options=mintls12@file",
                "traefik.http.routers.domaintld.entrypoints=https",
                "traefik.http.routers.domaintld.middlewares=domaintld-headers@consulcatalog",
                "traefik.http.routers.domaintld.rule=Host(`test.domain.tld`)",

                "traefik.http.middlewares.domaintld-headers.headers.customResponseHeaders.Strict-Transport-Security=max-age=63072000",
                "traefik.http.middlewares.domaintld-headers.headers.frameDeny=true",
                "traefik.http.middlewares.domaintld-headers.headers.browserXssFilter=true",
                "traefik.http.middlewares.domaintld-headers.headers.contentTypeNosniff=true",
                "traefik.http.middlewares.domaintld-headers.headers.stsIncludeSubdomains=true",
                "traefik.http.middlewares.domaintld-headers.headers.stsPreload=true",
                "traefik.http.middlewares.domaintld-headers.headers.stsSeconds=31536000",
                "traefik.http.middlewares.domaintld-headers.headers.forceSTSHeader=true",
                "traefik.http.middlewares.domaintld-headers.headers.accessControlMaxAge=15552000",
                "traefik.http.middlewares.domaintld-headers.headers.customFrameOptionsValue=SAMEORIGIN",
                "traefik.http.middlewares.domaintld-headers.headers.sslHost=test.domain.tld",
                "traefik.http.middlewares.domaintld-headers.headers.sslForceHost=true"
      ]

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "database"
              local_bind_port  = 3306
            }
          }
        }
      }
    }

    task "traccar" {
      driver = "docker"

      env {
        
      }

      config {
        image = "traccar/traccar"

        ports = ["webinterface"]
        volumes = [
                    "local/traccar.xml:/opt/traccar/conf/traccar.xml:ro",
                    "/data/test.domain.tld/logs:/opt/traccar/logs:rw"
        ]
      }

      resources {
        cpu    = 1024
        memory = 1024
      }

      template {
        change_mode = "noop"
        destination = "local/traccar.xml"
        data = <<EOH
<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE properties SYSTEM 'http://java.sun.com/dtd/properties.dtd'>

<properties>

    <entry key='config.default'>./conf/default.xml</entry>

    <entry key='database.driver'>com.mysql.cj.jdbc.Driver</entry>
    <entry key='database.url'>jdbc:mysql://127.0.0.1:3306/traccar?serverTimezone=UTC&amp;useSSL=false&amp;allowMultiQueries=true&amp;autoReconnect=true&amp;useUnicode=yes&amp;characterEncoding=UTF-8&amp;sessionVariables=sql_mode=''</entry>
    <entry key='database.user'>traccar</entry>
    <entry key='database.password'>bKQm3L444Dyj</entry>
</properties>

EOH
      }
    }
  }
}

With the right port for database :crazy_face: Job failed - consul connect

Another problem now. The job successfully run, but traefik not working when consul connect is used in job.

With other job, traefik works fine.

In Traefik’s logs, I have this in loop:

time="2021-10-29T13:31:19Z" level=error msg="Router defined multiple times with different configurations in [traccar-traccar-nomad-task-535eef15-831f-9cd2-be74-c419e28fe66b-group-app-traccar-webinterface traccar-traccar-sidecar-proxy-nomad-task-535eef15-831f-9cd2-be74-c419e28fe66b-group-app-traccar-webinterface-sidecar-proxy]" routerName=domaintld providerName=consulcatalog
time="2021-10-29T13:31:34Z" level=error msg="Router defined multiple times with different configurations in [traccar-traccar-nomad-task-535eef15-831f-9cd2-be74-c419e28fe66b-group-app-traccar-webinterface traccar-traccar-sidecar-proxy-nomad-task-535eef15-831f-9cd2-be74-c419e28fe66b-group-app-traccar-webinterface-sidecar-proxy]" providerName=consulcatalog routerName=domaintld

Sorry, I don’t understand.

I see somes topics about traefik and consul connect but on kubernetes and somes other on github issues, but I don’t understand about my case.

Thanks
Have great day!

  • Which traefik-version are you using?
  • The job that is working; any specific differences compared to this one?

I am running traefik w/ consul-connect on Nomad without issues.

This flag should be set (unless it’s enabled by default). I think one of the reasons for the flag could be related to this; sidecar proxies having same tags as service.

"traefik.consulcatalog.connect=true"

The error-message says you are using the router-name for multiple configs. Try another name in this job (traefik.http.routers.domaintld…)?

Hello,

I found solution, but I’ll described what I tried before, I don’t know if it’s good solution.

Traefik 2.5.3

When I add

"traefik.consulcatalog.connect=true"

I don’t have the Router defined multiple times but, nothing. No more rules appears in traefik dashboard.

Same result when I changed router-name.

In consul dashboard, I see all tag. :grimacing:

I read traefik’s docs again with @kds-rune indications. And I found this: Consul Catalog - Traefik

Connect Aware

So like JCVD, “I am AWAAAAARE!”

And add in traefik toml:

[providers.consulCatalog]
  prefix           = "traefik"
  exposedByDefault = false
  connectAware = true

  [providers.consulCatalog.endpoint]
    address = "172.17.0.1:8500"
    scheme  = "http"

And in my job tags:

"traefik.consulcatalog.connect=true"

And It works !

Traefik and TLS to my domain.

Something I noticed. When I add in traefik config toml:

connectByDefault = true

Traefik not working at all.

So don’t add this.

Hope can help someone other.

Do you think it’s good solution ?

Thanks again to @kds-rune ! :slightly_smiling_face:

1 Like

Glad to hear it works.

My config (snip):

# https://doc.traefik.io/traefik/providers/consul-catalog
consulCatalog:
  exposedByDefault: false
  connectAware: true
  connectByDefault: true

This means if you have any “non-connect” endpoints, you would need to set:

"traefik.consulcatalog.connect=false"

…otherwise they won’t work. So it’s which you prefer, really.
Dynamic configs (service-tags) can override static-config defaults.

This drive me crazy…

In Nomad:

|connect-proxy-traccar |xxx.xxx.xxx.xxx:29581 |29581|
|webinterface          |xxx.xxx.xxx.xxx:30211 |8082|

BUT!

Traefik service point to https://xxx.xxx.xxx.xxx:29581 the connect-proxy ?! On database…

However in nomad job above, I put webinterface as port.

this is boring…

Sorry

EDIT:
I tried to add this tag:

"traefik.http.services.domaintld.loadbalancer.server.port=${NOMAD_HOST_PORT_webinterface}"

Traefik send on right port, but I had a beautiful:

Internal Server Error

But When I checked on my browser the page: http://xxx.xxx.xxx.xxx:30211, I get web interface of Traccar without problem.

Traefik send networks to https:// rather than http://

:sob:

Not sure I understand, but:

Connect-services connect via envoy-sidecars (unless native support). This means that you can have two separate jobs/groups connected transparently & encrypted.

Example:
You have an application conencted to your database, using consul-connect. Your application can then access the database like they were on the same host (on 127.0.0.1:port, see the “NOMAD_UPSTREAM” variables).

[database-task] [envoy-sidecar] <--- tls (usually consul-ca) ---> [envoy-sidecar task] [application-task]

You CAN also expose as port on the HOST, but that would be different connection (e.g. <hostname>:<hostport>).

Traefik communicates with the connect-proxies (envoy) when connect is enabled in the config.

The “http/https” for external access, and “http/https” connections to backend services are all configured in traefik.

ok.

Sorry I explained incorrectly. :worried:

Consul connect works fine between groups. app access to database by 127.0.0.1:3306, all fine.

But problem is, when consul connect is used. Traefik change the scheme to backend to https instead http and cause Internal Server Error.

I tried another job without consul connect and all work fine.

Two screenshots:

What can cause scheme change when consul connect is used in job?

And finally why when I used consul connect the port indicate in job service is override by port used by connect-proxy ? And I have to add another traefik tag to force the right port.

"traefik.http.services.domaintld.loadbalancer.server.port=${NOMAD_HOST_PORT_webinterface}"

Hope is understandable! :grinning:
And many thanks to try to help me :hugs:

Ok; you shouldn’t need to configure backend/lb. The following works fine for me.

Example application service-tags (both http & https w/LE):

[
  "traefik.enable=true",
  "traefik.http.routers.EXAMPLE.entrypoints=web",
  "traefik.http.routers.EXAMPLE.rule=Host(`example.com`)",
  "traefik.http.routers.EXAMPLE_TLS.entrypoints=websecure",
  "traefik.http.routers.EXAMPLE_TLS.rule=Host(`example.com`)",
  "traefik.http.routers.EXAMPLE_TLS.tls=true",
  "traefik.http.routers.EXAMPLE_TLS.tls.certresolver=le_test",
  "traefik.http.routers.EXAMPLE_TLS.tls.domains[0].main=example.com",
]

Static config (traefik)

# ...
consulCatalog:
  exposedByDefault: false
  connectAware: true
  connectByDefault: true
  serviceName: traefik
  endpoint:
    address: '172.17.0.1:8500'
    scheme: http
    token: !insecure "{{ env "TRAEFIK_CONSUL_TOKEN" }}"
# ...

Hi @fred-gb , this may be unrelated but a few weeks back I was facing the same issues you are. I’m not using many of the same tags, but I too at one point had multiple services with the same router tag. Once I got past that though, the root of my issue was that my Consul Intentions did not allow Traefik to talk to other services. You should consider adding one or more Intentions to allow Traefik to talk to the service(s) you’re routing to within the mesh.

If your default ACL policy is still set to “allow” though, then my advice may not apply to you. But, this was the final fix I made to get my Traefik routes working.

1 Like

Hello!

@runeron Thanks. My configuration is approximatively the same:

[providers.consulCatalog]
  prefix           = "traefik"
  exposedByDefault = false
  connectAware = true
  connectByDefault = true

  [providers.consulCatalog.endpoint]
    address = "172.17.0.1:8500"
    scheme  = "http"

And the result is backend in https instead of http. :cry:
Thanks to had share your configuration.

Others jobs without consul connect work fin.!

For intentions.
I don’t have any intention configured, I don’t know intentions system, I will learn, but it seems weird to me. Why Traefik need to know that? I just want Traefik end traffic to my backend traccar, it doesn’t need to know communications between app and db. And above all, why backend in https ?!

Thanks for your advice @paladin-devops !

I think I’m close to solution, but only for one “s”, it doesn’t work. :crazy_face:

Thanks again!

Ok, strange.

You’re not overriding backend scheme manually, either? Example:

‘‘‘
“traefik.http.services.BACKENDSERVICE.loadbalancer.server.scheme=https”
‘‘‘

EDIT: Had wrong setting. healthcheck → server

It’s opposite.

I tried this morning:

"traefik.enable=true",
"traefik.http.routers.domaintld.entrypoints=https",
"traefik.http.routers.domaintld.rule=Host(`domain.tld`)",
"traefik.http.routers.domaintld.tls=true",
"traefik.http.routers.domaintld.tls.certresolver=myresolver",
"traefik.http.routers.domaintld.tls.options=mintls12@file",
"traefik.http.services.domaintld.loadbalancer.server.port=${NOMAD_HOST_PORT_webinterface}",
"traefik.http.services.domaintld.loadbalancer.server.scheme=http"

I retire almost of tag, always in https.

backend stay in https.

I think, consul connect integration in Traefik v2.5.3 creates somes changes.

I open thread on Traefik discuss:

I don’t know… :anguished:

Solution!

With help of Kevin Pollet from Traefik, added this tags:

"traefik.consulcatalog.connect=false",
"traefik.http.services.domaintld.loadbalancer.server.port=${NOMAD_HOST_PORT_webinterface}"

So Traefik will not use consul connect for this service and reach backend to http and force port with tag and variable ${NOMAD_HOST_PORT_webinterface} because sidecar_proxy inherit of tags from parents and create problems.

I don’t understand why sidecar_proxy inherit tags. It doesn’t make sense to me.

If you have an explanation.

Thanks a lot!