Prometheus installation

Nomad
1

Hi :wave:

Nomad: 1.2.6 (install by ansible community role)
Consul: 1.11.3 (install by ansible community role)
Traefik: 2.6.1
Consul connect enable
One node only

I tried to install and configure prometheus with Using Prometheus to Monitor Nomad Metrics | Nomad - HashiCorp Learn

But my target and rules are desperately empty… :worried:

This is my job:

job "prometheus.domain.tld" {
  region = "global"
  datacenters = ["dc1"]
  type = "service"


   update {
     max_parallel     = 1
     canary           = 1
     min_healthy_time = "10s"
     healthy_deadline = "5m"
     auto_revert      = true
     auto_promote     = true
     health_check     = "checks"
     stagger          = "30s"
   }

  group "prometheus" {

    count = 1

    restart {
      attempts = 10
      interval = "5m"
      delay = "10s"
      mode = "delay"
    }

    network {

      mode = "bridge"

        port "prometheus_ui" {
          to = 9090
          static = 9090
        }
      }

    ephemeral_disk {
        size = 300
        sticky = "true"
        migrate = true
    }

    service {
      name = "prometheus-domain-tld"

      port = 9090

      connect {
        sidecar_service {}
      }

      tags = [
"traefik.enable=true",
"traefik.http.routers.prometheusdomaintld.tls=true",
"traefik.http.routers.prometheusdomaintld.tls.certresolver=myresolver",
"traefik.http.routers.prometheusdomaintld.tls.options=mintls12@file",
"traefik.http.routers.prometheusdomaintld.entrypoints=https",
"traefik.http.routers.prometheusdomaintld.rule=Host(`prometheus.domain.tld`)",
"traefik.http.middlewares.prometheusdomaintld.redirectscheme.scheme=https",
"traefik.http.middlewares.prometheusdomaintld.redirectscheme.permanent=true",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.customResponseHeaders.X-Robots-Tag=noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.customResponseHeaders.Strict-Transport-Security=max-age=63072000",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.frameDeny=true",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.browserXssFilter=true",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.contentTypeNosniff=true",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.stsIncludeSubdomains=true",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.stsPreload=true",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.stsSeconds=31536000",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.forceSTSHeader=true",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.accessControlMaxAge=15552000",
"traefik.http.middlewares.prometheusdomaintld-headers.headers.customFrameOptionsValue=SAMEORIGIN",
"traefik.http.middlewares.prometheusdomaintld-basicauth.basicauth.users=user:password",
"traefik.http.routers.prometheusdomaintld.middlewares=prometheusdomaintld-basicauth@consulcatalog,prometheusdomaintld-headers@consulcatalog",
      ]
    }

    service {
      name = "prometheus"
      tags = []
      port = "prometheus_ui"
    }

    task "prometheus" {
      driver = "docker"

      config {
        image = "prom/prometheus:v2.33.3"

        args = [
          "--config.file=/etc/prometheus/prometheus.yml",
          "--storage.tsdb.path=/prometheus",
          "--web.console.libraries=/usr/share/prometheus/console_libraries",
          "--web.console.templates=/usr/share/prometheus/consoles",
          "--web.enable-admin-api"
        ]

        volumes = [
          "local/webserver_alert.yml:/etc/prometheus/webserver_alert.yml",
          "local/prometheus.yml:/etc/prometheus/prometheus.yml",
          "/data/prometheus.domain.tld/data:/prometheus",
        ]

        ports = ["prometheus_ui"]
      }

      template {
        change_mode = "noop"
        destination = "local/prometheus.yml"
        data = <<EOH
---
global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'nomad_metrics'

    consul_sd_configs:
    - server: '172.26.64.1:8500'
      services: ['nomad-clients', 'nomad-servers']

    relabel_configs:
    - source_labels: ['__meta_consul_tags']
      regex: '(.*)http(.*)'
      action: keep

    # scrape_interval: 30s
    metrics_path: /v1/metrics
    params:
      format: ['prometheus']
EOH
      }

      resources {
        cpu    = 256
        memory = 256
      }
    }
  }
}

No problem to run job and access to prometheus interface.

for that:

    consul_sd_configs:
    - server: '172.26.64.1:8500'
      services: ['nomad-clients', 'nomad-servers']

IP is used instead env, I tried with env, same problem.

When I tried inside container

bash-5.1# curl -i 172.26.64.1:4646/v1/metrics
curl: (28) Failed to connect to 172.26.64.1 port 4646 after 129616 ms: Operation timed out

How I can interact container in bridge mode with consul and nomad IP:PORT ?

Thanks