Prometheus get 403 to consul after migrate to Service Identity

Nomad
,
1

Hi, have problem with prometheus after migrate to Service Identity, my job file:

job "prometheus" {
  datacenters = ["dc1"]
  type        = "service"

  group "monitoring" {
    count = 1

    network {
      port "prometheus_ui" {
        static = 9090
      }
    }

    restart {
      attempts = 2
      interval = "30m"
      delay    = "15s"
      mode     = "fail"
    }

    ephemeral_disk {
      size = 300
    }

    task "prometheus" {
      template {
        change_mode = "noop"
        destination = "local/prometheus.yml"

        data = <<EOH
---
global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:
    consul_sd_configs:
    - server: '{{ env "NOMAD_IP_prometheus_ui" }}:8500'
      datacenter: dc1
      tag_separator: ','
      scheme: http
      services: ['nomad-client', 'nomad']

    relabel_configs:
    - source_labels: ['__meta_consul_tags']
      regex: '(.*)http(.*)'
      action: keep

    scrape_interval: 5s
    metrics_path: /v1/metrics
    params:
      format: ['prometheus']

  - job_name: 'traefik_metrics'
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['traefik.service.consul:8081']
    params:
      format: ['prometheus']
EOH
      }

      driver = "docker"

      config {
        image = "prom/prometheus:latest"

        volumes = [
          "local/prometheus.yml:/etc/prometheus/prometheus.yml",
        ]

        ports = ["prometheus_ui"]
      }

      service {
        name = "prometheus"
        tags = [        ]
        port = "prometheus_ui"

        check {
          name     = "prometheus_ui port alive"
          type     = "http"
          path     = "/-/healthy"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

When i start job - get error:

failed to setup alloc: pre-run hook "consul" failed: 1 error occurred: * failed to derive Consul token for task prometheus: Unexpected response code: 403 (Permission denied)

In syslog

consul[788]: 2024-03-18T07:58:47.025Z [ERROR] agent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:44262 error="Permission denied"
consul[788]: agent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:44262 error="Permission denied"

Before migrating to Service Identity all works well. Other jobs works fine. What a promblem with prometheus? Maybe need to add some policies?

1 Like