Scraping nomad services with prometheus

hi everyone

i have a standard nomad cluster (3 servers, N clients) with ACLs enabled everywhere

i am trying to use the prometheus nomad_sd_config config to discover nomad services to scrape :: Configuration | Prometheus

since i have ACLs enabled everywhere, i need to pass prometheus a legit nomad token it can use to access the nomad API … for the life of me, i cannot find a good example on how i might do that …

THE ASK :: does anyone have a good example of how they are scraping the nomad service catalog, from prometheus, when ACLs are enabled on the nomad API ??

i assume i need to somehow pass a token, and thought i might be able to use some of the prometheus proxy configs – but really not sure … the prometheus consul_sd_config (Configuration | Prometheus) does have a token option – but i don’t see anything similar in the nomad config …

any help is greatly appreciated …

below is an example of my config, and as you might expect, i am getting a 403 (permission denied) error …

ts=2023-04-18T19:42:35.899Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=nomad msg="Unable to refresh target groups" err="Unexpected response code: 403 (Permission denied)"
---
global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:
- job_name: 'prometheus'
  static_configs:
  - targets: ['127.0.0.1:9090']

- job_name: 'node_exporter'
  nomad_sd_configs:
  - server: 'https://nomad.example.com/'
    region: 'us'
    namespace: 'sys'
    refresh_interval: 5s

  relabel_configs:
  - source_labels: ['__meta_nomad_tags']
    regex: '.*,prometheus=node_exporter,.*'
    action: keep

I haven’t tried it myself, as we are ingesting metrics from Nomad cluster using the API endpoint instead of nomad_sd_configs.

In your settings would be like:

- job_name: 'node_exporter'
  nomad_sd_configs:
  - server: 'https://nomad.example.com/'
    region: 'us'
    namespace: 'sys'
    refresh_interval: 5s
    authorization:
      credentials: <YOUR_TOKEN>

Our settings if you wanna try:

      scrape_configs:
        - job_name: integrations/nomad
          metrics_path: /v1/metrics
          params:
            format: ['prometheus']
          static_configs:
            - targets: [<IP_ADDRESS>:4646]

Pd: We have enabled ACLs but it seems v1/metrics is not applying in our cluster! So we are enabling metrics to the world! Thanks to you we have found this issue in our infrastructure.

By the way, did you enable metrics in the nomad settings file when running the nomad daemon?

:man_facepalming:

@hector.medina.cabane – your authorization example is what ended up working … i had gone down the authorization path before – but i kept modifying the type option because i thought i had to set the header to be X-Nomad-Token – based on some nomad curl work i had done in the past … the prometheus docs are a little confusing – but then that was me not realizing i could just use a good old fashioned Authorization: Bearer xxx header …

thanks for re-focusing my efforts

1 Like

I’m really glad I could help somehow :grinning: You also helped me with this post! If you don’t mind, you could mark it as solved ! :wink: