hi everyone
i have a standard nomad cluster (3 servers, N clients) with ACLs enabled everywhere
i am trying to use the prometheus nomad_sd_config
config to discover nomad services to scrape :: Configuration | Prometheus
since i have ACLs enabled everywhere, i need to pass prometheus a legit nomad token it can use to access the nomad API … for the life of me, i cannot find a good example on how i might do that …
THE ASK :: does anyone have a good example of how they are scraping the nomad service catalog, from prometheus, when ACLs are enabled on the nomad API ??
i assume i need to somehow pass a token, and thought i might be able to use some of the prometheus proxy
configs – but really not sure … the prometheus consul_sd_config
(Configuration | Prometheus) does have a token
option – but i don’t see anything similar in the nomad config …
any help is greatly appreciated …
below is an example of my config, and as you might expect, i am getting a 403 (permission denied) error …
ts=2023-04-18T19:42:35.899Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=nomad msg="Unable to refresh target groups" err="Unexpected response code: 403 (Permission denied)"
---
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['127.0.0.1:9090']
- job_name: 'node_exporter'
nomad_sd_configs:
- server: 'https://nomad.example.com/'
region: 'us'
namespace: 'sys'
refresh_interval: 5s
relabel_configs:
- source_labels: ['__meta_nomad_tags']
regex: '.*,prometheus=node_exporter,.*'
action: keep