i have a standard nomad cluster (3 servers, N clients) with ACLs enabled everywhere
i am trying to use the prometheus
nomad_sd_config config to discover nomad services to scrape :: Configuration | Prometheus
since i have ACLs enabled everywhere, i need to pass prometheus a legit nomad token it can use to access the nomad API … for the life of me, i cannot find a good example on how i might do that …
THE ASK :: does anyone have a good example of how they are scraping the nomad service catalog, from prometheus, when ACLs are enabled on the nomad API ??
i assume i need to somehow pass a token, and thought i might be able to use some of the prometheus
proxy configs – but really not sure … the prometheus
consul_sd_config (Configuration | Prometheus) does have a
token option – but i don’t see anything similar in the nomad config …
any help is greatly appreciated …
below is an example of my config, and as you might expect, i am getting a 403 (permission denied) error …
ts=2023-04-18T19:42:35.899Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=nomad msg="Unable to refresh target groups" err="Unexpected response code: 403 (Permission denied)"
--- global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['127.0.0.1:9090'] - job_name: 'node_exporter' nomad_sd_configs: - server: 'https://nomad.example.com/' region: 'us' namespace: 'sys' refresh_interval: 5s relabel_configs: - source_labels: ['__meta_nomad_tags'] regex: '.*,prometheus=node_exporter,.*' action: keep