Vault Monitoring with Prometheus ServiceMonitor

Vault
1

Hello folks!

I’m trying to scrape vault metrics via Prometheus ServiceMonitor , in order to allow the servicemonitor to authenticate with vault I generate a token and it’s been added to the ServiceMonitor as bearerTokenSecret , but looks Prometheus operator doesn’t work as expected with the bearer token as a secret, since it starts throwing HTTP 400 error when it tries to scrape the metrics.

As a fix for above, I used bearerTokenFile instead following below steps:

  1. create serviceaccount without the token.
  2. create a secret which has vault token.
  3. add the secret to the service account created in step 1

And after that I created the ServiceMonitor as below manifest, then I start getting 500 errors when promethues scraped the metrics and vault logs were dropping:

[ERROR] core: failed to lookup acl token: error=“failed to look up namespace from the token: no namespace”

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-vault
  namespace: vault
  labels:
    release: prometheus
automountServiceAccountToken: false
secrets:
- name: vault-prometheus-token

---

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vault-servicemonitor
  namespace: vault
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
  endpoints:
    - interval: 20s
      path: /v1/sys/metrics
      port: http
      params:
       format: 
        - prometheus
      tlsConfig:
        insecureSkipVerify: true
      bearerTokenFile:  "/var/run/secrets/kubernetes.io/serviceaccount/token"

Notes:

  1. vault version 10.1.2 - helm chart deployment.
  2. Promethues version is 2.35.0 - helm chart deployment.

I would truly appreciate your guidance.
Abeer

2

You appear to be trying to send a Kubernetes service account token to Vault - this is not correct - Vault needs a Vault token that it has issued itself.

The easiest option is to just set unauthenticated_metrics_access in the Vault configuration, and turn off the requirement to authenticate for metrics.

If you won’t do that, you need to configure a vault agent container within your Prometheus pod, to maintain a bearer token.

3

Actually, my serviceaccount token uses a k8s secret which has a vault token

I tried unauthenticated_metrics_access but getting 403 :frowning:

automountServiceAccountToken: false
secrets:
- name: vault-prometheus-token
4

I do not understand what you are trying to say.

You must not have successfully configured unauthenticated_metrics_access then.

5

I restarted my vault cluster and things works!

Thanks for your help much appreciated.