Consul and Prometheus with ACLs and TLS enable

Hi all,

we are using Consul (v1.9.7-ent) on Kubernetes and we would like to monitor beyond the service mesh activity, the platform too with Prometheus and Grafana.

We are enabled the ACLs and the TLS: is it feasible to reach our goal with this setup or could we only observe the metrics coming from the service mesh?

If the answer to the question is “Yes”, where could we find some guide for the setup?

Reading the documentation and also after some preliminary tests it seems that when we enable the TLS we have to put in the helm values yaml the property global.metrics.enableAgentMetrics to false and this is preventing Prometheus to be able to scrape all the consul_* metrics.

Is it there a workaround?

Thank you very much,
Davide

1 Like

Hi Davide,
Currently the only workaround would be to use a Prometheus ServiceMonitor which provides more configurability around scraping metrics. You could then configure it with a CA cert so it can pull from Consul’s TLS endpoints. The reason it doesn’t work out of the box is because the prometheus annotations don’t support TLS.

Unfortunately we don’t have any instructions for how to set this up. Here is the TLSConfig that can be used to configure a ServiceMonitor CRD (prometheus-operator/api.md at master · prometheus-operator/prometheus-operator · GitHub).

Hi @lkysow ,
how can I able to test it? When global.tls.enabledandglobal.tls.httpsOnlyare set to true` , and Consul agent metrics are enabled also, the installation be error out during Helm template rendering.
Only if I will have both enabled I will able to investigate on the right prometheus ServiceMonitor you suggested settings…am i missing somethings?

Thanks a lot.

Antonio

1 Like

Ahh I see. I think for now you would need to set metrics to false in the Helm chart and then use server.extraConfig and client.extraConfig to manually set the metrics config:

global:
  metrics:
    enableAgentMetrics: false
server:
  extraConfig: |
    {
      "telemetry": {
        "prometheus_retention_time": "1m"
      }
    }
client:
  extraConfig: |
    {
      "telemetry": {
        "prometheus_retention_time": "1m"
      }
    }

In addition you’d need to create an ACL policy/token with agent:read permissions:

agent_prefix "" {
  policy = "read"
}

that’s passed via the X-Consul-Token HTTP header.

@lkysow Thank you very much for these really useful tips.

Is there also a way to enable the Prometheus scraping options? (On server: consul-helm/server-statefulset.yaml at v0.31.1 · hashicorp/consul-helm · GitHub / on client: consul-helm/client-daemonset.yaml at v0.31.1 · hashicorp/consul-helm · GitHub)

If you’re using the ServiceMonitor CRD then I think you can specify what it scrapes based on other labels, i.e. you don’t need the annotations.