Spring boot actuator metrics for Prometheus in Consul Connect

Consul
,
1

I have a spring boot application running in a Nomad cluster with Consul Connect enabled.

network {
  mode = "bridge"
}

service {
  name = "api"
  port = "9966"

  connect {
    sidecar_service {}
  }
}

There is no port mapping defined and the API is reachable only within consul service mesh through the proxy. Now I have prometheus running in the same cluster. How does prometheus discover the individual API instances and scrape metrics out of it. I used the below config. But without having a mapping port in the host level, it is not able to reach the individual API instance.

- job_name: 'actuator'
    metrics_path: /api/actuator/prometheus
    consul_sd_configs:
    - server: '{{ env "NOMAD_IP_prometheus_ui" }}:8500'
      services: ['api']

How to solve this problem? What is general practice to scrape metrics from a spring boot application running inside a service mesh with no host port mapping?

2

Finally found it. Nomad has an option to expose a particular endpoint via another sidecar proxy without mTLS authentication. The use case of this option is specifically for health check or metrics.

The expose stanza inside connect stanza helps to achieve this.

connect {
        sidecar_service {
          proxy {
            expose {
              path {
                path             =  "/actuator/prometheus"
                protocol         =  "http"
                local_path_port  =  9966
                listener_port    =  "metrics"
              }
            }
          }
        }
      }