Deploy Opentelemetry collector in service mesh

Consul
, ,
1

Hi, I’m trying to deploy opentelemetry collector into service mesh. That needs to receive telemetry data to be exposed from any service in the service mesh. But it seems not to be able to access to the endpoint that is defined as a receiver in the collector.
Is it possible to configure to access to the endpoint from services in the service mesh? (any special annotations are required?)

2

Hi @Shige99011,

Welcome to the HashiCorp Forums!

I assume your OpenTelemetry Collector is not a part of mesh (you don’t have a sidecar running). If this is the case, you should use the following annotation to exclude your receiver port from being intercepted by the transparent proxy rules.

ref: Annotations and Labels | Consul | HashiCorp Developer

For e.g., if you use the Zipkin receiver, you should add the following annotation.

consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: '9411'

I hope this helps.

3

Hi, @Ranjandas ,
Thanks for your quick reply. I would like to tell you the detail of my situation.
The Opentelemetry collector is now a part of service mesh by adding the annotation (consul.hashicorp.com/connect-inject: “true”)
Also, the service to expose metrics to the collector is a part of service mesh as well.
In this case, is it possible to expose metrics from the service by specifying the receiver endpoint of the collector? (e.g. “http://otel-collector:4318”)
I could confirm that metrics can be exposed to the collector when the collector is NOT a part of service mesh.

Thank you.

4

Hi @Shige99011,

In that case, you will have to do the following things.

  1. Use the connect-service-port annotation so that the sidecar-proxy on the collector would deliver the incoming traffic to the right port. Update the collector values.yaml file and run a helm upgrade.

    podAnnotations:
  consul.hashicorp.com/connect-inject: 'true'
  consul.hashicorp.com/connect-service-port: '4318'

    ref: Annotations and Labels | Consul | HashiCorp Developer

  2. Next, if ACLs are enabled, you should have service-intentions that allows the downstream services to send the traces into the collector.

    apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: web-to-otel
spec:
  destination:
    name: my-opentelemetry-collector
  sources:
    - name: web
      action: allow

Once the above two are done, your application can reach the collector to send traces.

I hope this helps.

5

Hi @Ranjandas,
Thanks for your suggestions. However, it doesn’t work even with them yet.
For your suggestion #1, values.yaml for otel-collector is as below:

podAnnotations: {
  consul.hashicorp.com/connect-inject: "true",
  consul.hashicorp.com/connect-service-port: "4318",
  consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "8889" # this is to scrape metrics from prometheus that is out of service mesh. is this correct?
}

by the way, the config section of values.yaml for otel-collector is as below:

config:
  exporters:
    logging: {}

    prometheus:
      endpoint: ${env:MY_POD_IP}:8889
      
  extensions:
    health_check: {}
    memory_ballast: {}
  processors:
    batch: {}
    memory_limiter: null
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318
  service:
    extensions:
      - health_check
      - memory_ballast
    pipelines:
      logs:
        exporters:
          - logging
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
      metrics:
        exporters:
          - prometheus
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
      traces:
        exporters:
          - logging
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp

For #2, I added the service intention as below and this has been confirmed on Consul UI:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: service-to-otel
spec:
  destination:
    name: opentelemetry-collector
  sources:
    - name: my-service
      action: allow

And the endpoint to expose metrics to the collector is defined in the values.yaml of the service as below:

otel_collector:
  endpoint: "http://opentelemetry-collector:4318/v1/metrics"

How should it be checked if the connection works or not? for example, can it be checked on the terminal of the pod of the service by using curl command …etc?

Thank you.

6

Hi @Shige99011,

Could you please override the endpoint to bind to 127.0.0.1 (or 0.0.0.0)? I think adding the following to the values.yaml file should do the trick.

config
  receivers:
    http:
      endpoint: 127.0.0.1:4318

The sidecar would expect the local applications to listen on loopback, and not the POD_IP.

Could you please let me know if this fixes the issue for you?

7

Hi @Ranjandas ,

BINGO!! It has worked finally by change the endpoint configuration to “127.0.0.1:4318”.
The expected metrics can be found on prometheus UI.

Thanks a lot for your kind/quick help.

1 Like
8

Hi @Ranjandas

I have a quick question regarding collector service port configuration.

Is it possible to configure multiple ports to specify ports to listen on?
now, “consul.hashicorp.com/connect-service-port” annotation is used to listen on the specified port of the collector service. However, this annotation allows only one port to configure, correct? I would like to configure the collector service not only for metrics data but also trace data.
Or any workaround?

Thank you.

9

Hi @Shige99011,

You can refer to this documentation to configure multi-port for Consul.

ref: How does Consul Service Mesh Work on Kubernetes? | Consul | HashiCorp Developer

Certain caveats are also documented in there, and I request you to have a read before proceeding with the implementation.

Consul 1.17 now provides a preview of the experience for multi-port service mesh, which is documented here: Configure multi-port services | Consul | HashiCorp Developer. But it is a preview and not meant to run in production atm.

I hope this helps.

10

Hi @Ranjandas ,
Thanks for the suggestions. I tried the document:

I created service account and service for each port.
However, some errors seem to happen as below regarding the deployment object of the collector service that I want to configure multiple ports:

status:
  conditions:
  - lastTransitionTime: "2023-12-19T06:31:07Z"
    lastUpdateTime: "2023-12-19T06:31:07Z"
    message: Created new replica set "opentelemetry-collector-bc7798fb8"
    reason: NewReplicaSetCreated
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-12-19T06:31:08Z"
    lastUpdateTime: "2023-12-19T06:31:08Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2023-12-19T06:31:07Z"
    lastUpdateTime: "2023-12-19T06:31:07Z"
    message: 'admission webhook "consul-connect-injector.consul.hashicorp.com" denied
      the request: multi port services are not compatible with metrics'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure

My current consul version is here, btw:
Chart: consul-0.41.1
APP Version: 1.11.3

Any more configurations are still required?

Thank you.

11

There were some my mistakes. I needed to override the annotations for the collector service as below:

  consul.hashicorp.com/enable-metrics: "false",
  consul.hashicorp.com/enable-metrics-merging: "false"

now the pods for the service seem to be running. Will check the configuration behavior.

Thank you.

12

Hi @Ranjandas,
I could finally confirm that the multiple ports configuration works for my otel collector service.

Thanks for your help again.

1 Like