Kafka Receiver does not include topic or receiver as a label

[jhuizingh]

Component(s)

receiver/kafka

What happened?

Description

The metrics emitted by the Kafka Receiver to not contain enough labels to determine which Kafka Receiver and topic they are referring to if you use multiple instances of the Kafka Receiver. Many other receivers include a receiver label which shows exactly which receiver the metric comes from by name, but the Kafka Receiver does not.

Steps to Reproduce

Create an otel collector with the following relevant pieces

  ... irrelevent parts omitted ...
  receivers:
    kafka/first:
      topic: kafka-topic-first
    kafka/second:
      topic: kafka-topic-second
  service:
    pipelines:
      metrics/kafka:
        receivers:
          - kafka/first
          - kafka/second
  ... irrelevent parts omitted ...

Expected Result

Metric in the form:

otelcol_kafka_receiver_offset_lag {
  cluster="mycluster",
  http_scheme="http",
  net_host_name="10.110.0.242",
  net_host_port="8888",
  partition="24",
  server_address="10.110.0.242",
  server_port="8888",
  service_name="otelcol-contrib",
  service_version="0.103.1",
  url_scheme="http",
  receiver="kafka/first",
  topic="kafka-topic-first"
}

Actual Result

Metric in the form:

otelcol_kafka_receiver_offset_lag {
  cluster="mycluster",
  http_scheme="http",
  net_host_name="10.110.0.242",
  net_host_port="8888",
  partition="24",
  server_address="10.110.0.242",
  server_port="8888",
  service_name="otelcol-contrib",
  service_version="0.103.1",
  url_scheme="http"
}

Collector version

0.103.1

Environment information

Environment

AWS EKS
Kubernetes v 1.30.3

Image: opentelemetry-collector-contrib
Helm Chart: opentelemetry-collector-0.95.0

OpenTelemetry Collector configuration

receivers:
    kafka/first:
      topic: kafka-topic-first
    kafka/second:
      topic: kafka-topic-second
  service:
    pipelines:
      metrics/kafka:
        receivers:
          - kafka/first
          - kafka/second

Log output

N/A

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[StephanSalas]

Looks like this fix may be as simple as adding the missing attributes in a few places, such as:

1. opentelemetry-collector-contrib/receiver/kafkareceiver/kafka_receiver.go

   Line 554 in 025cae2

   attrs := attribute.NewSet(

2. opentelemetry-collector-contrib/receiver/kafkareceiver/kafka_receiver.go

   Line 631 in 025cae2

   attrs := attribute.NewSet(

3. opentelemetry-collector-contrib/receiver/kafkareceiver/kafka_receiver.go

   Line 708 in 025cae2

   attrs := attribute.NewSet(

[MovieStoreGuy]

Hey,

I just wanted to clarify that you are referring to the internal metrics, correct?
I presume you don't want to mutate the data you're reading from the actual stream itself.

Then the fields you're looking to include are:

• receiver
• topic

Is that correct?

[jhuizingh]

@MovieStoreGuy yes, exactly. Without those attributes, it can be challenging to understand how things are performing on a single deployment with multiple replicas and multiple Kafka receivers configured.

[StephanSalas]

@MovieStoreGuy, looks like this was partially fixed in v0.104.x version of OTEL Contrib with the "name" and "partition" attributes being added: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/release/v0.104.x/receiver/kafkareceiver/kafka_receiver.go. The problem is in the the v0.103.x version that @jhuizingh is referring to, so would recommend to them to upgrade to v0.104.x to at least obtain the new "name" and "partition" internal telemetry attributes.

As for the add topic attribute ask... I think adding topic attribute to a few of the relevant metrics makes a lot of sense because it will make it easier to observe what topic is actually being consumed rather than only having kafkareceiver/receiver name yaml as an item the troubleshooter has to backtrack to, just to obtain the topic name.

To address his other request to add topic to the attributes, I do have PR #36068 lined up. Please review :)