size of topic

[steveburkett]

Is there anyway to call thru ruby-kafka to get the current (unread) size of the topic? (something efficient)

i dont really want to write a consumer to read all the messages, i just want to add a metric to our dashboard that shows some metric that this topic has X messages (or something like that) and is waiting to be consumed.

thanks!

[geoff2k]

I don't believe there is anything in the API itself to retrieve what is usually referred to as "consumer lag".

FWIW, we have been using https://github.com/danielqsj/kafka_exporter to pull data from our confluent cloud cluster and export it to DataDog for dashboarding/alerting purposes so we can quickly detect which consumers are falling behind and add wokers if/when necessary.

[tjwp]

There is instrumentation of the consumer lag within ruby-kafka:

here: https://github.com/zendesk/ruby-kafka/blob/master/lib/kafka/consumer.rb#L232
and here: https://github.com/zendesk/ruby-kafka/blob/master/lib/kafka/consumer.rb#L325

If you're already using a consumer built on ruby-kafka then you can tap into those notifications: https://github.com/zendesk/ruby-kafka#instrumentation

This is what built-in metrics support use, such as the Datadog integration.