0

kafka consumer using librdkafka (high level consumer) connected to kafka cluster and subscribed to 10 topics and consuming data. There was assign partitions event.

There was network issue due to which cluster was not reachable. Lost connection with group co-ordinator and heartbeats got stuck. There was revoked partitions event where the code calls unassign on consumer.

When cluster came back up, the consumer was not consuming any data although its in the while true loop calling consume with timeout of 1 sec.

Does consumer needs to resubscribe to topics again once it is connected to cluster? What is a reliable way to detect the consumer is connected to cluster in code?

Youli Luo
  • 121
  • 1
  • 9

1 Answers1

2

Does consumer needs to resubscribe to topics again once it is connected to cluster?

Yes. New group members will cause a rebalance amongst existing members, and they need to resubscribe

What is a reliable way to detect the consumer is connected to cluster in code?

You could describe the consumer group and see if there are active clients for the consumer group you are interested in

OneCricketeer
  • 126,858
  • 14
  • 92
  • 185
  • Thanks. This group has only one consumer. Do you think it will work if the consumer keeps trying let's say every one minute to subscribe after it gets revoked partitions event until it gets an assign partitions event? – Youli Luo Jan 28 '20 at 20:59
  • If you are subscribed to 10 topics, you can scale up to 10*(number of partition) consumers, by the way. I would suggest not waiting for a minute between poll intervals. If you need to batch data, then maybe Kafka isn't completely suited for that use-case – OneCricketeer Jan 28 '20 at 22:27
  • No issues performance wise. Only issue is after network outage fixed, consumer was not consuming. Had to manually restart consumer then it worked. consume with poll interval of 1 second not a minute..also if data comes in earlier it's consumed immediately – Youli Luo Jan 28 '20 at 22:31
  • So the issue is fixed? – OneCricketeer Jan 29 '20 at 04:15
  • per librdkafka documentation, consumer should recover automatically from network issues and rebalances...will upgrade to latest librdkafka and try... – Youli Luo Jan 29 '20 at 20:13