Right now I have functionality that writes a couple hundred messages onto a kafka queue. But when all of those messages have been consumed I need to also execute additional functionality. Is there a way to place a listener on a kafka queue to get notified when it has been emptied?
Asked
Active
Viewed 1,777 times
0
-
what do you mean by emptied? If all messages are consumed then it doesn't mean queue will become empty. – avr Apr 05 '16 at 15:03
-
You could put a special message in the queue. – nha Apr 05 '16 at 16:49
1 Answers
4
You could solve this two ways, I think:
- Kafka's Fetch Response contains a
HighwaterMarkOffset, which essentially is an offset of the last message in a partition. You could check whether your message has that offset and if so - you've reached the end. However, this won't work if you have producer and consumer working at the same time - consumer can just consume messages faster and thus stop earlier than you need. - Send a "poison pill" message - say you need to produce 100 messages. Then your producer sends these 100 messages + 1 special message (some UUID for example, but be sure it never appears under normal circumstances in your logic) that would mean "the end". On consumer side you would check whether the received message is a poison pill and shutdown if it is.
serejja
- 19,968
- 6
- 54
- 69