30

I'm using kafka_2.9.2-0.8.1.1 with zookeeper 3.4.6.

Is there a utility that can automatically remove a consumer group from zookeeper? Or can I just remove everything under /consumers/[group_id] in zookeeper? If the latter, is there anything else I'm missing & can this be done with a live system?

Update: As of kafka version 2.3.0, there is a new utility:

> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group

Related doc: http://kafka.apache.org/documentation/#basic_ops_consumer_lag

See below for more discussion

Foo L
  • 10,376
  • 8
  • 35
  • 49

4 Answers4

45

As of v0.9.0, Kafka ships with a suite of tools in the /bin one of which is the kafka-consumer-groups.sh tool. This will delete a consumer group. ./kafka-consumer-groups.sh --zookeeper <zookeeper_url> --delete --group <group-name>

eodgooch
  • 1,124
  • 1
  • 9
  • 17
  • 1
    Thanks for the input, but this was not available in v0.8.1.1 (as stated in the question). It's available in the latest release (0.10.0.0) – Foo L Jul 05 '16 at 22:41
  • `tar tvfz kafka_2.9.2-0.8.1.1.tgz | grep consumer | cut -c 49-` kafka_2.9.2-0.8.1.1/bin/kafka-simple-consumer-shell.sh kafka_2.9.2-0.8.1.1/bin/kafka-console-consumer.sh kafka_2.9.2-0.8.1.1/bin/kafka-simple-consumer-perf-test.sh kafka_2.9.2-0.8.1.1/bin/kafka-consumer-perf-test.sh kafka_2.9.2-0.8.1.1/bin/windows/kafka-console-consumer.bat kafka_2.9.2-0.8.1.1/config/consumer.properties – Foo L Jul 05 '16 at 22:45
  • 4
    @foo-l I know it didn't exist for 0.8. Due to the kafka updates I thought this would help people currently searching for the issue. – eodgooch Jul 06 '16 at 20:13
  • If this doesn't work, delete from zk. rmr /consumers/ – KaliCharan Dec 27 '18 at 18:41
31

For new consumers (which use a kafka topic to manage offsets instead of zookeeper) you cannot delete the group information using kafka's built in tools.

Here is an example of trying to delete the group information for a new style consumer using the kafka-consumer-groups.sh script:

bin/kafka-consumer-groups.sh --bootstrap-server "kafka:9092" --delete --group "indexer" --topic "cleaned-logs"
Option '[delete]' is only valid with '[zookeeper]'. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.

Here's the important part of that response:

Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.

This is kind of annoying from a monitoring perspective (esp. when tracking offsets via something like burrow) because it means that if you change consumer group names in your code you'll keep seeing that old groups are behind on their offsets until those offsets expire.

Hypothetically you could write a tombstone to that topic manually (which is what happens during offset expiration) but I haven't found any tools that make this easy.

turtlemonvh
  • 7,111
  • 2
  • 41
  • 49
23

you can delete group from kafka by CLI

kafka-consumer-groups --bootstrap-server localhost:9092 --delete --group group_name
TT.
  • 14,883
  • 6
  • 41
  • 77
Rudy
  • 371
  • 3
  • 10
  • 1
    the original question asked under a version of kafka which did not have the `kafka-consumer-groups` utility – Foo L Jul 25 '19 at 20:25
  • not possible, getting: `Option '[delete]' is only valid with '[zookeeper]'. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.` – Dimitris Moraitidis Sep 23 '20 at 13:41
22

Currently, as I know, the only way to remove a Kafka consumer group is manually deleting Zookeeper path /consumers/[group_id].

If you just want to delete a consumer group, there is nothing to worry about manually deleting the Zookeeper path, but if you do it for rewinding offsets, the below will be helpful.

First of all, you should stop all the consumers belongs to the consumer group before removing the Zookeeper path. If you don't, those consumers will not consume newly produced messages and will soon close connections to the Zookeeper cluster.

When you restart the consumers, if you want the consumers to start off from the beginning, give auto.offset.reset property to smallest (or earliest in new Kafka releases). The default value of the property is largest (or latest in new Kafka releases) which makes your restarting consumers read after the largest offset which in turn consuming only newly produced messages. For more information about the property, refer to Consumer Config in the Kafka documentation.

FYI, there is a question How can I rewind the offset in the consumer? in Kafka FAQ, but it gave me not much help.

OneCricketeer
  • 126,858
  • 14
  • 92
  • 185
Heejin
  • 4,145
  • 3
  • 23
  • 28
  • Just cleaning up consumer groups that were misnamed or random console consumer groups.. Thanks! – Foo L Mar 25 '15 at 18:31
  • Related: [What determines Kafka consumer offset?](http://stackoverflow.com/questions/32390265/what-determines-kafka-consumer-offset) – Filipe Correia May 18 '16 at 16:12
  • Note `auto.offset.reset` is ignored after there are offsets for a consumer group so they can pick up where they left off if shut down. This was confusing to me but confirmed from the developers. You can manually delete the group, or next run start a new `group.id` with the `earliest` or `smallest` flag and start over. Check out Burrow and has API for deleting groups and other maintenance and lag checking from team at LinkedIN. – Mike S. Jul 03 '16 at 04:54
  • If using Kafka's CLI, add the `--from-beginning` flag to your `kafka-console-consumer ...` call to replay the topic – Mike S. Jul 03 '16 at 04:55
  • 4
    Just to help anyone that searches for the commands on how to delete a consumer's path in zookeeper. First connect to zookeeper `./zookeeper-shell.sh localhost:2181`, second check the node (that is optional) with `get /consumers/`, and finally third delete the group with `rmr /consumers/`. – Philipp Feb 14 '18 at 00:26
  • @Philipp consumer ids in zookeeper were generated automatically in my case. Is there a way to correlate the regular names that one gets via `kafka-consumer-group.sh` and the random ones via `ls /consumers` – Dimitris Moraitidis Sep 23 '20 at 13:37