2

By saying cluster info, I am referring to information like

  1. subscribed consumers/consumer groups
  2. read and committed offsets
  3. leaders and followers of a partition
  4. topics on the server etc.

Does zookeeper keep this info in its own db (though I never heard of zookeeper having any db of its own till date) or it stores this info in the Kafka cluster on some topics etc?

EDIT: and a follow up Q: How does Zookeeper retrive the consumer offsets from __consumer_offsets topic?

samshers
  • 1,269
  • 1
  • 18
  • 39

1 Answers1

3

The ZooKeeper Data Model ZooKeeper has a hierarchal name space, much like a distributed file system. The only difference is that each node in the namespace can have data associated with it as well as children. It is like having a file system that allows a file to also be a directory. Paths to nodes are always expressed as canonical, absolute, slash-separated paths; there are no relative reference.

So there is no database.

Offsets are kept in kafka, not in zookeeper, in this topic: __consumer_offsets

In a zookeeper node:

Gives the list of active brokers

ls /brokers/ids

Gives the list of topics

ls /brokers/topics

Gives more detailed information of the broker id '0'

get /brokers/ids/0

here you can read more about the kafka datastructures in zookeeper: Kafka datastructures zookeeper

There is a KIP for removing zookeeper alltogether: KIP Remove zookeeper

Armando Ballaci
  • 822
  • 4
  • 16
  • 1
    Well Armando... first things first.. Tony Stark.. looks much like you. – samshers Jul 05 '20 at 09:48
  • 1) where is this zookeeper data model located on the zookeeper machine and how can i customize this location – samshers Jul 05 '20 at 09:55
  • 2) when i do a list topics or brokers or partitions does kafka have to pull up this info from zookeeper to dispaly results. (I ask this bcoz only zookeeper seems to keep track of this info through its distributed data model). – samshers Jul 05 '20 at 09:57
  • take a look at this https://jaceklaskowski.gitbooks.io/apache-kafka/kafka-admin-TopicCommand.html, by the way yes, appart from internal caching mechanisms. – Armando Ballaci Jul 05 '20 at 10:07
  • 3) https://stackoverflow.com/questions/62739443/how-does-zookeeper-retrive-the-consumer-offsets-from-consumer-offsets-topic – samshers Jul 05 '20 at 10:21
  • @samshers You can customize the location using [this](http://kafka.apache.org/documentation.html#zookeeper.connect) config parameter. More information of the kafka znodes [here](http://kafka.apache.org/documentation.html#impl_zookeeper) – Arvanitis Christos Jul 06 '20 at 22:28