Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

150 questions
3
votes
2 answers

fixing kafka cluster with under-replicated partitions

We are having a problem with one of our kafka clusters. We have 6 nodes with v1.0, all the topics have a replication factor of 3 and 10 partitions/topic which seemed to be enough for us. Due to a power failure, 3 of the nodes went down for a while,…
Acampoh
  • 439
  • 8
  • 21
2
votes
1 answer

Removing one message from a topic in Kafka

I'm new at using Kafka and I have one question. Can I delete only ONE message from a topic if I know the topic, the offset and the partition? And if not is there any alternative?
Ander99
  • 33
  • 2
2
votes
1 answer

Kafka: auto create topic with options

I'm using auto-creation of topics in Kafka. Some topics require compaction, some not. Is it possible to specify custom options when auto-creating a topic; or do I need to test for existence and create it 'manually' with the desired options before I…
BenoitParis
  • 2,363
  • 4
  • 25
  • 46
2
votes
1 answer

Size of Kafka Partitions and writing to a Topic with no free disk space

I am starting to learn Kafka for enterprise solution purposes. During my readings, some questions came to my mind: Are all the partitions on a topic of the same size? What will happen if a Kafka message is sent on a topic and the topic is…
Saurabh Prakash
  • 1,775
  • 1
  • 9
  • 15
2
votes
1 answer

understanding kafka log.dirs

I have Kafka cluster and the log.dirs=/data/kafka is set to the data directory in server.properties. my DATA partition is kept getting full due to these logs which take a big part of it. (talking about binary logs in topic directory like…
NoamiA
  • 395
  • 1
  • 10
2
votes
2 answers

Kafka topics with different retention although having the same configuration

I have a question regarding retention of topics messages. I have the following situation: Two Kafka topics; Broker has properties log.retention.hours=1; Describe command for each of the two topics: [oracleas@zkafka01vdc bin]$ ./kafka-topics.sh…
F.Lazarescu
  • 1,342
  • 2
  • 13
  • 26
2
votes
1 answer

Where does zookeeper store kafka cluster and related information?

By saying cluster info, I am referring to information like subscribed consumers/consumer groups read and committed offsets leaders and followers of a partition topics on the server etc. Does zookeeper keep this info in its own db (though I never…
samshers
  • 1,269
  • 1
  • 18
  • 39
2
votes
2 answers

Should topic partitions be replicated across all broker nodes in a Kafka cluster?

Though answers are available to question similar to above. My curiosity lies in the fact that suppose n1-5 nodes are in cluster where topics t1 is on n1,n2 and n3, topic t2 is on n3,n4,n5. Now if suppose p1 pushes messages in t1 and c1 consumes from…
dvsakgec
  • 2,388
  • 2
  • 19
  • 29
2
votes
2 answers

Why can it take so long to delete Kafka topics?

On a 3 node cluster, I created few topics with thousands of messages. I have noticed that it takes long time to delete a topic. It took me more than 14 mins to delete 500 topics. Are there any best practices for topic deletion? Is there any…
JR ibkr
  • 711
  • 3
  • 21
2
votes
0 answers

kafka electing new leader after 2/3 is down

2 of my 3 Kafka cluster is down with replica_factor=2ץ I don't care about the DATA, I just want to save my Kafka and reuse it's topics. I test one of my topic who has 4 partitions, I added unclean.leader.election.enable=true to the server.conf but…
NoamiA
  • 395
  • 1
  • 10
2
votes
2 answers

Splitting Kafka into separate topic or single topic/multiple partitions

As usual, it's bit confusing to see benefits of splitting methods over others. I can't see the difference/Pros-Cons between having Topic1 -> P0 and Topic 2 -> P0 over Topic 1 -> P0, P1 and a consumer pull from 2 topics or single topic/2…
Ahmed Alaa El-Din
  • 1,663
  • 10
  • 19
2
votes
1 answer

How to programmatically create topics using kafka-python?

I am getting started with Kafka and fairly new to Python. I am using this library named kafka-python to communicate with my Kafka broker. Now I need to dynamically create a topic from my code, from the docs what I see is I can call create_topics()…
Paras
  • 2,077
  • 2
  • 24
  • 54
2
votes
1 answer

Can Kafka Streams output topic be on a separate cluster?

I have a topic where all logs are pushed to centralized topic but I would like to filter out some of those records to a separate topic and cluster if possible. Thanks
user432024
  • 3,572
  • 6
  • 39
  • 70
2
votes
2 answers

How to delete data which already been consumed by consumer? Kafka

I am doing data replication in kafka. But, the size of kafka log file is increases very quickly. The size reaches 5 gb in a day. As a solution of this problem, ı want to delete processed data immediately. I am using delete record method in…
omerstack
  • 348
  • 1
  • 14
2
votes
1 answer

How to delete a topic configuration via KafkaAdminClient

I want to delete a configuration (reset it to default) for a topic which was overridden before. This is possible with the provided script $> ./kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics \ --entity-name test…
maltesmann
  • 365
  • 5
  • 12
1
2
3
…
9 10