Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

150 questions
36
votes
7 answers

How to programmatically create a topic in Apache Kafka using Python

So far I haven't seen a python client that implements the creation of a topic explicitly without using the configuration option to create automatically the topics.
jpgerek
  • 598
  • 1
  • 6
  • 13
26
votes
5 answers

How to delete multiple topics in Apache Kafka

Assuming that I have a number of topics with the same prefix, e.g: giorgos-topic1 giorgos-topic2 giorgos-topic3 ... The command used for deleting a single topic (say giorgos-topic1) is the following: ./bin/kafka-topics.sh --zookeeper localhost:2181…
Giorgos Myrianthous
  • 24,168
  • 10
  • 80
  • 106
20
votes
1 answer

Why is kafka not creating a topic? bootstrap-server is not a recognized option

I am new to Kafka and trying to create a new topic on my local machine. I am following this link. Here are the steps which i followed: Start zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties Start…
KayV
  • 9,560
  • 8
  • 68
  • 111
9
votes
1 answer

Kafka topic has partitions with leader=-1 (Kafka Leader Election), while node is up and running

I have a 3 member kafka-cluster setup, the __consumer_offsets topic has 50 partitions. The following is the result of describe command on: root@kafka-cluster-0:~# kafka-topics.sh --zookeeper localhost:2181 --describe Topic:__consumer_offsets …
SpiXel
  • 3,558
  • 1
  • 23
  • 38
8
votes
1 answer

AWS MSK - Timeout when creating Kafka topic with ACL turned-on

I'm using AWS MSK and I want to enable ACLs but I'm unable to create a topic when ACLs are turned-on. I'm using the command-line tools for all the operations. Here's a summary of what I'm doing: Create a fresh cluster Create a topic - this works…
Araf
  • 303
  • 4
  • 10
6
votes
1 answer

Kafka Streams Testing : java.util.NoSuchElementException: Uninitialized topic: "output_topic_name"

I've written a test class for kafka stream application as per https://kafka.apache.org/24/documentation/streams/developer-guide/testing.html , the code for which is import com.EventSerde; import org.apache.kafka.common.serialization.Serde; import…
Arbaz Sheikh
  • 177
  • 7
5
votes
0 answers

When does Kafka topic-level configuration changes take effect?

There are documents such as the ones below stating that Kafka topic-level configuration changes can be taken place while Kafka cluster is still…
Alex Fung
  • 1,886
  • 9
  • 20
4
votes
3 answers

Kafka Log Compacted Topic Duplication Values against same key not deleted

Log compacted topics are not supposed to keep duplicates against the same key. But in our case, when a new value with the same key is sent, the previous one isn't deleted. What could be the issue? val TestCompactState: KTable[String, TestCompact] =…
Danish
  • 301
  • 1
  • 8
4
votes
2 answers

What is the maximum replication factor for a partition of kafka topic

I hava kafka cluster having 3 brokers and a couple of topics with each having 5 partitions. Now i want to set the replication factor for the partitions. What is the maximum replication factor which i can set for a partition of kafka topic?
KayV
  • 9,560
  • 8
  • 68
  • 111
4
votes
2 answers

Kafka topic partitions with leader -1

I noticed that few of my kafka topics are behaving in a manner i cannot explain clearly. For eg: ./kafka-topics.sh --describe --zookeeper ${ip}:2181 --topic test Topic:test PartitionCount:3 ReplicationFactor:1 Configs:retention.ms=1209600000 …
irrelevantUser
  • 741
  • 7
  • 20
3
votes
1 answer

Ideal number of partitions for Kafka topic

I am currently working on a setup which has 6 kafka-brokers, Data is being pushed into my topic from two producers at a rate of about 4000 messages per second, I have 5 Consumers for this topic working as a group. What should be the ideal number of…
SRI HARSHA S V S
  • 45
  • 4
3
votes
1 answer

Does updating broker settings (topic defaults) change already existing topics?

In kafka there is a log.retention.ms configuration property that can be set at the Broker level and it can be overridden by retention.ms on topic level. Imagine I have a topic in my Kafka cluster with default config - no explicitly set retention.…
moffeltje
  • 4,095
  • 4
  • 24
  • 49
3
votes
1 answer

How Kafka guarantee the messages order while we increase the partitions in runtime?

I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic…
injoy
  • 3,197
  • 7
  • 29
  • 54
3
votes
1 answer

Kafka topic cannot be deleted or recreated

I have a Kafka topic that seems to simultaneously exist and not exist. kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --delete returns an error: Topics in [] does not exist Meanwhile, trying to re-create the topic…
wrschneider
  • 15,487
  • 10
  • 70
  • 152
3
votes
4 answers

Kafka consumer not consuming from beginning

I have Kafka setup on my local machine and have started the zookeeper and a single broker server. Now i have a single topic with following description: ~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-topics.sh --zookeeper 127.0.0.1:2181…
KayV
  • 9,560
  • 8
  • 68
  • 111
1
2 3
…
9 10