Questions tagged [kafka-topic]
150 questions
36
votes
7 answers
How to programmatically create a topic in Apache Kafka using Python
So far I haven't seen a python client that implements the creation of a topic explicitly without using the configuration option to create automatically the topics.
jpgerek
- 598
- 1
- 6
- 13
26
votes
5 answers
How to delete multiple topics in Apache Kafka
Assuming that I have a number of topics with the same prefix, e.g:
giorgos-topic1
giorgos-topic2
giorgos-topic3
...
The command used for deleting a single topic (say giorgos-topic1) is the following:
./bin/kafka-topics.sh --zookeeper localhost:2181…
Giorgos Myrianthous
- 24,168
- 10
- 80
- 106
20
votes
1 answer
Why is kafka not creating a topic? bootstrap-server is not a recognized option
I am new to Kafka and trying to create a new topic on my local machine.
I am following this link.
Here are the steps which i followed:
Start zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Start…
KayV
- 9,560
- 8
- 68
- 111
9
votes
1 answer
Kafka topic has partitions with leader=-1 (Kafka Leader Election), while node is up and running
I have a 3 member kafka-cluster setup, the __consumer_offsets topic has 50 partitions.
The following is the result of describe command on:
root@kafka-cluster-0:~# kafka-topics.sh --zookeeper localhost:2181 --describe
Topic:__consumer_offsets …
SpiXel
- 3,558
- 1
- 23
- 38
8
votes
1 answer
AWS MSK - Timeout when creating Kafka topic with ACL turned-on
I'm using AWS MSK and I want to enable ACLs but I'm unable to create a topic when ACLs are turned-on. I'm using the command-line tools for all the operations. Here's a summary of what I'm doing:
Create a fresh cluster
Create a topic - this works…
Araf
- 303
- 4
- 10
6
votes
1 answer
Kafka Streams Testing : java.util.NoSuchElementException: Uninitialized topic: "output_topic_name"
I've written a test class for kafka stream application as per https://kafka.apache.org/24/documentation/streams/developer-guide/testing.html
, the code for which is
import com.EventSerde;
import org.apache.kafka.common.serialization.Serde;
import…
Arbaz Sheikh
- 177
- 7
5
votes
0 answers
When does Kafka topic-level configuration changes take effect?
There are documents such as the ones below stating that Kafka topic-level configuration changes can be taken place while Kafka cluster is still…
Alex Fung
- 1,886
- 9
- 20
4
votes
3 answers
Kafka Log Compacted Topic Duplication Values against same key not deleted
Log compacted topics are not supposed to keep duplicates against the same key. But in our case, when a new value with the same key is sent, the previous one isn't deleted. What could be the issue?
val TestCompactState: KTable[String, TestCompact] =…
Danish
- 301
- 1
- 8
4
votes
2 answers
What is the maximum replication factor for a partition of kafka topic
I hava kafka cluster having 3 brokers and a couple of topics with each having 5 partitions. Now i want to set the replication factor for the partitions.
What is the maximum replication factor which i can set for a partition of kafka topic?
KayV
- 9,560
- 8
- 68
- 111
4
votes
2 answers
Kafka topic partitions with leader -1
I noticed that few of my kafka topics are behaving in a manner i cannot explain clearly.
For eg:
./kafka-topics.sh --describe --zookeeper ${ip}:2181 --topic test
Topic:test PartitionCount:3 ReplicationFactor:1 Configs:retention.ms=1209600000
…
irrelevantUser
- 741
- 7
- 20
3
votes
1 answer
Ideal number of partitions for Kafka topic
I am currently working on a setup which has 6 kafka-brokers, Data is being pushed into my topic from two producers at a rate of about 4000 messages per second, I have 5 Consumers for this topic working as a group. What should be the ideal number of…
SRI HARSHA S V S
- 45
- 4
3
votes
1 answer
Does updating broker settings (topic defaults) change already existing topics?
In kafka there is a log.retention.ms configuration property that can be set at the Broker level and it can be overridden by retention.ms on topic level.
Imagine I have a topic in my Kafka cluster with default config - no explicitly set retention.…
moffeltje
- 4,095
- 4
- 24
- 49
3
votes
1 answer
How Kafka guarantee the messages order while we increase the partitions in runtime?
I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic…
injoy
- 3,197
- 7
- 29
- 54
3
votes
1 answer
Kafka topic cannot be deleted or recreated
I have a Kafka topic that seems to simultaneously exist and not exist.
kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --delete
returns an error:
Topics in [] does not exist
Meanwhile, trying to re-create the topic…
wrschneider
- 15,487
- 10
- 70
- 152
3
votes
4 answers
Kafka consumer not consuming from beginning
I have Kafka setup on my local machine and have started the zookeeper and a single broker server.
Now i have a single topic with following description:
~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-topics.sh --zookeeper 127.0.0.1:2181…
KayV
- 9,560
- 8
- 68
- 111