Questions tagged [kafka-topic]
150 questions
2
votes
1 answer
What is topic.registered in ksql when running list topics
I'm using kafka-connect to stream the rows from a mysql table to a kafka topic.
When I run this on my ksql-client:
LIST TOPICS;
I get a list of topics looking like this:
Kafka Topic | Registered | Partitions | Partition Replicas…
lloiacono
- 3,504
- 2
- 27
- 40
1
vote
2 answers
Kafka log.segment.bytes vs log.retention.hours
I was following the book "Kafka: The Definitive Guide" First Edition to understand when log segments are deleted by the broker.
As per the text I understood, a segment will not become eligible for deletion until it is closed. A segment can be closed…
Mayank Madhav
- 177
- 2
- 14
1
vote
1 answer
Kafka topic returns 0 messages
I have a real-time process that streams events occuring in MongoDB directly to Snowflake using Confluent Platform. Confluent is established on my local machine as a docker image and I see no issues:
Name Command …
marcin2x4
- 619
- 3
- 13
1
vote
1 answer
Can compacted Kafka topic be used as key-value database?
In many articles, I've read that compacted Kafka topics can be used as a database. However, when looking at the Kafka API, I cannot find methods that allow me to query a topic for a value based on a key.
So, can a compacted Kafka topic be used as a…
xarx
- 507
- 8
- 22
1
vote
2 answers
Get the partition number of a record before producing in Kafka
I need to know the partition number of Kafka topic to which a Kafka record goes.
Before the execution of
producer.send(record);
Is there any way to know to which partition that record goes?
H S Reddy
- 75
- 5
1
vote
1 answer
Does Kafka maintain the In-sync replication factor after leader goes down?
Can someone please explain the in-sync replication factor maintainance in Kafka in the below scenario:
Scenario:
Consider, initially we have replication factor as 3.
In Kafka if the leader goes down one of the in-sync replicas becomes the new…
Nilay Narlawar
- 15
- 6
1
vote
1 answer
Does kafka support millions of partitions?
Will we have any problem if we have millions of partitions for one topic?
Due to our business requirement, we are thinking if we can make a partition for every user in kafka.
We have millions of users.
Any insight would be appreciated!
Moon
- 321
- 1
- 9
1
vote
1 answer
How to check kafka topic's partitions memory usage?
how can I check memory usage on specific topic's partitions in kafka? I tried this one "kafka-log-dirs.sh --describe...", but I'm not sure what output means:
"partition": "simpleTopic-8",
…
Rapiernik
- 61
- 7
1
vote
1 answer
What happens to the offsets that are compacted in a kafka partition?
Let us say, I have a partion-0 with 2 log segments. The first segment contains keys v1, v2 and v3 at offsets 0, 1 and 2. The second segment has keys v3, v4, v1 at offsets 3, 4 and 5.
When the log compaction happens, the keys at offset 0 and 2 in…
swetha
- 65
- 6
1
vote
1 answer
Does Kafka consumer reads the message from active segment in the partition?
Let us say I have a partition (partition-0) with 4 segments that are committed and are eligible for compaction. So all these segments will not have any duplicate data since the compaction is done on all the 4 segments.
Now, there is an active…
swetha
- 65
- 6
1
vote
1 answer
Does Kafka chose an alternative partition in the same topic if the partition I want to send message to was full of disk?
My topic has 3 partitions (in 3 different brokers). I want to send message to particular partition 1 (add partition id or specify a key in my message). Assuming the partition 1 became full of disk, does an alternative partition (partition 2 or 3 in…
Hieu Doan
- 85
- 5
1
vote
1 answer
How uber-go/zap logger logs to a Kafka topic?
Can I get an example to publish the logs to kafka topic. I am currently using uber-go/zap logger and using https://github.com/confluentinc/confluent-kafka-go to publish.
keerthi reddy
- 31
- 2
1
vote
3 answers
Kafka internal topic : Where are the internal topics created - source or target broker?
We are doing a stateful operation. Our cluster is managed. Everytime for internal topic creation , we have to ask admin guys to unlock so that internal topics can be created by the kafka stream app. We have control over target cluster not source…
Shweta Gulati
- 526
- 6
- 15
1
vote
1 answer
kafka custom partitioner and repartitioning
We can specify custom partitioner for kafka topics. So the kafka producer can deterministically send message to a particular partition based on certain custom algorithm.
Now the question is, when I increase the number of partitions then How will…
Durgesh O Mishra
- 41
- 1
- 2
- 5
1
vote
0 answers
Speed up deletion of older records with same key (log compacted topic)
I am running Kafka broker (version 2.12) with the default server setting on my Mac. I created a log compacted topic, with the following configuration -
Configs:…
Ahmed A
- 2,702
- 5
- 31
- 50