Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

150 questions
0
votes
1 answer

kafka producer - how to know which key will go to what partition?

I am aware of kafka console producer command and it is quite handy. I want to know is there a way to know for sure which key will go to which partition when we have key as well? Assuming we have 10 partitions in kafka topic, how will producer decide…
CuriousMind
  • 6,665
  • 12
  • 49
  • 95
0
votes
2 answers

Does the same Kafka partitions of the same topic on different brokers contains the same messages ( duplicate each other)?

Broker 1: +-------------------+ | Topic 1 | | Partition 0 | | | | | | Topic 2 | | Partition 1 | +-------------------+ Broker 2: +-------------------+ | Topic 1 | | …
J.J. Beam
  • 1,404
  • 8
  • 24
0
votes
2 answers

What is the ideal number of partitions in kafka topic?

I am learning Kafka and trying to create a topic for my recent search application. The data being pushed to kafka topics is assumed be a high number. My kafka cluster have 3 brokers and there are already topics created for other requirements. Now…
KayV
  • 9,560
  • 8
  • 68
  • 111
0
votes
0 answers

Unable to delete Kafka topic using AdminClient deleteTopic

I am using Kafka_2.12 version 2.3.0. For topic delete purpose I am using the below code snippet AdminClient adminClient = KafkaAdminClient.create(conf); adminClient.deleteTopics(Collections.singleton(topic)); But when the above code is being…
Abhishek Gharai
  • 157
  • 1
  • 11
0
votes
1 answer

kafka is not electiong new leader after 2 nodes are down

we test scalein in enviorment with 3 kafka nodes each one has a broker and zookeeper service on in the replica.factor=2 after the service went down on 2 nodes the leader of the partition stayed the old one and didnt movve to the only node that is…
NoamiA
  • 395
  • 1
  • 10
0
votes
0 answers

Streaming database data to Kafka topic without using a connector

I have a use case where I have to push all my MySQL database data to a Kafka topic. Now, I know I can get this up and running using a Kafka connector, but I want to understand how it all works internally without using a connector. In my spring boot…
Nithin Prasad
  • 474
  • 1
  • 6
  • 21
0
votes
1 answer

How to align my consumer to consume 2 depending topics

I have this use case that I have 2 topics Topic 1 (Units) -> P0 / Topic 2 (Reservations) -> P0 I have a single consumer that needs to have the up to data from both topics/partitions in order to take the correct decision (either delete a unit if not…
Ahmed Alaa El-Din
  • 1,663
  • 10
  • 19
0
votes
1 answer

How do I ask kafka which topics have extraordinary overrides?

I understand my default retention period, but I want to see if any topics have special overrides for the retention period. I've tried using the kafka-topics, but it doesn't connect with zookeeper. kafka-topics --zookeeper localhost:2181 --describe…
Micah Rothenbuhler
  • 66
  • 4
0
votes
1 answer

source kafka connect not publish with timestamp mode

I try to create a source Kafka connect with timestamp mode in Windows. I have this table: CREATE TABLE "usu"."mytable" ( "first_name" CHAR(8 BYTE) DEFAULT ' ', "last_name" CHAR(8 BYTE) DEFAULT ' ', "regist" TIMESTAMP (0) DEFAULT…
Lanre
  • 91
  • 6
0
votes
1 answer

Kafka: Topic vs Partition data

Going through Kafka documentation and various other resources, I understand that messages in Kafka are organized into topics. Also, a topic can be broken down into partitions and each partition can be hosted on a different server. This gives…
Mandroid
  • 3,359
  • 3
  • 28
  • 59
0
votes
1 answer

Delete messages and reset topic offset to 0 irrespective of consumer group

I need to create a Kafka setup where at the start of each day, all messages in a topic must get deleted and its offset reset to 0. I have tried to do this by setting log.retention to 1 day but it didn't seem to work. I also ran…
SRMara
  • 11
  • 4
0
votes
1 answer

How to mount volume for kafka through docker-compose file?

I am using "wurstmeister/kafka" docker image with latest tag. Whenever I tried to stop & start the kafka container, it will start container with default configuration. How can I mount volume, so that data persists even when container stops or…
Bhavik Shah
  • 11
  • 5
0
votes
1 answer

How to read Kafka Topic line by line in Flink program

First, I load a CSV file in Kafka topic and I can print the Topic via Flink program in. The code is in following: final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); Properties prop = new Properties(); …
M_Gh
  • 660
  • 1
  • 9
  • 24
0
votes
0 answers

NestedError: topic/partition change check failed

Suddenly I got an issue in running process of apache kafka consumer group. Anyone please share your thoughts on why this issue happens. NestedError: topic/partition change check failed at _checkTopicPartitionChange…
Bhavik Shah
  • 11
  • 5
0
votes
2 answers

High Scalability Question: How to sync data across multiple microservices

I have the following use cases: Assume you have two micro-services one AccountManagement and ActivityReporting that processes event U. When a user registers, event U containing the user information will published into a broker for the two…
F.O.O
  • 4,024
  • 4
  • 20
  • 31
1 2 3
…
9
10