Stack Overflow
Stack Overflow

Questions tagged [apache-kafka]

Apache Kafka is a distributed streaming platform designed to store and process high-throughput data streams.

Apache Kafka

Apache Kafka is a distributed streaming platform. It lets you

  • Publish and subscribe to streams of data like a messaging system
  • Store streams of data in a distributed, replicated cluster
  • Process streams of data in real-time

Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. It also grants access to the complete history of the streams unlike a database, where you only see its current state.

Links

Related Tags

24030 questions
417
votes
16 answers

When to use RabbitMQ over Kafka?

I've been asked to evaluate RabbitMQ instead of Kafka but found it hard to find a situation where a message queue is more suitable than Kafka. Does anyone know use cases where a message queue fits better in terms of throughput, durability, latency,…
Joe
  • 8,787
  • 14
  • 63
  • 129
253
votes
7 answers

Using Kafka as a (CQRS) Eventstore. Good idea?

Although I've come across Kafka before, I just recently realized Kafka may perhaps be used as (the basis of) a CQRS, eventstore. One of the main points that Kafka supports: Event capturing / storing, all HA of course. Pub / sub architecture Ability…
Geert-Jan
  • 16,760
  • 10
  • 68
  • 121
242
votes
3 answers

Understanding Kafka Topics and Partitions

I am starting to learn Kafka for enterprise solution purposes. During my readings, some questions came to my mind: When a producer is producing a message - it will specify the topic it wants to send the message to, is that right? Does it care about…
Pinidbest
  • 2,676
  • 4
  • 13
  • 15
218
votes
21 answers

Purge Kafka Topic

Is there a way to purge the topic in kafka? I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error: kafka.common.InvalidMessageSizeException: invalid message size Increasing the fetch.size is…
Peter Klipfel
  • 4,441
  • 2
  • 25
  • 43
212
votes
25 answers

Leader Not Available Kafka in Console Producer

I am trying to use Kafka. All configurations are done properly but when I try to produce message from console I keep getting the following error WARN Error while fetching metadata with correlation id 39 : {4-3-16-topic1=LEADER_NOT_AVAILABLE}…
Vishesh
  • 2,798
  • 4
  • 18
  • 32
200
votes
3 answers

What determines Kafka consumer offset?

I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it will start reading from is determined…
Asif Iqbal
  • 3,450
  • 4
  • 20
  • 29
176
votes
4 answers

Data Modeling with Kafka? Topics and Partitions

One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?". I've read and watched some introductory materials. In particular, take, for example, Kafka: a…
David J.
  • 28,507
  • 20
  • 108
  • 162
159
votes
13 answers

Is Zookeeper a must for Kafka?

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using…
Paaji
  • 1,999
  • 3
  • 12
  • 11
133
votes
14 answers

How to find the kafka version in linux

How to find the kafka version in linux? whether there is a way to find the installed kafka version other than mentioning the version while downloading it?
kumar
  • 2,335
  • 5
  • 18
  • 26
132
votes
8 answers

How can I send large messages with Kafka (over 15MB)?

I send String-messages to Kafka V. 0.8 with the Java Producer API. If the message size is about 15 MB I get a MessageSizeTooLargeException. I have tried to set message.max.bytesto 40 MB, but I still get the exception. Small messages worked without…
Sonson123
  • 9,321
  • 10
  • 46
  • 68
129
votes
3 answers

Is key required as part of sending messages to Kafka?

KeyedMessage keyedMessage = new KeyedMessage(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed…
gaurav
  • 2,214
  • 6
  • 21
  • 26
117
votes
2 answers

Kafka: Consumer API vs Streams API

I recently started learning Kafka and end up with these questions. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world. How Stream is different as this…
sabtharishi
  • 2,371
  • 5
  • 18
  • 24
108
votes
3 answers

What's the purpose of Kafka's key/value pair-based messaging?

All of the examples of Kafka | producers show the ProducerRecord's key/value pair as not only being the same type (all examples show ), but the same value. For example: producer.send(new ProducerRecord("someTopic",…
smeeb
  • 22,487
  • 41
  • 197
  • 389
104
votes
7 answers

Apache Kafka vs Apache Storm

Apache Kafka: Distributed messaging system Apache Storm: Real Time Message Processing How we can use both technologies in a real-time data pipeline for processing event data? In terms of real time data pipeline both seems to me do the job…
Ananth Duari
  • 2,713
  • 11
  • 33
  • 39
103
votes
2 answers

Difference between Redis and Kafka

Redis can be used as realtime pub-sub just as Kafka. I am confused which one to use when. Any use case would be a great help.
Sweta Sharma
  • 1,356
  • 2
  • 8
  • 13
1
2 3
…
99 100