Questions tagged [clj-kafka]
12 questions
9
votes
1 answer
Kafka Only One Consumer in Consumer Group Getting Messages
In my setup, I have a consumer group with three processes (3 instances of a service) that can consume from Kafka. What I've found to be happing is that the first node is receiving all of the traffic. If one node is manually killed, the next node…
joshuaegclark
- 3,611
- 3
- 14
- 12
4
votes
0 answers
Kafka Leader not found for Topic and Partition
I have been facing an issue, some of the partitions for multiple topics on the Kafka cluster have no leader and even no replica in the isr(in-sync replica) set.
This has happened for all the partitions of a particular broker id - 5.
topic: topic1 …
ashendra bansal
- 41
- 1
- 4
4
votes
1 answer
KAFKA consumer and partition
1) Can any any one explain how KAFKA consumer knows about partition and consume the data from specific partition?.
What is consumer group?
2) I have 3 partitions and data stored in all partitions,
for example ( msg1, msg2 , msg3 and msg4).
Msg1…
madheswaran
- 196
- 11
3
votes
1 answer
Kafka's ZookeeperConsumerConnector.createMessageStreams never returns
I'm trying to retrieve data from my Kafka 0.8.1 cluster. I have brought into existence an instance of ZookeeperConsumerConnector and then attempt to call createMessageStreams on it. However, no matter what I do, it seems createMessageStreams just…
G Gordon Worley III
- 7,474
- 5
- 40
- 75
2
votes
1 answer
clj-kafka - consumer empty
I am trying usgin library clj-kafka.
Here my code
(use [clj-kafka.producer]
[clj-kafka.zk]
[clj-kafka.consumer.zk]
[clj-kafka.core]))
(brokers {"zookeeper.connect" "localhost:2181"})
(def p (producer {"metadata.broker.list"…
Édipo Féderle
- 3,799
- 4
- 27
- 31
2
votes
1 answer
Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/spout/MultiScheme
I am a trying to integrate Storm with Kafka. I am running a Storm topology using KafkaSpout. This is the basic word-count topology. I am using Apache Kafka as the source and storm for processing the data. While submitting the topology I am facing…
user4115825
2
votes
2 answers
Kafka Error: Could not find or load main class org.apache.kafka.clients.tools.ProducerPerformance
I was trying to run Performance test as per the following article for Apache Kafka
Reference:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Linked-in tests:…
Mayukh Roy
- 1,820
- 3
- 18
- 31
2
votes
1 answer
How to have sync producer batch messages?
We were using the Kafka 0.8 async producer but it is dropping messages (and there is no aysnc response from another thread or we could keep using async).
We have set the batch.num.messages to 500 and our consumer is not changing. I read that…
Dean Hiller
- 17,183
- 19
- 103
- 176
1
vote
1 answer
how configure setting for kafka to delete messages?
I'm using Kafka (with Yahoo Kafka manager)
I want to set a rule for resetting messages or how they call it : "Sum of partition offsets"
Is there any parameter of rolling the kafka offsets at server.properties?
(i.e: I want to reset or delete all the…
user5467599
0
votes
1 answer
Figuring out retention time using clj-kafka?
Is there a way to discover a Kafka topic retention time during runtime using clj-kafka?
shakedzy
- 2,392
- 5
- 27
- 50
0
votes
1 answer
Check if Kafka Queue is Empty
Right now I have functionality that writes a couple hundred messages onto a kafka queue. But when all of those messages have been consumed I need to also execute additional functionality. Is there a way to place a listener on a kafka queue to get…
Jake
- 97
- 10
0
votes
1 answer
Clojure - core.async interface for apache kafka
I am using clj-kafka, and I am trying to make a core.async interface to it in the REPL.
I am getting some messages, but my structure feels wrong : I either cannot stop receiving messages, or have to launch the go routine again to receive more…
nha
- 16,039
- 11
- 76
- 108