Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

102 questions
10
votes
1 answer

Kafka consumer gets stuck after exceeding max.poll.interval.ms

When the consumer does not receives a message for 5 mins (default value of max.poll.interval.ms 300000ms) the consumer comes to a halt without exiting the program. The consumer process hangs and does not consume any more messages. The following…
esquarer
  • 303
  • 3
  • 12
8
votes
1 answer

Python librdkafka producer perform against the native Apache Kafka Producer

I am testing Apache Kafka Producer with native java implementation against Python's confluent-kafka to see which has the maximum throughput. I am deploying a Kafka cluster with 3 Kafka brokers and 3 zookeeper instances using docker-compose. My…
Amit Tripathi
  • 5,967
  • 3
  • 25
  • 50
7
votes
1 answer

Kafka offset management: enable.auto.commit vs enable.auto.offset.store

A Kafka Consumer by default periodically commits the current offsets unless it is turned off by disabling enable.auto.commit. According to the documentation you're then responsible for committing the offsets yourself. So when I want manual control,…
Bastian Venthur
  • 4,062
  • 4
  • 22
  • 46
6
votes
1 answer

bazel rules_go: linking go binary against a static c++ library (.a file) produced by another target in the workspace

I'm using the confluent-kafka-go library inside my go binary, and this library needs to be linked against librdkafka. Other targets in my project use librdkakfa, so I have produced the static librdkafka.a and librdkafka++.a using rules_foreign_cc's…
mancini0
  • 3,075
  • 21
  • 25
5
votes
1 answer

librdkafka consumer and ssl configuration

I'm using librdkafka as a client consumer, and I have configured the broker and the client to support SSL, for the broker: listeners =…
Ismail
  • 1,793
  • 1
  • 7
  • 17
3
votes
1 answer

ignore tests in librd kafka

My golang project depends on librd kafka When I try to run go vet ./... or go test ./... from my jenkin, I get the following error. I beleive that is due to the fact that I am running ./... but even if I have to ignore the vendor I am not sure what…
Gayatri
  • 193
  • 1
  • 2
  • 15
3
votes
1 answer

How does RD_KAFKA_PARTITION_UA work in librdkafka?

I have a producer in php which publishes to queue. I am using php-rdkafka library for the same. Following is the code: $conf->set('log_level', LOG_DEBUG); $conf->set('debug', 'all'); $rk = new RdKafka\Producer($conf); …
Rohit Aggarwal
  • 43
  • 5
3
votes
1 answer

librdkafka not found by CMake

I'm trying to use Kafka in a project that uses CMake to be built. I have librdkafka-dev (version 0.8.6-1.1) installed. I also tried downloading the project from Git and building it manually, which seem to be successful. I noticed a README.md file in…
André Ferrari Moukarzel
  • 56
  • 6
3
votes
0 answers

node-rdkafka is not getting installed in local windows system as well as in remote linux server

node-rdkafka needs Python 2.7 as one of the dependency and it is already available in the system, path is also set for it. Still npm is throwing errors related to librdkafka as per my understanding form the stacktrace. Please find below the error…
Mrinal Deo
  • 61
  • 7
3
votes
1 answer

How does librdkafka producer learn about new topic partitions in Kafka

I'm running rdkafka_simple_producer.c to produce messages to a Kafka cluster. I have one topic and 30 partitions. Using the default round-robin partitioner. While the producer is working and generating messages to Kafka, I add more partitions to…
rodolk
  • 4,969
  • 3
  • 22
  • 32
2
votes
1 answer

How To Implement A Wait Retry Transient Fault Handling Policy?

I am fairly new to Kafka and Polly. I am seeking advice with respect to how to implement failure resiliency when using the Admin Client with Kakfa Confluent .NET client. I am using the Admin Client to create a topic if it does not already exist…
anon_dcs3spp
  • 802
  • 3
  • 23
2
votes
1 answer

librd Kafka config settings for kerberos authentication

My target cluster is kerberos enabled. From the console, the producer connection is working. ./kafka-console-producer.sh --broker-list targetHost:port --topic test --producer.config /path/to/client.properties I have added below entry in my…
Anil
  • 91
  • 2
  • 6
2
votes
1 answer

How to read message one by one in node-rdkafka

I'm using node-rdkafka (https://github.com/Blizzard/node-rdkafka) to consume messages, the basic setup is working fine but it triggers the function every time I push something to the queue, irrespective of completion of previous method. I want the…
Mayank Bansal
  • 23
  • 5
2
votes
0 answers

Sometimes node-rdkafka consumer not reading any messages from topic

The below code snippet is working fine. But sometimes it's not reading messages from Kafka's topic. I am not getting any errors. At Kafka's side(we are using Aiven Managed Kafka), the consumer group has associated with the topic and consumer script…
Priyabrata
  • 391
  • 2
  • 17
2
votes
1 answer

Error: Invalid value "sasl_ssl" for configuration property "security.protocol")

When trying to connect to kafka consumer ( kafka_2.11-2.0.0 in Centos ) using golang, getting below error : [ 2019_12_12_12:45:05 ] ----> Kafka Consumer Connection Error (kafka.Error=Invalid value "sasl_ssl" for configuration property…
Vani Polnedi
  • 425
  • 4
  • 13
1
2 3 4 5 6 7