Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

102 questions
0
votes
1 answer

unable to run librdkafka=1.3.0 over docker

I was trying to run librdkafka version 1.3.0 from alpine distribution over my docker container using this: FROM golang:1.13.6-alpine3.10 as base RUN apk add --no-cache --update librdkafka=1.3.0 librdkafka-dev=1.3.0 --update-cache --repository…
Khandelwal-manik
  • 313
  • 3
  • 9
0
votes
1 answer

KafkaConsumer CPP API assign() with auto commit

I have a CPP Kafka consumer which uses assign to specify the partitions. Since I assign the partitions using assign() and not use the subscribe() which I am fine with. Because of this, my the re-balancing doesn't take place which also I am fine…
Vinodini Natrajan
  • 95
  • 7
0
votes
0 answers

Transient "Local: Broker transport failure"

I have a thin wrapper over the librd Kafka Consumer which has a separate poll thread that would call the consume method. When the application does forking, the poll thread is paused and once the forking is done, the poll thread resumes. At the time…
Vinodini Natrajan
  • 95
  • 7
0
votes
0 answers

Kafka Broker Version Question and enable.idempotence

Probably a noob question: I have started a Bitnami Kafka Container from bitnami/kafka:2 In startup messages it claims to be Version 2.5 Now i wanted to use enable.idempotence but when i try this with librdkafka 1.4.0, i get an error message…
Blafasel42
  • 91
  • 4
0
votes
1 answer

Confluent.Kafka for UWP application. (HoloLens)

I am creating a UWP application for Hololens and I need the Kafka-Client. I used the pre-built .dlls from confluent-kafka-dotnet (https://github.com/confluentinc/confluent-kafka-dotnet) and librdkafka (https://github.com/edenhill/librdkafka) but…
Dimitris
  • 3
  • 3
0
votes
1 answer

Cannot pipenv install confluent-kafka 1.4.0 from source (pypi) - no workaround seems to work

There seems to be an issue logged with the current Confluent-Kafka package on pypi: I have a Dockerfile with the foll code which used to work until the issue happened : RUN cd /tmp && git clone https://github.com/edenhill/librdkafka.git \ && cd…
banditKing
  • 8,915
  • 26
  • 89
  • 151
0
votes
1 answer

Python confluent_kafka: consume(0) cannot trigger callbacks

I am using confluent-kafka-python and librdkafka. But I think, confluent_kafka.Consumer.consume(0) cannot trigger stats_cb. Consumer_consume(0) => rd_kafka_consume_batch_queue => rd_kafka_q_serve_rkmessages In, rd_kafka_poll_cb is called in the…
BAE
  • 7,178
  • 12
  • 57
  • 136
0
votes
1 answer

Kafka credentials and consumer parameters for c++ client

I am trying to consume kafka msg using c++. I have installed librdkafka and trying the example https://github.com/edenhill/librdkafka/blob/master/examples/rdkafka_example.cpp Now my concern is how to pass username, password and truststore to kafka…
Sarada Chelluboyena
  • 322
  • 1
  • 12
0
votes
1 answer

Does Kafka Consumer reconnect and resubscribe to topics after cluster goes down and comes back up

kafka consumer using librdkafka (high level consumer) connected to kafka cluster and subscribed to 10 topics and consuming data. There was assign partitions event. There was network issue due to which cluster was not reachable. Lost connection with…
Youli Luo
  • 121
  • 1
  • 9
0
votes
1 answer

How to convert librdkafka payload to json to get a parameter value?

I'm playing around with the consumer.c example file for librdkafka and I'm trying to figure out how to convert the rkm payload (that is printed out at line 244) to json so that I can grab a parameter's value from the json. Right now I'm using…
Jordan
  • 17
  • 1
  • 5
0
votes
1 answer

how to re-send a failed message properly in librdkafka?

static void DeliverCallback(rd_kafka_t *rk, void *payload, size_t len, rd_kafka_resp_err_t error_code, void *opaque, void *msg_opaque) { if (error_code != RD_KAFKA_RESP_ERR_NO_ERROR) { // todo resend the message } } my…
BunsenBurner
  • 21
  • 5
0
votes
0 answers

Avro C and librdkafka - Consumer implementation; Getting NULL for parsing value of a field - always

Sample snippet of code KEY = (char *)malloc(sizeof(argRkMessage->key)+1); if( KEY != NULL ) { //Logger } reader_class = avro_generic_class_from_schema(KEYSchema); check_i(avro_generic_value_new(reader_class, &val)); …
Jyo
  • 1
0
votes
0 answers

How to set username and password for plaintext protocol (no SASL) with librdkafka?

I am trying to connect a producer with security.protocol conf property to plaintext. How to pass username and password to the Conf object before invoking RdKafka::Producer::create ? None of the set call seems to fit. I read the doc about sasl for…
norisknofun
  • 807
  • 1
  • 8
  • 22
0
votes
1 answer

How to pass multiple groupid & topic pairs to consume with node-rdkafka?

I am using node-rdkafka for one of my application. I want to consume messages from multiple topics with groupid.(groupid & topic pair) But I am unable to find any details on node-rdkafka documention. So my question is, is it possible to pass…
Sourabh Mishra
  • 13
  • 5
0
votes
0 answers

Amazon linux : node rdkafka setup not working

Please find OS Details below VERSION="2017.03" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2017.03" PRETTY_NAME="Amazon Linux AMI…
Priyabrata
  • 391
  • 2
  • 17
1 2 3 4 5
6
7