2

The below code snippet is working fine. But sometimes it's not reading messages from Kafka's topic. I am not getting any errors. At Kafka's side(we are using Aiven Managed Kafka), the consumer group has associated with the topic and consumer script is running nicely.

I need your guidance to resolve the above issue.

Kafka Version - 2.0.1

Node Module Version - "node-rdkafka": "^2.7.0"

Code Ref - https://github.com/Blizzard/node-rdkafka

const Kafka = require('node-rdkafka');
const path = require('path');
console.log(Kafka.features);
const consumer = new Kafka.KafkaConsumer({
      'group.id': 'testxyz',
      'metadata.broker.list': "kafka-production-url",
      'security.protocol': 'SSL',
      'ssl.key.password': "password",
      'ssl.key.location': '/var/www/html/config/service.key',
      'ssl.certificate.location': '/var/www/html/config/service.cert',
      'ssl.ca.location': '/var/www/html/config/ca.pem',
      'socket.keepalive.enable': true,
      'enable.auto.commit': false,
      'rebalance_cb': function(err, assignment) {

    if (err.code === Kafka.CODES.ERRORS.ERR__ASSIGN_PARTITIONS) {
        // Note: this can throw when you are disconnected. Take care and wrap it in
        // a try catch if that matters to you
        this.assign(assignment);
    } else if (err.code == Kafka.CODES.ERRORS.ERR__REVOKE_PARTITIONS){
        // Same as above
        this.unassign();
    } else {
        // We had a real error
        console.error(err);
    }

      },
      'offset_commit_cb': function (err: any, topicPartitions: any) {
        if (err) {
          // There was an error committing
          console.error("error in commit", err);
        } else {
          // Commit went through. Let's log the topic partitions
          console.log("Success in commit", topicPartitions);
        }
      }
    }, 
'auto.offset.reset': 'earliest');

consumer.on('ready', {
    consumer.subscribe("mytopic");
    consumer.consume();
});

// handle the messgae
consumer.on('data', {
    console.log(data);
});

consumer.on('disconnected', {
    console.log("onDisconnected",args);
});
consumer.connect();
Stennie
  • 57,971
  • 14
  • 135
  • 165
Priyabrata
  • 391
  • 2
  • 17
  • If you describe the consumer group, what do you see? – OneCricketeer Jan 03 '20 at 08:15
  • Your code doesn't quite look like the example https://github.com/Blizzard/node-rdkafka/blob/master/README.md#standard-api-1 – OneCricketeer Jan 03 '20 at 08:24
  • @cricket_007This is working code. Could you please let me know "what am I missing here"? I will share the o/p of describing the consumer group soon. – Priyabrata Jan 04 '20 at 07:59
  • I'm sure "it works" just fine. I'm only pointing out that following the documentation examples makes it more easier to identity what might be wrong – OneCricketeer Jan 04 '20 at 09:30
  • I am experiencing a very similar issue. The client connects, the server is aware of the consumer, but the rebalance event is never triggered, the consumer is never assigned, and it just sits there, not receiving any messages. – Leighton Wallace Jan 31 '20 at 05:30

0 Answers0