0

I'm playing around with the consumer.c example file for librdkafka and I'm trying to figure out how to convert the rkm payload (that is printed out at line 244) to json so that I can grab a parameter's value from the json.

Right now I'm using jansson but am having some problems with it which I can expand on if needed.

Is there a feature for this in librdkafka or the standard C libraries that I'm unaware of?

Jordan
  • 17
  • 1
  • 5

1 Answers1

0

Simply pass the message payload (binary) and length to Jansson, like so:

/* For each consumed message .. */
rd_kafka_message_t *rkmessage = rd_kafka_consumer_poll(consumer, 1000);

if (!rkmessage) {
   /* No message available */
   continue;

} else if (rkmessage->err) {
   /* Handle consumer event/error (typically not fatal) */
   handle_consumer_error(rkmessage->err, "%s", rd_kafka_message_errstr(rkmessage));

} else if (rkmessage->len > 0) {
   /* Parse JSON value */
   json_error_t err;
   json_t *json = json_loadb(rkm->payload, rkm->len, JSON_ALLOW_NUL, &err);
   if (!json) {
      handle_consumer_error(RD_KAFKA_RESP_ERR__BAD_MSG,
                            "Failed to parse JSON for message at %s [%"PRId32"] offset %"PRIu64: line %d: %s\n",
                            rd_kafka_topic_name(rkmessage->rkt), rkmessage->partition),
                            rkmessage->offset, err.line, err.text);
   } else {
      /* Process the message */
      process_message(json);

      json_decref(json);
   }
}

rd_kafka_message_destroy(rkmessage);

See the Jansson docs for more info: https://jansson.readthedocs.io/en/latest/tutorial.html

Edenhill
  • 2,298
  • 17
  • 30
  • Is handle_consumer_error() and process_message() apart of librdkafka or jansson? I get the response `undefined reference to process_message'` when I use them. – Jordan Jan 20 '20 at 15:24
  • Those are your functions, i.e., what you do on consumer error (typically just log), and how you process the received message. – Edenhill Jan 20 '20 at 15:46
  • Ah, ok. Well, I used `json_t *json = json_loads(rkm->payload, rkm->len, & err);` and the if statement was executed meaning there was an error. Not sure what's going wrong. – Jordan Jan 20 '20 at 16:37
  • You can't use json_loads() (string) since the payload is not null-terminated, you need to use json_loadb() (binary) as in the example above. – Edenhill Feb 03 '20 at 09:47