1

I can use kafkacat to get a tuple from a topic and print it out:

kafkacat -b kafka10.myorg.com:9092 -t MyTopic -o -1 -f '%s\n'

And I get something like this in my terminal:

�ǐے�ғ�����������ǐے�ғ���������S1_153314S3_153314S4_5422973S2_5420991...

Is it possible for me to convert the printed out payload to binary format so I can feed that to my local code to debug?

breezymri
  • 2,867
  • 7
  • 25
  • 52

1 Answers1

3

You can consume the messages directly as binary data without applying formatting.

kafkacat -b kafka10-staging.salesforceiq.com:9092 -t MyTopic -o -1 -D "" > out.file

The usage of -D "" is there to avoid adding the "\n" to the binary data. https://github.com/edenhill/kafkacat/issues/72

Shay
  • 46
  • 5
  • 1
    Unless the messages contains a length or delimiter to allow separating messages in the file, you might want to append the length of each message by `-f %R%s` – Edenhill Dec 10 '18 at 10:12
  • It makes sense that I'd want to prefix each payload with something, but I was thinking it would be something constant so I could use something like awk's record separator to split the payloads back apart. @Edenhill would you mind explaining how you would make use of the payload length printed in the output? I like the idea, I'm just not sure how to take advantage of it. – Jonny Appleseed Apr 01 '21 at 18:07