Stack Overflow
Stack Overflow

Questions tagged [kafkacat]

Kafkacat is a generic non-JVM producer and consumer for Apache Kafka >= 0.8

From the kafkacat GitHub page:

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.

In producer mode kafkacat reads messages from stdin, delimited with a configurable delimiter (-D, defaults to newline), and produces them to the provided Kafka cluster (-b), topic (-t) and partition (-p).

In consumer mode kafkacat reads messages from a topic and partition and prints them to stdout using the configured message delimiter.

There's also support for the Kafka >=0.9 high-level balanced consumer, use the -G switch and provide a list of topics to join the group.

kafkacat also features a Metadata list (-L) mode to display the current state of the Kafka cluster and its topics and partitions.

Supports Avro message deserialization using the Confluent Schema-Registry, and generic primitive deserializers (see examples below).

kafkacat is fast and lightweight; statically linked it is no more than 150Kb.

31 questions
6
votes
3 answers

Kafkacat: how to delete a topic or all its messages?

I am looking for a way to delete a topic or all its messages using kafkacat. Is it possible or the only way is through the script listed here? bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic
1Z10
  • 1,672
  • 2
  • 15
  • 48
6
votes
1 answer

How to resolve "Leader not available" Kafka error when trying to consume

I'm playing around with Kafka and using my own local single instance of zookeeper + kafka and running into this error that I don't seem to understand how to resolve. I started a simple server per the Apache Kafka Quickstart Guide $…
Jon Erickson
  • 102,662
  • 42
  • 131
  • 170
4
votes
4 answers

MacOS throwing dyld Error when running kafkacat

I want to use kafkacat command line utility(https://docs.confluent.io/current/app-development/kafkacat-usage.html) on my MacOS Mojave(10.14.5). I executed the below steps brew install kafkacat kafkacat -C -b brokerID -t kafkaTopic -e -K '+' I am…
Sujan Davangere Sunil
  • 201
  • 3
  • 12
3
votes
0 answers

Kafkacat Produce message from a file with headers

I need to produce batch messages to Kafka so I have a file that I feed kafkacat: kafkacat -b localhost:9092 -t -T -P -l /tmp/msgs The content of /tmp/msgs is as follows -H "id=1" {"key" : "value0"} -H "id=2" {"key" : "value1"} When I…
Jenia Ivanov
  • 1,849
  • 1
  • 25
  • 49
2
votes
1 answer

How copy some message from one kafka topic to another from bash?

pls help We have 2 kafka topic. I want copy 10 message from beginning from topic1 to topic2. I`m try do it with kafka-console-consumer and kafka-console-producer First i save 10 message from topic1 to some directory: for (( i=1; i<=10; i++ )); do…
Sergey
  • 99
  • 2
  • 11
2
votes
1 answer

How to use kafkacat with message-hub

In trying to use kafkacat with message hub, I've used the following: kafkacat -X client.id=xxxxx \ -X sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxx" password="xxxx";' \ -X…
Zach
  • 43
  • 8
2
votes
1 answer

kafkacat's -o (offset to start consuming from) option

Will execute the following kafkacat command with the -o (offset to start consuming from) option but without the -G (group id) option affect other consumer groups? kafkacat -C -b 10.52.1.1:9092,10.52.1.2:9092,10.52.1.3:9092 -t MyTopic -o beginning
ytw
  • 1,253
  • 2
  • 19
  • 40
2
votes
1 answer

How can I install and configure KafkaCat in windows machine?

I am trying to install kafkaCat tool in windows machine. While doing a quick build by running ./bootstrap.sh, I am getting following error: $ ./bootstrap.sh Directory librdkafka already exists, not downloading…
Praveen
  • 1,218
  • 1
  • 14
  • 25
1
vote
2 answers

Force kafka to connect brokers through IPs, not via hostnames

We have following kafka-ssh-tunneling setup. ssh -N $JUMPHOST -L 2181:w.x.y.z:2181 -L 9092:a.b.c.d:9092 -L 9091:e.e.f.f:9092 broker IP is a.b.c.d , There is local lo0 device alias with same IP address zookeper IP is w.x.y.z , There is local device…
libxelar.so
  • 320
  • 1
  • 10
1
vote
2 answers

Problem with ADVERTISED_LISTENER on macos

I start kafka with this docker-compose.yml on my Mac: mydb: [...] environment: kafka_bootstrap_servers: kafka:9093 kafka: image: wurstmeister/kafka:2.13-2.6.0 ports: - "9092:9092" environment: …
Frank
  • 59
  • 4
1
vote
1 answer

Connecting to kafka from another namespace inside k8s

I have the following configuration for my kafka and zookeeper in my minikube: apiVersion: v1 kind: Service metadata: name: kafka-service namespace: kafka spec: selector: app: kafka ports: - protocol: TCP port: 9092 name:…
Rodolfo
  • 709
  • 2
  • 6
  • 25
1
vote
2 answers

How to install kafkacat on kafka container resulted from docker-compose

My context is I am trying to create a docker-compose wich will start few containers for running ELK+FileBeats and 3 Kafka containers. The whole docker compose and its yml configuration files can be found in github docker-compose To sumarize, this is…
Jim C
  • 2,519
  • 13
  • 54
  • 105
1
vote
1 answer

How to turn a kafkacat payload printout into binary

I can use kafkacat to get a tuple from a topic and print it out: kafkacat -b kafka10.myorg.com:9092 -t MyTopic -o -1 -f '%s\n' And I get something like this in my terminal: �ǐے�ғ�����������ǐے�ғ���������S1_153314S3_153314S4_5422973S2_5420991... Is…
breezymri
  • 2,867
  • 7
  • 25
  • 52
1
vote
1 answer

trying to connect to heroku kafka from local computer using kafkacat: ssl.ca.location failed: No error

I am trying to follow these instructions for connecting locally to my heroku kafka using kafkacat: https://gist.github.com/crcastle/cb21c2148fc57ad89753bf28b561d910 I am creating an env file like this: heroku config -s > .env and then running this…
Alex028502
  • 2,415
  • 12
  • 28
1
vote
1 answer

Performance issues running kafacat over slow speed link

I have weird performance issues with fetch.max.message.bytes parameter in librdkafka consumer implementation (version 0.11). I run some tests using kafkacat over slow speed network link (4 Mbps) and received following results: 1024 bytes =…
meaclum
  • 96
  • 5
1
2 3