Questions tagged [kafkacat]

Kafkacat is a generic non-JVM producer and consumer for Apache Kafka >= 0.8

From the kafkacat GitHub page:

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.

In producer mode kafkacat reads messages from stdin, delimited with a configurable delimiter (-D, defaults to newline), and produces them to the provided Kafka cluster (-b), topic (-t) and partition (-p).

In consumer mode kafkacat reads messages from a topic and partition and prints them to stdout using the configured message delimiter.

There's also support for the Kafka >=0.9 high-level balanced consumer, use the -G switch and provide a list of topics to join the group.

kafkacat also features a Metadata list (-L) mode to display the current state of the Kafka cluster and its topics and partitions.

Supports Avro message deserialization using the Confluent Schema-Registry, and generic primitive deserializers (see examples below).

kafkacat is fast and lightweight; statically linked it is no more than 150Kb.

31 questions
1
vote
4 answers

Kafka deployment on minikube

Hi i am new to kubernets, i am using minikube single node cluster for local development and testing. Host: Ubuntu 16.04 LTS. Minikube: Virtual box running minikube cluster My requirement is i need to deploy kafka and zookeeper on minikube and…
user1184777
  • 847
  • 1
  • 16
  • 35
0
votes
1 answer

Kerberros GSSAPI doesn't work within kafkacat alpine container

Previously I've reported it into kafkacat tracker but the issue has been closed as related to cyrus-sasl/krb5. podman run --rm -it --name kafkacat-DEV \ -v$(pwd)/conf/integration:/conf -v$(pwd)/conf/integration/krb5.conf:/etc/krb5.conf \ …
Hubbitus
  • 4,481
  • 2
  • 37
  • 42
0
votes
1 answer

Kafkacat unable to send messages to topics

I'm new to Apache Kafka and Kafkacat trying to learn new things from here: https://www.youtube.com/watch?v=5Mgni6AYnWg&t=123s at time 17:32. I setup a kafka and kafkacat on my remote VM. [user@user bin]$ ./kafka-topics.sh --list --zookeeper…
Pra_A
  • 7,134
  • 12
  • 89
  • 170
0
votes
1 answer

Kafkacat how to republish a binary message maintaining key

I have been trying to use kafkacat to find a message in a topic and publish it back into the topic. We use protobuf so the message values should be in bytes (Keys can be different such as strings or bytes). However, I am unable to publish the…
0
votes
0 answers

Is it Possible to modify or update message of a topic by kafkacat?

I am trying to built an alert streaming application with confluent platform community & ksql headless mode. My application flow is I produce message (json format which have column like threshold, alert_flag, device_id etc.) to a base ksql stream…
Saiful Islam
  • 116
  • 1
  • 10
0
votes
1 answer

how manually analyse headers from kafka message topics? Getting confluentinc_kafkacat_1 exited with code 2

Goal: I want to analyse the headers from a topic and I am looking for some straighforward way to see the header. So I don't want to develop an extra application or extense code just for that. Any straighforward tool to see the header will be…
Jim C
  • 2,519
  • 13
  • 54
  • 105
0
votes
0 answers

kafkacat returning content of topic half of the time

How is it possible that one of two time that I run kafkacat, it's not returning anything from the topic? bash-3.2$ kafkacat -b localhost:9092 -t "what-is-going-on2" -C -c1 {"hello": "world"} bash-3.2$ kafkacat -b localhost:9092 -t…
Jenia Ivanov
  • 1,849
  • 1
  • 25
  • 49
0
votes
1 answer

Configuration of a specific topic. Kafkacat

I have a topic "topic-one" and I want to know if it has "log.cleanup.policy = compact" configured or not. Is it possible with kafkacat, extract the properties and / or configuration of a specific topic?
Jose
  • 1,569
  • 1
  • 17
  • 38
0
votes
2 answers

Syntax error while using COPY to fill specific columns in Postgres table from stdin

I have a syntax error in my bash command when I try to fill a Postgres table with data from Kafka using kafkacat: [k ~]$ kafkacat -b XXX.XX.Y.Z:9092 -t test -o 20501 -f '%k|%s|%T|%p|%o|213\n' -e | psql -c "copy raw_from_kafka(key, value, timestamp,…
Konstantin Popov
  • 1,017
  • 1
  • 11
  • 17
0
votes
1 answer

Kafka on Kubernetes in minikube not working

I am trying to setup Kafka on Minikube, a very basic setup. I can't validate if Kafka and Zookeeper have been setup correctly because kafkacat fails. Here is my config: zookeeper kind: Deployment apiVersion: apps/v1 metadata: name:…
maopuppets
  • 190
  • 3
  • 15
0
votes
1 answer

Set timestamp using kafkacat when producing

I want to be able to transmit messages to kafka using kafkacat but also set the timestamp of the messages. But it seems like this is not possible. Is this true or is there something that I am missing. I can't seem to find a full kafkacat reference…
Steven
  • 3
  • 1
0
votes
1 answer

How to talk to Kafka KUDO Internal Setup?

I have Kafka setup via KUDO: https://kudo.dev/docs/runbooks/kafka/external-access.html#internal-access KUDO Kafka has a headless service. How do I go about creating topics for kafka? I have installed kafkacat CLI…
wwjdm
  • 2,178
  • 6
  • 27
  • 51
0
votes
1 answer

Cannot write to kafka, brokers are down

I'm running Kafka locally. When I try to write to Kafka I get the following error: kafkacat -b localhost:9092 -t req -T -P -l msgs hello world % ERROR: Local: Broker transport failure: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed:…
Jenia Ivanov
  • 1,849
  • 1
  • 25
  • 49
0
votes
2 answers

Linux kafka producer cli tool with round-robin

I have a worker in spring-boot which listen to kafka topic with 20 partitions. I created the following listener: @KafkaListener(topics = "mytopic") public void listen(@Payload(required = true) IncomingMessage msg, …
toto
  • 715
  • 1
  • 5
  • 16
0
votes
1 answer

Find the lead of kafka cluster using kafkacat

I have kafka cluster in docker containers created using this tutorial. When I run kafkacat -L -b kafka-1:19092 i get: Metadata for all topics (from broker 1: kafka-1:19092/1): 3 brokers: broker 2 at kafka-2:29092 broker 3 at kafka-3:39092 …
Rudziankoŭ
  • 8,411
  • 10
  • 66
  • 146