Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

333 questions
66
votes
9 answers

Kafka in Docker not working

I am trying to use wurstmeister\kafka-docker image with docker-compose, but I am having real problems with connecting everything. All the posts or questions that I check, seems not to have any problems, but I am frankly lost. (And there are at…
nanounanue
  • 6,692
  • 7
  • 36
  • 65
36
votes
7 answers

How to programmatically create a topic in Apache Kafka using Python

So far I haven't seen a python client that implements the creation of a topic explicitly without using the configuration option to create automatically the topics.
jpgerek
  • 598
  • 1
  • 6
  • 13
28
votes
6 answers

How to get latest offset for a partition for a kafka topic?

I am using the Python high level consumer for Kafka and want to know the latest offsets for each partition of a topic. However I cannot get it to work. from kafka import TopicPartition from kafka.consumer import KafkaConsumer con =…
Saket
  • 2,671
  • 3
  • 25
  • 44
21
votes
2 answers

How to subscribe to a list of multiple kafka wildcard patterns using kafka-python?

I'm subscribing to Kafka using a pattern with a wildcard, as shown below. The wildcard represents a dynamic customer id. consumer.subscribe(pattern='customer.*.validations') This works well, because I can pluck the customer Id from the topic…
Ben Harrison
  • 1,771
  • 4
  • 19
  • 38
14
votes
7 answers

kafka-python: producer is not able to connect

kafka-python (1.0.0) throws error while connecting to the broker. At the same time /usr/bin/kafka-console-producer and /usr/bin/kafka-console-consumer work fine. Python application used to work well also, but after zookeeper restart, it no longer…
alex_123
  • 201
  • 1
  • 2
  • 7
13
votes
2 answers

Does Kafka python API support stream processing?

I have used Kafka Streams in Java. I could not find similar API in python. Do Apache Kafka support stream processing in python?
user3126637
  • 133
  • 1
  • 1
  • 5
11
votes
0 answers

Python AVRO reader returns AssertionError when decoding kafka messages

Newbie playing with Kafka and AVRO. I am trying to deserialise AVRO messages in Python 3.7.3 using kafka-python, avro-python3 packages and following this answer. The function responsible for decoding the Kafka messages is def…
Mattia Paterna
  • 859
  • 1
  • 11
  • 28
11
votes
1 answer

Kafka optimal retention and deletion policy

I am fairly new to kafka so forgive me if this question is trivial. I have a very simple setup for purposes of timing tests as follows: Machine A -> writes to topic 1 (Broker) -> Machine B reads from topic 1 Machine B -> writes message just read to…
Mohammad Ahmad
  • 123
  • 1
  • 1
  • 8
11
votes
3 answers

How to pass data from Kafka to Spark Streaming?

I am trying to pass data from kafka to spark streaming. This is what I've done till now: Installed both kafka and spark Started zookeeper with default properties config Started kafka server with default properties config Started kafka…
HackCode
  • 1,795
  • 3
  • 29
  • 61
10
votes
6 answers

NoBrokersAvailable: NoBrokersAvailable-Kafka Error

i have already started to learn Kafka. Trying basic operations on it. I have stucked on a point which about the 'Brokers'. My kafka is running but when i want to create a partition. from kafka import TopicPartition (ERROR THERE) consumer =…
Beyhan Gul
  • 1,000
  • 1
  • 12
  • 23
10
votes
5 answers

kafka-python - How do I commit a partition?

Using kafka-python-1.0.2. If I have a topic with 10 partitions, how do I go about committing a particular partition, while looping through the various partitions and messages. I just cant seem find an example of this anywhere, in the docs or…
Johnny Gasyna
  • 399
  • 1
  • 2
  • 12
9
votes
2 answers

Python: how to mock a kafka topic for unit tests?

We have a message scheduler that generates a hash-key from the message attributes before placing it on a Kafka topic queue with the key. This is done for de-duplication purposes. However, I am not sure how I could possibly test this deduplication…
user1658296
  • 1,160
  • 1
  • 14
  • 37
9
votes
6 answers

kafka-python consumer not receiving messages

I am having trouble with KafaConsumer to make it read from the beginning, or from any other explicit offset. Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning option and it hangs…
Karthik Raj
  • 161
  • 1
  • 2
  • 4
9
votes
5 answers

Kafka Consumer: How to start consuming from the last message in Python

I am using Kafka 0.8.1 and Kafka python-0.9.0. In my setup, I have 2 kafka brokers setup. When I run my kafka consumer, I can see it retrieving messages from the queue and keeping track of offsets for both the brokers. Everything works great! My…
user1797335
  • 91
  • 1
  • 1
  • 3
8
votes
2 answers

kafka-python consumer start reading from offset (automatically)

I'm trying to build an application with kafka-python where a consumer reads data from a range of topics. It is extremely important that the consumer never reads the same message twice, but also never misses a message. Everything seems to be working…
Steven Van Dorpe
  • 518
  • 1
  • 5
  • 14
1
2 3
…
22 23