I have created a Kafka cluster on MSK and now I'm trying to connect to the cluster with python.
I wrote this shortcode:
from kafka import KafkaProducer
import json
producer = KafkaProducer(
bootstrap_servers=['host1:9092', 'host2:9092'],
value_serializer=lambda x: json.dumps(x).encode('utf-8'),
api_version=(2, 4, 1)
)
producer.send('test', value={'hello':'world'})
The problem is that every time I running it I'm getting this error:
KafkaTimeoutError: Failed to update metadata after 60.0 secs.
I thought it might be related to Kafka creating topics so I added this line to the configuration.
auto.create.topics.enable=true
But I'm still getting the same error.
This is my full configuration file:
default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
socket.request.max.bytes=104857600
unclean.leader.election.enable=true
auto.create.topics.enable=true
zookeeper.connection.timeout.ms=5000
What am I missing here? I read somewhere that is may relate to SSL authentication but at any step, there wasn't any .pem file, .ca file, or anything like that.