4

I have created a Kafka cluster on MSK and now I'm trying to connect to the cluster with python.

I wrote this shortcode:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['host1:9092', 'host2:9092'],
    value_serializer=lambda x: json.dumps(x).encode('utf-8'),
    api_version=(2, 4, 1)
)

producer.send('test', value={'hello':'world'})

The problem is that every time I running it I'm getting this error:

KafkaTimeoutError: Failed to update metadata after 60.0 secs.

I thought it might be related to Kafka creating topics so I added this line to the configuration.

auto.create.topics.enable=true

But I'm still getting the same error.

This is my full configuration file:

default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
socket.request.max.bytes=104857600
unclean.leader.election.enable=true
auto.create.topics.enable=true
zookeeper.connection.timeout.ms=5000

What am I missing here? I read somewhere that is may relate to SSL authentication but at any step, there wasn't any .pem file, .ca file, or anything like that.

Sean Goldfarb
  • 191
  • 11
  • Can you share your configuration? (interested to see `advertised.listeners`) – Giorgos Myrianthous Jul 08 '20 at 14:33
  • I added the configuration to the question body. But I thought the whole point of AWS MSK is that it is managed and I don't have to deal with this kind of configuration. – Sean Goldfarb Jul 08 '20 at 14:47
  • Also, have you restarted your cluster after adding `auto.create.topics.enable=true` ? – Giorgos Myrianthous Jul 08 '20 at 14:50
  • I'm actually very new to MSK but I believe it was restarted on its own because the updating process took about 5 to 10 minutes – Sean Goldfarb Jul 08 '20 at 14:54
  • Whenever I have a connection timeout in AWS, my first thought is security groups. Does the MSK cluster's security group allow ingress on tcp port 9092 from your application's security group? – PMah Jul 08 '20 at 14:56
  • The port was indeed close... but even after allowing it on the security group, the result remains the same. – Sean Goldfarb Jul 08 '20 at 15:27
  • Once you created configuration properties have you updated them with the cluster. I was facing the same issue but once it worked after I set ```auto.create.topics.enable=true``` and update the cluster with new properties. – SangamAngre Jan 30 '21 at 19:12
  • Things to try: (i) `bootstrap_servers = 'host1:9092'` (ii) remove `api_version=(2, 4, 1)`; (iii) `producer.flush()` after `producer.send(...` - One per time. Combinations perhaps. – Paulo Marques Mar 05 '21 at 23:30
  • Could you please confirm whether your producer application lives in Ec2? – Jaya Ananthram Mar 06 '21 at 13:26
  • @SeanGoldfarb where are you trying to connect to MSK cluster from? are you on the same network (VPC) as your MSK? – Chen A. Mar 07 '21 at 09:57

1 Answers1

2

You probably fail to connect to the MSK. The error you experience is a Timeout error.

One wrong assumption when first starting to use MSK is, you can connect to it from outside AWS network. That is a wrong assumption. AWS has a detailed document on how can you access your MSK cluster.

If you're not running your client from an EC2 instance inside your MSK VPC, you won't be able to reach it. Even if you open the relevant security rules in the MSK security group policy.

I have spend time trying multiple proxies to access the MSK from outside AWS, with no success. Just follow the guide I referred above, and it will probably fix your connection issue.

Also, If you're new to MSK I highly suggest you to go through the Getting Started tutorial, or at least Steps 5 (Create a Topic) and 6 (Produce and Consume Data).

Chen A.
  • 7,798
  • 2
  • 26
  • 48
  • Have you ever tried port forwarding from the EC2 MSK VPC instance to your local machine? I haven't worked with MSK before but have had this issue where I've had to port forward from an instance in non-public facing VPC to a jump box to my local machine. Clearly following the documentation is the better alternative. – VanBantam Mar 10 '21 at 21:38
  • 1
    My EC2 instance wasn't accessible from my local machine, had to go through bastion host. It was too much to toll to forward ports on multiple machines just for connecting MSK, so I haven't tried – Chen A. Mar 11 '21 at 07:25