Retrieve messages from a topic on a message hub

Question

I am trying to get messages from a topic on a message hub on bluemix using Confluent Kafka Python. My code is found below, but something is not working. The topic and the message hub is up and running, so there is probably something with the code.

from confluent_kafka import Producer, KafkaError, Consumer

consumer_settings = {
'bootstrap.servers': 'broker-url-here',
'group.id': 'mygroup',
'default.topic.config': {'auto.offset.reset': 'smallest'},
'sasl.mechanisms': 'PLAIN',
'security.protocol': 'ssl',
'sasl.username': 'username-here',
'sasl.password': 'password-here',
}

c = Consumer(**consumer_settings)
c.subscribe(['topic-here'])

running = True

while running:
    msg = c.poll()
    if msg.error():
        print("Error while retrieving message")
        c.close()
        sys.exit(10)
    elif (msg is not None):
        for x in msg:
            print(x)
    else:
        sys.exit(10)

When I run the code, it seems to get stuck at msg = c.poll(). So I guess it is either failing to connect, or failing to retrieve messages. The credentials themselves are correct.

score 3 · Accepted Answer · answered Mar 19 '18 at 21:05

The consume logic look fine but the configuration for the consumer is incorrect.

security.protocol needs to be set to sasl_ssl
ssl.ca.location needs to point to a PEM file containing trusted certificates. The location of that file varies for each OS, but for the most common it's:
- Bluemix/Ubuntu: /etc/ssl/certs
- Red Hat: /etc/pki/tls/cert.pem
- macOS: /etc/ssl/certs.pem

We also have a sample app using this client that can easily be started or deployed to Bluemix: https://github.com/ibm-messaging/message-hub-samples/tree/master/kafka-python-console-sample

Retrieve messages from a topic on a message hub

1 Answers1