Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

122 questions
32
votes
1 answer

Amazon Managed Streaming for Kafka- MSK features and performance

I'm evaluating AWS Managed Service Kafka (MSK) and I know that currently, it is in preview mode, hence might not have all features or proper documentation. I tried setting up the msk cluster and was validating whether msk could fulfill all the use…
britney
  • 431
  • 1
  • 4
  • 4
8
votes
1 answer

Amazon Kinesis vs AWS Manage Service Kafka (MSK) - (Connect from on-prem)

I'm evaluating AWS Kinesis vs Managed Service Kafka (MSK). Our requirement is sending some messages (JSON) to AWS to from the on-prem system (system develop using c++). Then we need to persist above messages into the relational database like…
HASH
  • 83
  • 1
  • 4
8
votes
1 answer

AWS MSK - Timeout when creating Kafka topic with ACL turned-on

I'm using AWS MSK and I want to enable ACLs but I'm unable to create a topic when ACLs are turned-on. I'm using the command-line tools for all the operations. Here's a summary of what I'm doing: Create a fresh cluster Create a topic - this works…
Araf
  • 303
  • 4
  • 10
7
votes
2 answers

How to set auto.create.topics.enable as default config on AWS MSK cluster

Our AWS MSK cluster is not automatically creating topics when a producer first publishes a message to the topic. This is surprising because the default behaviour according to https://kafka.apache.org/documentation/#brokerconfigs should have…
rpassza
  • 186
  • 1
  • 7
5
votes
2 answers

kafka-python raise UnrecognizedBrokerVersion Error

I am getting this error when constructing KafkaProducer with the kafka-python package: [ERROR] UnrecognizedBrokerVersion: UnrecognizedBrokerVersion Traceback (most recent call last): File "/var/lang/lib/python3.7/imp.py", line 234, in load_module …
Rene B.
  • 3,421
  • 3
  • 28
  • 49
4
votes
1 answer

Failing to connect to Kafka on MSK

I have created a Kafka cluster on MSK and now I'm trying to connect to the cluster with python. I wrote this shortcode: from kafka import KafkaProducer import json producer = KafkaProducer( bootstrap_servers=['host1:9092', 'host2:9092'], …
Sean Goldfarb
  • 191
  • 11
4
votes
1 answer

How to get brokers endpoint of Amazon MSK as an output

we have an AWS cloudformation template, through which we are creating the Amazon MSK(Kafka) cluster. which is working fine. Now we have multiple applications in our product stack which consume the Brokers endpoints which is created by the Amazon…
chitender kumar
  • 302
  • 1
  • 13
3
votes
3 answers

AWS MSK User/Password Authentication/Authorization

i want to be able to authenticate/authorize clients to produce/consume messages on certain topics. they would be part of our vpn (incl. aws). as i understand the available documentation the only option to do this is to issue client certificates and…
niesfisch
  • 83
  • 6
2
votes
0 answers

AWS MSK with IAM - Timeout Exception

We are using Aws MSK, recently they announce the AWS MSK IAM AUTH(https://github.com/aws/aws-msk-iam-auth) When i tried to use it with the bellow configuration on my clients i got the exception: org.apache.kafka.common.errors.TimeoutException:…
Lucas Gazire
  • 273
  • 3
  • 15
2
votes
1 answer

AWS Lambda with MSK topic as event source not able to read Kafka messages

I have created an AWS MSK cluster and have tested the set up by running a EC2 Producer/Consumer instance in the same VPC.I have used the same VPC and subnets to create an AWS lambda with a MSK cluster topic as an event source.All my subnets are…
Sam
  • 285
  • 2
  • 7
  • 20
2
votes
1 answer

How AWS MSK and Confluent Schema Registry and Confluent Kafka connect recommended to use together?

We are planning to use AWS MSK service for Managed Kafka and Schema Registry and Kafka Connect services from Confluent together to run our connectors (Elasticsearch Sink Connector). We have planned to run Schema Registry and Connectors in EC2. As…
Suresh Sanghani
  • 31
  • 4
2
votes
1 answer

Can I consume an AWS MSK kafka topic from a Lambda function?

I thought it would be trivial to use a lambda to consume messages coming to a topic in a AWS MSK Kafka cluster, but I could not find a way to do it from the AWS docs. Is it possible in some way?
Carlo Pires
  • 3,788
  • 5
  • 29
  • 31
2
votes
2 answers

Is there a way to dump Amazon MSK Topic to S3 directly?

I have planned to used Amazon MSK and i want to dump consumer logs to S3 . But i don't see any options. Do i need to write my own consumer or is there a way to consume Amazon MSK consumer output to s3 directly ?
Navin Kumar
  • 150
  • 1
  • 9
2
votes
1 answer

Connection problem with msk using spring kafka

So here on the aws instance we have tried to connect spring kafka with msk where it works with local kafka. next there is no ssl . I tried telnet its running. But while running the application this exception comes. Our cluster of msk is in same…
Tanmay Naik
  • 490
  • 4
  • 14
2
votes
2 answers

AWS Kafka (MSK) - How to generate Keystore and truststore and use the same in my Spring Cloud Stream application?

Is there any info as to how can I use the AWS MSK details in my Spring Cloud Stream application ? I believe we need to generate a keystore and truststore and then incorporate the same in our application ? I went through the "Client Authentication"…
Kenny Weeler
  • 63
  • 2
  • 12
1
2 3
…
8 9