For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.
Questions tagged [aws-msk]
122 questions
2
votes
1 answer
Handling kafka clients updates in kubernetes
I have a Kafka cluster running on AWS MSK with Kafka producer and consumer go clients running in kubernetes. The producer is responsible for sending the stream of data to Kafka. I need help solving the following problems:
Let's say, there is some…
Piyush Kumar
- 21
- 1
2
votes
2 answers
Can I write to an AWS MSK Kafka cluster from a Lambda function?
I need to pick up messages from AWS SES and publish them to an AWS MSK (Kafka) topic. Is this possible? My current understanding is that all MSK clients must live in the same VPC. How would I configure a Lambda function that appropriately?
Patrick
- 21
- 1
- 4
1
vote
1 answer
How to monitor disk space usage for Kafka Brokers in AWS MSK cluster
We need to Monitor disk space usage for Kafka Brokers running in AWS MSK cluster.
There're several metrices emitted by Kafka which can be used to monitor various aspects. But I was unable to find any specific metric that monitors "Disk Usage" for…
Himanshu Singh
- 95
- 9
1
vote
0 answers
How to achieve Aws MSK cost optimization
Hi i am deployed Amazon managed Kafka in AWS,
3 Brokers
50 gb size
t3 small instance
I want this MSK running 9 to 5 and scale down after that to save cost. Is there any mechanism to achieve this.
Its costing me 110$ every month, is there a way to…
George Thomas
- 4,178
- 4
- 27
- 58
1
vote
1 answer
MSK Not Deleting Old Messages
I have three MSK clusters; dev, nonprod & prod. They all have the following cluster configuration - there is no topic level…
Garreth
- 757
- 1
- 6
- 16
1
vote
0 answers
Kafka: Understanding Broker failure
I have a Kafka cluster with:
2 brokers b-1 and b-2.
2 topics with both: PartitionCount:1 ReplicationFactor:2 min.insync.replicas=1
Here is what happened:
%6|1613807298.974|FAIL|rdkafka#producer-2|…
max
- 11
- 2
1
vote
1 answer
MSK Kafka: Please explain why there are multiple endpoints listed for ZookeeperConnectString and how I should be using them
If you run the following aws command, you will get msk kafka cluster details:
aws kafka list-clusters
This returns something like this:
{
"ClusterInfoList": [
{
"BrokerNodeGroupInfo": { ...},
"ClusterArn":…
cosbor11
- 9,704
- 8
- 43
- 58
1
vote
1 answer
Missing required configuration "bootstrap.servers" which has no default value while ksql deploying ksqldb-server docker-compose image on fargate
I have managed to deploy ksqldb server on an ec2 instance with msk kafka as kafka server using docker-compose and its working perfactly(able to connect using python KSQL API client and access topics and stream in msk`.
version: '2'
services:
…
iamabhaykmr
- 1,199
- 11
- 33
1
vote
1 answer
Problems with Amazon MSK default configuration and publishing with transactions
Recently we have started doing some testing of our Kafka connectors to MSK, Amazon's managed Kafka service. Publishing records seem to work fine however not when transactions are enabled.
Our cluster consists of 2 brokers (because we have 2 zones)…
Gray
- 108,756
- 21
- 270
- 333
1
vote
1 answer
How to apply back pressure with AWS Lambda and MSK (Kafka) event stream
Is it possible to do this?
When dealing with SQS event streams, if the Lambda function does not have adequate reserved concurrency, the function will be throttled, and the unprocessed events / messages can be retried via the SQS redrive policy. I've…
InTooDeep
- 473
- 6
- 17
1
vote
1 answer
Does kafka salesforce-bulk-source-connector work with Amazon MSK?
happy new am experimenting with Kafka for the first time, i set up an AWS MSK Kafka cluster successfully, next step in my experiment is the using kafka-connect and in particular i want to use the salesforce-bulk-source connector, according to the…
austiine
- 328
- 1
- 3
- 11
1
vote
0 answers
Confluent RabbitMQ Source Connector - configuration, license related error?
our Kafka setup consists of brokers on AWS MSK, Confluent Kafka Connect (confluentinc/cp-kafka-connect:5.5.1) on AWS EKS pod.
We are trying to use Confluent RabbitMQ Source Connector (trial version of commercial connector)…
Yash Chaudhari
- 11
- 2
1
vote
2 answers
org.apache.kafka.clients.NetworkClient Bootstrap broker (id: -3 rack: null) disconnected when connecting to kafka consumer via NiFi
I am trying to consume messages from a kafka topic on Amazon MSK (Kafka Version 2.2.1) using NiFi ConsumeKafka_2_6 processor (NiFi version 1.12.1).In ConsumeKafka processor I am getting the warning as "org.apache.kafka.clients.NetworkClient…
Tris
- 63
- 4
1
vote
0 answers
How to aggregate different log groups generated on Cloudwatch?
I have created MSK cluster for event streaming and my producer and consumer lambda functions are producing different log groups on cloudwatch. I want to aggregate all the logs of producer and consumer lambda functions for debugging purpose in order…
stg
- 57
- 4
1
vote
1 answer
How do you use a nodejs Lambda in AWS as a producer to send messages to MSK topic without creating EC2 client server?
I am trying to create a Lambda in AWS that serves as a producer to an MSK topic. All the AWS docs say to create a new EC2 instance, but as my Lambda is in the same VPC I feel like this should work. I am very new to this and I notice my log statement…
drumurr
- 23
- 3