Kafka Connect running out of heap space

Question

After starting Kafka Connect (connect-standalone), my task fails immediately after starting with:

java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:343)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:291)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:193)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:316)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:222)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

There's a mention of heap space in some Kafka documentation, telling you to try it with "the default" and only modifying it if there are problems, but there are no instructions to modify the heap space.

The "default size" seems to be [determined at run time](http://stackoverflow.com/questions/4667483/how-is-the-default-java-heap-size-determined). It appears to be large enough in my machine (8G). I still get the OOM error. Besides, there are all sorts of other exceptions in the connector log, and the server stalls. Very frustrating. — Kenji Noguchi, Feb 02 '17 at 19:38
From what I found, it's hard-coded to 256M: https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L209 — Robin Daugherty, Feb 03 '17 at 21:48

Robin Daugherty · Accepted Answer · 2019-01-17T15:45:08.817

39

You can control the max and initial heap size by setting the KAFKA_HEAP_OPTS environment variable.

The following example sets a starting size of 512 MB and a maximum size of 1 GB:

KAFKA_HEAP_OPTS="-Xms512m -Xmx1g" connect-standalone connect-worker.properties connect-s3-sink.properties

When running a Kafka command such as connect-standalone, the kafka-run-class script is invoked, which sets a default heap size of 256 MB in the KAFKA_HEAP_OPTS environment variable if it is not already set.

edited Jan 17 '19 at 15:45

answered Jan 31 '17 at 18:15

Robin Daugherty

5,698
1
40
50

2

As of Kafka 1.1.0, the new default for Connect is `-Xmx2g`, and it is set before `kafka-run-class` - https://github.com/apache/kafka/commit/cfa6a78c7c2529839d2f3cb1129098f2e552fd72 – OneCricketeer Jan 16 '19 at 07:28
Any other way to achieve the same? – c0degeas Mar 14 '19 at 14:01
1

The method in my answer is the same, only the default value changed. – Robin Daugherty Mar 18 '19 at 04:56

score 38 · Answer 2 · edited Apr 23 '20 at 13:32

38

When you have Kafka problems with

java.lang.OutOfMemoryError: Java heap space

it doesn't necessarily mean that it's a memory problem. Several Kafka admin tools like kafka-topics.sh will mask the true error with this when trying to connect to an SSL PORT. The true (masked) error is SSL handshake failed !

See this issue: https://issues.apache.org/jira/browse/KAFKA-4090

The solution is to include a properties file in your command (for kafka-topics.sh this would be --command-config) and to absolutely include this line:

security.protocol=SSL

edited Apr 23 '20 at 13:32

Robin Daugherty

5,698
1
40
50

answered Apr 22 '20 at 10:08

peedee

2,179
1
17
41

for my case, it does appear to be a bogus error message, hiding the actual issue – Randy L Aug 12 '20 at 16:33
Wow. This helped! I only had `producer: ssl: key-store-location: and password:` configured in application.yaml. And I was getting out of memory in testing environment. I think somewhere in the error log there was also *Topic not present in metadata after 60000 ms*. Adding `spring.kafka.producer.security.protocol: SSL` helped. Thank you. – Pawel Ryznar Dec 29 '20 at 16:53

score 1 · Answer 3 · edited Jun 07 '18 at 04:25

1

Even I was facing the issue could not start my producer and consumer for a given topic. Also deleted all unnecessary log files and topics.Even though that's not related to the issue.

Changing the kafka-run-class.sh did not work for me. I changed the below files

kafka-console-consumer.sh

kafka-console-producer.sh

and stopped getting OOM error. Both consumer and producer worked fine after this.

Increased the size to KAFKA_HEAP_OPTS="-Xmx1G" was 512m earlier.

edited Jun 07 '18 at 04:25

Stephen Rauch

40,722
30
82
105

answered Jun 07 '18 at 03:58

Sagarmatha

55
7

1

Sorry that my answer wasn't more clear. `KAFKA_HEAP_OPTS` is an environment variable and should be set at the command line or in the service that starts the Kafka job. You should not modify the scripts that are part of the Kafka distribution, as they'll be wiped out when you update Kafka. – Robin Daugherty Jun 08 '18 at 23:52

Kafka Connect running out of heap space

3 Answers3

Linked