There's a Java application that occasionally starts to utilise all available cores, starves GC and crashes with OOM. The application is quite complex. It uses Akka Streams, Kafka, Etcd and has a built-in HTTP server for metrics reporting. I've added the -XX:+CrashOnOutOfMemoryError
to create core dumps, but they aren't helpful, they show a thread that couldn't allocate memory but what is needed is what threads are actually eating all the CPU. Any guides or ideas on what could be be done to find out what's going on?
The application is running on OpenJDK 11 on Linux with 3 cores assigned via cgroups. Heap size is set to 3Gb and initial heap size is 1.5Gb, it uses G1 without any tweaks.
UPD: At the time of crash heap_lock
is held by neither the active thread nor by a GC thread and last 10 GC events essentially failed to free any memory.