OutOfMemoryError even though enough free memory

Question

I am getting java.lang.OutOfMemoryError errors, even when I still have enough free RAM. The memory dumps I took were between 200MB and 1GB, while my server has 24GB of RAM. I set -Xmx12288m -Xms12288m.

Also, when I try to log in to the server, I frequently get

-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

I narrowed it down to the code snippet below:

import org.snmp4j.Snmp;
import org.snmp4j.transport.DefaultUdpTransportMapping;

    long n = 0;
    while (true) {
        DefaultUdpTransportMapping transport = null;
        try {
            transport = new DefaultUdpTransportMapping();
            transport.listen();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
//      } finally {             // (*) I forgot this
//          transport.close();  // (*) I forgot this
        }

        n++;
        double freeMemMB = Runtime.getRuntime().freeMemory() / 1024 / 1024;
        System.out.println("Created " + n
                + " DefaultUdpTransportMappings. Free Mem (mb): "
                + freeMemMB);
    }

Output (on my developer machine, with mvn exec:java):

Created 2026 DefaultUdpTransportMappings. Free Mem (mb): 299.0
Created 2027 DefaultUdpTransportMappings. Free Mem (mb): 299.0
[WARNING] 
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at org.snmp4j.util.DefaultThreadFactory$WorkerThread.run(DefaultThreadFactory.java:91)
    at org.snmp4j.transport.DefaultUdpTransportMapping.listen(DefaultUdpTransportMapping.java:168)
    at App.main(App.java:19)
    ... 6 more

I found that I get the errors because I don't close the DefaultUdpTransportMapping. Enabling the finally { ... } block solves the problem. Now I'm wondering which limits (if not the amount of free memory) I reached. The ulimits on the server are:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 191968
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

On my developer Mac:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                2560

Which limit did I reach?

Locally, I have `-Xmx1024m`. When I set it to `-Xmx2048m`, I still get an OutOfMemoryError after 2027 DefaultUdpTransportMappings are started. For the production process on the server, I set `-Xmx12288m -Xms12288m`. Memory dump of the production process is 58MB. — Benedikt Köppel, Nov 19 '14 at 11:09
run again and attach jconsole (comes with the jdk) to the running process; if you add the visual gc plugin to it then it will show you the sizes and amount of allocations in each heap region. — Chris K, Nov 19 '14 at 11:10
possible duplicate of [Java: Unable to create new native thread](http://stackoverflow.com/questions/5253072/java-unable-to-create-new-native-thread) — Erich Kitzmueller, Nov 19 '14 at 11:20
I couldn't find out how to add the VisualGC in the jconsole, but I found it in `jvisualvm`. This is how it looks like: http://i.imgur.com/OvIko5w.png. — Benedikt Köppel, Nov 19 '14 at 11:26
ammoQ: The link mentiones that I hit the limit for open file descriptors. I tried printing the number of open FDs with the code from here: http://stackoverflow.com/a/16361505/1067124. I get the `OutOfMemoryError` when I have 2098 open FDs. ulimit for open FDs is 2560. — Benedikt Köppel, Nov 19 '14 at 11:30
@BenediktKöppel you should also read the linked blog entry http://blog.egilh.com/2006/06/2811aspx.html — Erich Kitzmueller, Nov 19 '14 at 11:57

score 0 · Answer 1 · answered Mar 19 '15 at 11:31

The java.lang.OutOfMemoryError: unable to create new native thread is a confusing message, since it has not really to do with running out of heap memory. Therefore the heap size settings (Xmx and Xms) have no influence on this case. The exception is thrown when a new operating process cannot not be created for your application, either because a maximum number of processess/open file handles is reached or there is no memory left on the system the create a new thread.

Regarding the ulimit settings, it can be either the number of file descriptors, the stack size or the number of processes. The stack size is the a per thread number, the number of threads times the stack size will be the amount of used memory.

Usually, as it your case, getting this exception means your application is not closing its threads properly and keeps hold of system processes. This is why closing the trasnport fixed the issue for you.

OutOfMemoryError even though enough free memory

1 Answers1