3

Our Java Application running with WildFly 8.2.1 and Java 1.8_92 hangs completely on huge load. A threaddump at this situation shows that a lot of threads are in state WAITING at monitor 0x00000005cc562228:

"default task-100" #825 prio=5 os_prio=0 tid=0x00000000033a2800 nid=0x49bd in Object.wait() [0x00007f238cb98000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1465)
    at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
    - locked <0x00000005cc562228> (a com.mchange.v2.resourcepool.BasicResourcePool)
    at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
    .......

How can we find the owner of this monitor lock because we assume this thread to be the reason for some connection leak? We assumed this monitor lock to appear in another context but this is not the case.

Or may be there is any other hint for the deadlock situation? Any help is very appreciated because we are struggling quite a long time with this problem.

user6345319
  • 31
  • 2
  • The reason it hangs is not that those threads wait for the lock to become available, they wait for something to call notify(All) & that something likely owns the resources from the pool (or forgot to notify). You'd see something like "waiting for monitor entry" (e.g. in http://stackoverflow.com/a/11343043 ) if the lock was blocked. – zapl May 17 '16 at 12:51

1 Answers1

2

Actually this is the thread (default task-100) that owns the lock corresponding to 0x00000005cc562228 as you can see thanks to - locked <0x00000005cc562228> in the call stack of your thread.

If you use a tool like the JConsole, in the Threads tab you have the ability to detect deadlocks thanks to the button "Detect Deadlock".

However in your case, it doesn't seem to be a deadlock as the owner of the lock obviously waits for the availability of an Object from an object pool. I guess that it is a connection pool so you should increase the max size of your connection pool to avoid such kind of issue.

Nicolas Filotto
  • 39,066
  • 11
  • 82
  • 105
  • Before you increase the pool size, you should however check (and test) if the resource on the other side of the connections (and your application) can cope with the increased number of connections. Or fix the root cause, which can also be a deadlock on the other side (e.g. if it''s a database, a deadlock of rows etc.). – dunni May 17 '16 at 16:38
  • Increasing the size of the connection pool did only delay the time when the application hangs. A closer look at the threaddump shows over thirty threads waiting at the same lock object, the stack trace always the same, only differing in the thread id. As only one single thread can be the owner of a lock monitor I assume it is a typo and should be "waiting to lock" instead "locked" (see http://stackoverflow.com/questions/20382091/thread-dump-blocked-and-locked). But the very strange thing is that this special lock does not appear in another context in the threaddump. – user6345319 May 18 '16 at 09:42
  • @user6345319 if so you probably have a connection leak in your code then – Nicolas Filotto May 18 '16 at 11:10