Trying to diagnose some bizarre Tomcat (7.0.21
) and/or JVM errors on a 64-bit linux (CentOS) machine.
I'm load testing our server application and tried hitting it with 100K messages. Launched jvisualvm
and kept my eye on the heap the whole time. Everything was looking great* (see below) until I got to about 93K processed messages and then Tomcat just died. Ran a ps
on Tomcat's PID number to confirm it was dead.
Up until this crash:
- Load test had been running for about 90 minutes; should have finished shortly thereafter since we were at 93K/100K)
- CPU was holding strong around 45%
- Used heap was around 2GB (plus or minus a bunch after GCs) but heap size grew from 4GB to
MAX_HEAP
after about 30 minutes - Class loading/unloading was cycling normally
- Thread dumps were normal
Nowhere in the server code are any calls to System.exit()
- so we can rule that right out (and yes I've double-checked!!!).
I'm not sure if this is Tomcat crashing or the JVM (how do I tell?). And even if I did know, I can't seem to find any indication of what went wrong:
- All of the server app's logs just stop without any ERROR messages (even though we have logging universally set to DEBUG and higher)
- Tomcat's
catalina.out
and respectlocalhost_access_*
files just stop without any info
I've heard it is possible to have Tomcat log a coredump when it does but not sure how to do that and online examples aren't helping much.
How would SO go about diagnosing this? What steps should I take to start ruling out all of the possible factors?
Thanks in advance!