We are using heavy multi threading in a Swing application or extensive calculations. From time to time it can happen that the application runs against an OOME and can not create any native threads any more. I absolutely understand that the application has to be aware of this and it is bad by design then, however it can not be avoided 100%. The problem is that in such a case the JVM is absolutely lost because it can not handle the error and the system is behaving non predictable. Usually we log every memory error and restart the application by -XX:OnOutOfMemoryError="kill -9 %p", however this does not work for obvious reason. On the other hand it is a bit frustrating the JVM has no control any more. So what might be a good way to come around this kind of problem?
PS: I do not search for a solution like extending systems process limits or reducing thread stack size via Xss. I am looking for an approach how to handle in general.