java.lang.OutOfMemoryError: unable to create new native thread executing Selenium WebDriver based tests using Selenium and Java

Question

I am running more than 2000s selenium tests which are migrated from Selenium RC to WebDriver using selenium-leg-rc.jar (a kind of bridge library) which allow to compile/execute RC based test code using WebDriver.

These tests are executed in 4 parallel processes by splitting into 4 groups. Approx 50% tests run fine and after that I see firefox and gecodriver processes are not killing consistently while I am invoking driver.quit() at the end of every tests. After this, I randomly get SocketTimeoutException/OutOfMemoryError errors as below call stack.

[junit] Exception in thread "ForkJoinPool.commonPool-worker-0" java.lang.OutOfMemoryError: unable to create new native thread
[junit]     at java.lang.Thread.start0(Native Method)
[junit]     at java.lang.Thread.start(Thread.java:714)
[junit]     at java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1338)
[junit]     at java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1460)
[junit]     at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
         . . .
         . . .
         . . .

[junit] org.openqa.selenium.WebDriverException: java.net.SocketTimeoutException: timeout
[junit] Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:48'
[junit] System info: host: 'mylinuxbox', ip: '101.24.15.172', os.name: 'Linux', os.arch: 'amd64', os.version: '3.8.13-118.36.1.el6uek.x86_64', java.version: '1.8.0_20-ea'
[junit] Driver info: driver.version: RemoteWebDriver
[junit]     at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:92)

If I run these tests in a single process all works fine. Any suggestion apart from using Runner classes to run multiple threads instead of processes or doing clean WebDriver approach. Similar problem with Windows 10 also.

Selenium 3.141.59 geckodriver 0.25 Firefox 60.8.0b8

Answer 1

java.lang.OutOfMemoryError

java.lang.OutOfMemoryError: unable to create new native thread occurs when an application is no more able to create new threads. This error can surface because of following two reasons:

• There is no room in the memory to accommodate new threads.
• The number of threads exceeds the Operating System limit.

There are a couple of diverse solutions depending on which event is triggering this error and either one or a combination of the below-mentioned solutions can resolve the problem:

• Fix Thread Creation Rate: You need to analyze if the application has started to create more threads.
• Increase the Thread Limits Set at Operating System: The Operating System has limits for the number of threads that can be created. The limit can be found by issuing ulimit –u command. In some hosts/servers, this value set to a low value such as 1024. It means totally only 1024 threads can be created in this machine. So if your application is creating more than 1024 threads it is going to run into this issue.
• Allocate More Memory to the Machine: If you don't see a high number of threads created and ulimit –u value is well ahead then it indicates that your application has grown organically and needs more memory to create threads. In such circumstances you need to allocate more memory to the machine.
• Reduce Heap Space: Threads are not created within the JVM heap. They are created outside the JVM heap. So if there is less room left in the RAM, after the JVM heap allocation, application will run into this issue.
• Reduce Number of Processes: Number of Processes must be minimal so that kernel processes, other user processes and threads can run uninterupted.
• Reduce Thread Stack Size (-Xss): Java system property –Xss can be configured to set the thread’s memory size. Using this property you can throttle down the memory size. As an example, if you configure -Xss256k, your threads will only consume 125 MB of space (i.e. 500 threads X 256 KB). So by lowering –Xss size also, you might be able to eliminate this error.

---

This usecase

To start with invoking driver.quit() should have killed all the gecodriver and firefox processes gracefully.

You can find a detailed discussion in Selenium : How to stop geckodriver process impacting PC memory, without calling driver.quit()?

A bit of more details about you Test Architecture and Test Setup along with your code trials would have helped us to analyze the issue in a better way. However, from the error stack trace:

Exception in thread "ForkJoinPool.commonPool-worker-0" java.lang.OutOfMemoryError: unable to create new native thread
.
System info: host: 'mylinuxbox', ip: '101.24.15.172', os.name: 'Linux', os.arch: 'amd64', os.version: '3.8.13-118.36.1.el6uek.x86_64', java.version: '1.8.0_20-ea'

It is pretty much evident that your JDK version is 1.8.0_20-ea which is pretty ancient.

So there is a clear mismatch between the JDK v8u20-ea , Selenium Client v3.141.59

---

Solution

Ensure that:

• JDK is upgraded to current levels JDK 8u222.
• Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
• (WindowsOS only) Use CCleaner tool to wipe off all the OS chores before and after the execution of your Test Suite.
• (LinuxOS only) Free Up and Release the Unused/Cached Memory in Ubuntu/Linux Mint before and after the execution of your Test Suite.
• If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
• Take a System Reboot.
• Execute your @Test as non-root user.
• Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.