fork: retry: Resource temporarily unavailable

Question

I tried installing Intel MPI Benchmark on my computer and I got this error:

fork: retry: Resource temporarily unavailable

Then I received this error again when I ran ls and top command.

What is causing this error?

Configuration of my machine:

Dell precision T7500
Scientific Linux release 6.2 (Carbon)

Check your Linux documentation on how to increase the number of processes. — theglauber, Aug 22 '12 at 18:25
Perhaps this would help: http://stackoverflow.com/questions/344203/maximum-number-of-threads-per-process-in-linux — theglauber, Aug 22 '12 at 18:26

score 81 · Answer 1 · edited Jan 29 '14 at 21:32

81

This is commonly caused by running out of file descriptors.

There is the systems total file descriptor limit, what do you get from the command:

sysctl fs.file-nr

This returns counts of file descriptors:

<in_use> <unused_but_allocated> <maximum>

To find out what a users file descriptor limit is run the commands:

sudo su - <username>
ulimit -Hn

To find out how many file descriptors are in use by a user run the command:

sudo lsof -u <username> 2>/dev/null | wc -l

So now if you are having a system file descriptor limit issue you will need to edit your /etc/sysctl.conf file and add, or modify it it already exists, a line with fs.file-max and set it to a value large enough to deal with the number of file descriptors you need and reboot.

fs.file-max = 204708

edited Jan 29 '14 at 21:32

Mark Lakata

18,024
5
88
112

answered Aug 22 '12 at 18:59

Satish

13,709
26
80
130

15

FYI, you can use "sysctl -p" to apply the current settings in /etc/sysctl.conf saving the reboot. – EmmEff Feb 28 '14 at 15:26
2

Also, if you have read only /etc, you can use 'sysctl -w fs.file-max=204708' at runtime. – Ondrej Galbavý Aug 24 '15 at 10:34
you can also add it directly to kernel without reboot but not permanently 'echo 999999 > /proc/sys/fs/file-max' – matson kepson Aug 24 '18 at 08:35

score 51 · Answer 2 · edited Apr 07 '20 at 08:39

51

Another possibility is too many threads. We just ran into this error message when running a test harness against an app that uses a thread pool. We used

watch -n 5 -d "ps -eL <java_pid> | wc -l"

to watch the ongoing count of Linux native threads running within the given Java process ID. After this hit about 1,000 (for us--YMMV), we started getting the error message you mention.

edited Apr 07 '20 at 08:39

roottraveller

6,328
4
50
59

answered Dec 21 '12 at 00:20

2

When I tested, `ps -eL` worked for showing all processes and `ps -L ` worked for showing processes regarding the ``. `ps -eL ` just shows all processes regardless of the ``. – Sanghyun Lee Jul 01 '16 at 09:51
1

@Willie Wheeler What did you do to overcome the thread limit? I have been searching and testing numerous solutions found online, to no avail, for DAYS. Nothing will allow more than 1k threads, except for a restart of systemd-logind, which lasts for about an hour before the errors start again! – Brandon Elliott Jul 27 '16 at 04:00
Machines have finite resources. If you are hitting a limit, use fewer threads (e.g. controlled with thread pool) or else more machines. – Sep 21 '16 at 15:41
I restarted my rails server – thedanotto Nov 30 '16 at 00:48
2

1000 threads isn't a lot though – frankster Jun 06 '17 at 10:49

fork: retry: Resource temporarily unavailable

2 Answers2

Linked