System memory exception on for loop

Question

I have the following code below:

List<long> numbers = new List<long>();

for (long i = 1; i <= 300000000; i++)
{
    numbers.Add(i);
}

What I wanted to do is to populate the list from 1-300 million. But when it hit the 67108865, it throws an exception on line 4: Exception of type 'System.OutOfMemoryException' was thrown.

I tried using ulong but still no luck.

I believe the maximum range for long data type is 9,223,372,036,854,775,807 but why am I having an error here?

Thanks in advance!

EDIT Thanks for all the answers. It helped my realized that my design is not good. I ended up changing my code design.

If this is a 32-bit app, you probably hit the limit for your available address space — Roger Rowland, Dec 12 '13 at 06:13
Thanks but I still got an error but now on the 134217729. I used the 64-bit settings. — Gerald, Dec 12 '13 at 06:16
Gerald what are you doing that you require that many object in memory at once? You may need to use a different design strategy, like partitioning the data in to smaller chunks or using a database instead of storing the entire list in memory. — Scott Chamberlain, Dec 12 '13 at 06:49
@ScottChamberlain I am solving one of the problems in the Project Euler. I agree with you because I tried partitioning the list, using x64-bit app but my computer gave up because my RAM is not sufficient (4GB only). — Gerald, Dec 12 '13 at 07:06
The point of Project Euler is to give you problems that are hard to solve directly and usually require critical thinking because the obvious brute force method wont work (do to thinks like memory constraints, for your problem). If it takes more than 30 seconds for your computer to come up with a answer once you enter the last input there is likely a better way to solve the problem. — Scott Chamberlain, Dec 12 '13 at 07:31

score 2 · Accepted Answer · edited May 23 '17 at 12:19

2

Well, it is not that your number are to large, but your list is...
Lets calculate it's size:

300,000,000 * 64 bits (size of long) = 19,200,000,000 bits
19,200,000,000 /8 (size of byte) = 2,400,000,000
2,400,000,000 / 2^10 = 2,343,750 KB
2,343,750 / 2^10 = 2,288~ MB
2,288/ 2^10 = 2.235~ GB

You wanted a list of about 2.35 GB. The current CLR limitation is 2 GB (see this or this SO thread)

If you need a list with size of 300,000,000, split it into 2 lists (either in place or using a wrapping object that will handle managing the lists).

edited May 23 '17 at 12:19

Community

1
1

answered Dec 12 '13 at 06:28

Avi Turner

9,342
7
43
68

I ended up splitting it into 3 lists but my machine ended up not responding because of too much memory being consumed up. Thanks for the answer btw! – Gerald Dec 12 '13 at 07:09

dbw · Answer 2 · 2013-12-12T14:27:16.620

First note System.OutOfMemoryException is not thrown when limit is reached for a variable,
Secondly, it is thrown because there is not enough memory available to continue the execution of a program.
Sadly you can not configure, .the NET runtime makes all the decisions about heap size and memory.
Either switch to 64-bit machine.

For info, on 32 bit machine you can increase the memory by using /3GB boot switch option in Boot.ini

EDIT While searching I found in MSDN Documentation under Remarks section

By default, the maximum size of an Array is 2 gigabytes (GB). In a 64-bit environment, you can avoid the size restriction by setting the enabled attribute of the gcAllowVeryLargeObjects configuration element to true in the run-time environment. However, the array will still be limited to a total of 4 billion elements, and to a maximum index of 0X7FEFFFFF in any given dimension (0X7FFFFFC7 for byte arrays and arrays of single-byte structures).

A List<long> is backed by an long[]. You will fail as soon as the backing array cannot be allocated, during the reallocation, there has to be enough total memory for both the old and the new arrays.

But, if you want a collection with more than 2³¹ elements, you have to write your own implementation by using multiple arrays or List, and managing them.

Is there any way to configure my machine to allocate enough memory on my application to run? — Gerald, Dec 12 '13 at 06:17
@Gerald There is no way to do so because it is a CLR limitation, not a "application setting" — Avi Turner, Dec 12 '13 at 06:46

score 0 · Answer 3 · answered Dec 12 '13 at 06:13

0

Your numbers are not too large; your list is too long. It is using all the available memory.

Reduce 300000000 to 3000000, and it will (probably) work fine.

answered Dec 12 '13 at 06:13

Paul Draper

64,883
37
172
246

Yes, it will work fine with `3000000` but what I need is `300000000` as part of the requirements. – Gerald Dec 12 '13 at 06:17

score 0 · Answer 4 · answered Dec 12 '13 at 06:24

It should ideally hold number of elements that you are telling.

But, as per current CLR implementation each object can have max 2 GB size. As you are storing long values (size 8 bytes), then while populating the List, it is somewhere trying to exceed the 2GB limit. That's why you are getting System.OutOfMemoryException

System memory exception on for loop

4 Answers4