C++ Populating deque bad allocation

Question

I'm trying to make a huge collection of random numbers to do algorithm analysis. Then I came across this problem that I can't explain. Consider the following code:

#include <exception>
#include <iostream>
#include <deque>

typedef unsigned long long mytype;

const mytype SIZE = 150000000;

int main()
{
    std::deque<mytype> rand;

    try 
    {
        for (mytype i = 0; i< SIZE; i++)
        {
            rand.push_back(1); //Just push a dummy number into the deque
        }

    }
    catch (std::exception& e)
    {
        {
           std::cout << e.what() << std::endl;
        }
    }

    return 0;
}

This will end up with a bad allocation exception. The thing is if I use vector and reserve() it will work. Correct me if I'm not getting this right, isn't deque the kind of data structure that have larger capacity, since its not allocating memory continuously like vector?

I'm running this on Win8 x64, visual studio 2012, intel i7 with 8G RAM. Thanks for sharing your thoughts

I just tried your example with Linux + gcc. Runs fine without problem. Maybe you have some `ulimit` memory restrictions on Windows. — Olaf Dietsche, Feb 21 '13 at 09:08
Thanks for the quick responses, so is there a way to set this `ulimit` I'll probably try it on linux also — hezzze, Feb 21 '13 at 09:11
Call [std::deque::max_size()](http://msdn.microsoft.com/en-US/library/17y0taey%28v=vs.110%29.aspx) to check. — Jesse Good, Feb 21 '13 at 09:13

score 2 · Answer 1 · edited May 23 '17 at 11:56

The problem is caused by memory fragmentation .

If use vector.reserve directly, you get a big block of memory at the start. The memory fragmentation is low at the start of the application, so the chance of allocating a huge memory is high at that time.

As you use deque.push_back in the loop, the deque will first reserve a small block of memory(such as the size is A bytes), as the loop runs, the deque reaches to its memory limit, then it will alloc 2*A bytes, then copy the data to the new memory; then 4*A bytes, then 8*A bytes....I read the code of deque long time ago, so the there's maybe something wrong with details, the idea here is that this loop will cause a lot of memory allocation/deallocation, which will lead to high memory fragmentation, if the memory fragmentation is high, the chance of allocating a huge memory is low.

For details of memory fragmentation, please refer this.

Thanks for the clarification and reference, I will look into this, and try to figure it out. I've already done the analysis using vector. Learning new stuff is always good! — hezzze, Feb 22 '13 at 01:19

C++ Populating deque bad allocation

1 Answers1