Custom memory management for creating an object per line in a file

Question

This is my situation:

I am reading in a very large file, say over 1GB and for every line I interpret and extract parts of the line to create an object. Therefore, I am continuously allocating memory:

//For each line in a 1GB file:
boost::shared_ptr<MyClass>(new MyClass(part_of_line[0], part_of_line[1]));

therefore I'd like to write some sort of custom memory manager. I haven't done this before and I am not too sure how to approach it, but basically I'd like to group together all the allocations and decrease the time spent allocating memory AND it would be better if the memory allocated was continuous.

Are there any pre-existing libraries already written (boost?) for such a problem domain which would help me realise performance gains?
Alternatively under what situation would it be better for me to write my own custom memory manager?

I presume the answer is tied with how general a library pool is written and how different my situation is to the general case??

The first step would be using `make_shared`, instead of creating the `shared_ptr` directly with the `new`. That will reduce the number of memory allocations in half, with a single line change. You won't get more buck for the money anywhere else ;) — David Rodríguez - dribeas, Nov 27 '13 at 21:04
When are you deallocating these objects? How many allocations (including internal to the constructor, temporary or not) does creating such an object entail? — Yakk - Adam Nevraumont, Nov 27 '13 at 21:07
@Yakk each object contains 3x doubles, 1x int, a 2x std::strings — user997112, Nov 27 '13 at 22:05

score 0 · Answer 1 · edited May 23 '17 at 12:12

0

Read the file into memory as fast as possible create the objects from the memory mapped file. You can experiment with zero memory copy, and creating the objects in treads and deliver the results in a queue.

Fastest way to read a file read input files, fastest way possible? I would use boost::interprocess.

edited May 23 '17 at 12:12

Community

1
1

answered Nov 27 '13 at 20:58

Damian

3,719
3
31
63

I'm already using boost interprocess- but I am still allocating memory when I create the MyClass objects? – user997112 Nov 27 '13 at 21:00
Yes, creating a lot of small objects is in general a bad idea if the objects are small. How big is each object? – Damian Nov 27 '13 at 21:02
Tiny (about 30 bytes)- thats why i'm thinking of using a pool – user997112 Nov 27 '13 at 22:04
You can use just Google C++ containers http://google-opensource.blogspot.in/2013/01/c-containers-that-save-memory-and-time.html – Damian Nov 28 '13 at 10:43

score 0 · Answer 2 · answered Nov 27 '13 at 22:18

0

You could look at the file size before reading and use this to estimate how much space you will need. Then use vector::reserve to allocate a contiguous block all in one step.

answered Nov 27 '13 at 22:18

ScottMcP-MVP

10,036
2
13
15

score 0 · Answer 3 · answered Nov 27 '13 at 22:25

0

The biggest performance gain will be to avoid allocating each object. Instead allocate an array of MyClass objects. Of course one significant side effect is your current memory management strategy would have to change.

answered Nov 27 '13 at 22:25

Dwayne Towell

6,895
3
32
46

Custom memory management for creating an object per line in a file

3 Answers3