Read large CSV file in C++(~4GB)

Question

I want to read and store a large CSV file into a map. I started by just reading the file and seeing how long it takes to process. This is my loop:

while(!gFile.eof()){
   gFile >> data;
}

It is taking me ~35 mins to process the csv file that contains 35 million lines and six columns. Is there any way to speed this up? Pretty new to SO, so apologies if not asking correctly.

Answer 1

Background
Files are stream devices or concepts. The most efficient usage of reading a file is to keep the data streaming (flowing). For every transaction there is an overhead. The larger the data transfer, the less impact the overhead has. So, the goal is to keep the data flowing.

Memory faster than file access
Search memory is many times faster than searching a file. So, searching for a "word" or delimiter is going to be faster than reading a file character by character to find the delimiter.

Method 1: Line by line
Using std::getline is much faster than using operator>>. Although the input code may read a block of data; you are only performing one transaction to read a record versus one transaction per column. Remember, keep data flowing and searching memory for the columns is faster.

Method 2: Block reading
In the spirit of keeping the stream flowing, read a block of memory into a buffer (large buffer). Process the data from the buffer. This is more efficient than reading line by line because you can read in multiple lines of data with one transaction, reducing the overhead of a transaction.

One caveat is that you may have a record cross buffer boundaries, so you'll need to come up with an algorithm to handle that. The execution penalty is small and only happens once per transaction (consider this part of the overhead of a transaction).

Method 3: Multiple threads
In the spirit of keeping the data streaming, you could create multiple threads. One thread is in charge or reading the data into a buffer while another thread processes the data from the buffer. This technique will have better luck keeping the data flowing.

Method 4: Double buffering & multiple threads
This takes Method 3 above and adds multiple buffers. The reading thread can fill up one buffer then start filling a second buffer. The data processing thread will wait until the first buffer is filled before processing the data. This technique is used to better match the speed of reading data to the speed of processing the data.

Method 5: Memory mapped files
With a memory mapped file, the operating system handles the reading of the file into memory on demand. Less code that you have to write, but you don't get as much control over when the file is read into memory. This is still faster than reading field by field.

Answer 2

Lets start with the bottlenecks.

1. Reading from disk
2. Decoding the data
3. Store in map
4. Memory speed
5. Amount of memory

Read from disk

• Read till you drop, if you can't read fast enough to use all bandwidth on the disk you can go faster. Ignore all other steps and only read.
• Start by adding buffers to your instream
• Set hints for reading
• use mmap
• 4GB is a trivial size, if you don't already have 32 GB upgrade
• Too slow buy M.2 disk.
• Still to slow then more exotic, change disk driver, dump OS. Mirror disks, only you $£€ is the limit.

Decode the data

• if you data is in lines where are all the same length then all decodes can be done in parallel, limited only by memory bandwidth.
• if the line lengths only wary a little the find end of line can be done in parallel followed by parallel decode.
• if the order of the lines doesn't matter for the final map just split the file in #hardwarethreads parts and let each process their part until the first newline in the next threads part.
• memory bandwidth will most likely be reach far before the CPU is anyway near used up.

Store in map

• hopefully you have thought about this map in advance as none of the std maps are thread safe.
• if you don't care about order a std::array can be used and you can run at full memory bandwidth.
• lets say you want to use std::unordered_map, there is a problem that it needs to update the size after each write, so effectively your are limited to 1 thread writing to it.
• You could use 1 thread at a time to update while the other precompute the hash of the record.
• having one thread write has the problem that nearly every write will be a cache miss severely limiting speed.
• so if that is not fast enough roll your own hash_map, without a size that must be updated every write.
• to ensure thread safety you also need to protect the write, having one mutex makes you as slow or slower than the single writer.
• you could try to make it lock and wait free ... if your not an expert you will get severe headache instead.
• if you have selected a bucket design for you hash then you could make X times number of writer threads mutexes, use the hash value to select the mutex. The extra mutexes increase the likelihood that two threads won't collide.

Memory speed

• Each line will be transferred at least 4 times over the memory bus, once from the disk to ram (at least once more if the driver is not good), once when the data is decoded, once when the map makes a reads request, and one more for when the map writes.
• A good setup can save one more memory access if the driver writes to cache and therefore decode not will result in a LLC-miss.

Amount of memory

• you should have enough memory to hold the total file, the data structure and some intermediate data.
• Check if RAM is cheaper than your programming time.