Is there a way rather than using getline() for each row in a csv, to instead read in a larger chunk, say 10000 rows into a string? The idea would be then to write code which splits the string into substrings/put elements into desired array/vectors.
Currently loading csvs (50-1500mb) is taking 5 mins ++, from trawling related questions it seems the bottleneck is calling getline() / the system calls are what is causing the slowness?
I'm a c++ newb so if anyone knows a better solution that would be appreciated!
This is my current slow code if it helps:
while (!myFile.eof())
{
string aLine; //holds in read in line
getline(myFile, aLine); //reads line from file into aLine
std::string input = aLine;
std::istringstream ss(input);
std::string token;
while (std::getline(ss, token, ',')) {
t++;
if (t == 2) {
y.push_back(0);
y[i] = atof(token.c_str());
cout << y[i] << endl;
}
}
t = 0;
i++;
}
EDIT: Thanks John Zwinck, the time has decreased from 232.444 seconds to 156.248. Also thanks Richard Critten, I will update the time elapsed using memory maps with boost.