Large Input text file with two numbers each line, how to take input faster?

Question

I have a text file with format as below-

1 2

3 4

5 6

So on till 5 mil lines.I am taking the input using this code-

ifstream fin;
fin.open("abc.txt");
while(!fin.eof()){
    fin>>vert>>adj;
    cout<<vert << "  "<<adj;

}

The program is taking around 15 min to process the input. Is there any way to speed up the processing.

How large is the file, how many resources do you have (does it fit in RAM?) and why do you `cout` all the stuff? Your program might run in a fraction of the time if you skipped the console output. — nvoigt, Oct 28 '16 at 11:38
The console output will massivley slow down your program. `cout` is pretty expensive. — Nidhoegger, Oct 28 '16 at 11:40
Yes, file is 60mb and I have 8gb ram. I cout to see that program is executing correctly. — Nimit Aggarwal, Oct 28 '16 at 11:41
Incidentally, it isn't. See [Why is iostream::eof inside a loop condition considered wrong?](http://stackoverflow.com/questions/5605125/why-is-iostreameof-inside-a-loop-condition-considered-wrong) — Quentin, Oct 28 '16 at 12:01

score 1 · Answer 1 · edited May 23 '17 at 10:34

You're most likely using a debug build or a very bad implementation of the standard library. Note:

Writing to a file is usually faster than writing to cout.
Using std::endl forces a file flush, and is thus very slow. Don't do that, instead output a newline '\n'.
The while(! fin.eof()) is wrong. Never do that.

Here are my results:

Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)

t1=3.00009s
t2=10.9166s
t3=18.1806s

Test case:

// https://github.com/KubaO/stackoverflown/tree/master/questions/ios-timing-40304260
#include <fstream>
#include <iostream>
#include <chrono>
using namespace std;

int main() {
   const int N = 5000000;
   const char kFile1[] = "tmp1.txt", kFile2[] = "tmp2.txt";

   auto start = chrono::system_clock::now();
   {
      ofstream fOut;
      fOut.open(kFile1);
      for (int i = 0; i < N; ++i)
         // !! DO NOT use endl here!
         fOut << i-N << ' ' << N-i << '\n';
   }
   auto t1 = chrono::system_clock::now();
   cerr << "t1=" << chrono::duration<double>(t1-start).count() << "s" << endl;

   double vert, adj;

   {
      ifstream fIn;
      ofstream fOut;
      fIn.open(kFile1);
      fOut.open(kFile2);
      while (fIn >> vert && fIn >> adj)
         // !! DO NOT use endl here!
         fOut << vert << ' ' << adj << '\n';
   }
   auto t2 = chrono::system_clock::now();
   cerr << "t2=" << chrono::duration<double>(t2-t1).count() << "s" << endl;

   {
      ifstream fIn;
      fIn.open(kFile1);
      while (fIn >> vert && fIn >> adj)
         // !! DO NOT use endl here!
         cout << vert << ' ' << adj << '\n';
   }
   auto t3 = chrono::system_clock::now();
   cerr << "t3=" << chrono::duration<double>(t3-t2).count() << "s" << endl;
}

Large Input text file with two numbers each line, how to take input faster?

1 Answers1