0

The only two requirements I'm limited to is that it must be OS independent and that the data is not saved as binary, I need to save it as text. I've heard of memory mapping, but I'm told I can only use it on windows. Currently, my program's speed is being mainly held back by how fast I can load a file, so any help would be appreciated. So far, this is the best solution I've come to.

std::ifstream file(filepath);
fileContents.assign(std::istreambuf_iterator<char>(file), std::istreambuf_iterator<char>());
Joseph Grimaldo
  • 31
  • 2
  • 1
    All major OSes offer memory mapping. – ypnos Apr 25 '20 at 16:49
  • Memory mapping is a binary only deal. It may be worth loading the file in binary mode and converting after the file is loaded. This may be of use https://stackoverflow.com/questions/51352863/what-is-the-idiomatic-c17-standard-approach-to-reading-binary-files – Galik Apr 25 '20 at 17:22
  • i have no problem reading from my m.2 ssd at full speed with an ifstream when i [read](https://en.cppreference.com/w/cpp/io/basic_istream/read) in larger blocks, reserve my buffers, .. – Thomas Apr 25 '20 at 18:54
  • Yea, I don't either. As I said, the problem is I'm looking for something faster. – Joseph Grimaldo Apr 25 '20 at 19:12
  • how do you expect the data rate to be faster than what the device can deliver? for that you need compression. – Thomas Apr 25 '20 at 20:53
  • Well I mean there are other techniques to loading data than the piece of code I wrote...some and going to be faster, and some are going to be slower. This whole question was posed to find faster ways. With the answers and comments from other people, I've been able to find one that's faster, but I'm still looking for more. – Joseph Grimaldo Apr 26 '20 at 03:43
  • Do you actually know the performance capabilities of the disk system you're reading from? Also, why does the data need to be in text format? If the data were binary then you could at least organize it for fast access to bytes of interest at know positions. Regarding memory mapped files you could look at [`boost.interprocess`](https://www.boost.org/doc/libs/1_72_0/doc/html/interprocess/sharedmemorybetweenprocesses.html#interprocess.sharedmemorybetweenprocesses.mapped_file). – G.M. Apr 26 '20 at 09:37

1 Answers1

1

Speed and OS-agnostism are pretty much mutually exclusive. As the comment says, all major OSes offer memory mapping but the way they offer it will vary. Windows has CreateFileMapping while posix systems have mmap.

It would not surprise me if boost has a component library that glosses over these differences but I could not tell you which library that would be.

Of course there is also the whole "premature optimization is the root of all evil" thing to consider, are you sure that what the standard library offers is not "good enough"?

SoronelHaetir
  • 11,346
  • 1
  • 8
  • 18
  • If the piece of code I wrote is the best the standard library offers for extracting files, I'm 100% positive I need it to be faster. The file extraction currently takes up about a third of cpu time. The rest is mainly spent on vector and map allocations. – Joseph Grimaldo Apr 25 '20 at 17:26