2

name of this topic is probably incorrect, but I haven't idea how to name this issue. Something about background, I am programming one game, which 3d surface is divided to chunks. I thought out a saving mechanism, in which all chunk objects with their properties are saved in compressed form to unordered map, which is then serialized to file, so parts of world can be loaded and saved effectively regarding current needs. Of course, when loading, file is loaded, deserialized to unordered map and strings are converted to chunks objects in real time. That is a plann, but with hard problems with realization.

I tried all possible searches, but without result. during my play with tests, I wrote a small test script like this:

#include <iostream>
#include <fstream>
#include <sstream>

int main()
    {
    std::ifstream reader("output.dat", std::ios::binary);
    std::string data;
    reader>>data;
    reader.close();
    std::cout<<data.size()<<std::endl;

    std::stringstream ss;
    ss.str(data);

    unsigned char id_prefix=0, zone_prefix=1;
    while (ss.peek()!=EOF)
        {
        unsigned char type;
        ss>>type;
        if (type==id_prefix)
            {
            unsigned char tempx, tempy, tempz;
            unsigned short tempid;

            if (!(ss>>tempx)) std::cout<<"reading of x failed."<<std::endl;
            if (!(ss>>tempy)) std::cout<<"Reading of y failed"<<std::endl;
            if (!(ss>>tempz)) std::cout<<"Reading of z failed."<<std::endl;
            if (!(ss>>tempid)) std::cout<<"Reading of id failed, position is "+std::to_string(ss.tellg())+", values are "+std::to_string(type)+" "+std::to_string(tempx)+" "+std::to_string(tempy)+" "+std::to_string(tempz)<<std::endl;

            std::cout<<(int)tempx<<" "<<(int)tempy<<" "<<(int)tempz<<" "<<(int)tempid<<std::endl;
            }
        else if (type==zone_prefix)
            {
            unsigned char tempx, tempy, tempz;
            unsigned int tempzone;
            ss>>tempx;
            ss>>tempy;
            ss>>tempz;
            ss>>tempzone;
            std::cout<<(int)tempx<<" "<<(int)tempy<<" "<<(int)tempz<<" "<<(int)tempzone<<std::endl;
            }
        }

    }

Output.dat is a file with one experimental decompressed chunk to reproduce parsing process in the game. You can download it from: https://www.dropbox.com/s/mljsb0t6gvfedc5/output.dat?dl=1 if you want, it have about 160 kb in size. And here is a first problem. It is probably only my stupidity, but I thought that when I use std::ios::binary to open ifstream, and then extract its content to string, it will load whole file, but it loaded only first 46 bytes. That is first problem, next in the game, I used other system to load data which worked, but then stringstream processing as can be seen in lower part of code failed too around this position. I guess there are problems also with data, as you can see, format is uchar type (indicates whether following bytes refer to id or zone), coordinates (each as uchar), and ushort in case of id, uint in case of zone. But when I looked into the file with my own created binary editor, it showed id as one byte only, not two as I expected from short value. Saving was done also with stringstream, in form: unsigned short tempid=3; //example value ss<

and in result file this was represented as a 51 (in one byte), what is ascii code for 3, so I am little confused, or little more than little.

Can you please help me with this? I am using Mingw g++ 4.9.3 on win7 64-bit.

Thanks much!

Edit from 1.1.2017

Now whole file is read in stringstream, but extraction of values still fails. When >> extraction reads to the next whitespace, how is it with extraction to unsigned short for example? I was playing with code bit, trying to change for example unsigned short tempid to unsigned char tempid. And output does not make sense to me. In short version, bytes like: 0;1;0;0;51 were read as type 0, x 1, y 0, z 0 and id 3 what is correct, even I don't understand why 51 is here instead of a 3. Writing to the stream before seemed as:

unsigned short idtowrite=3;
ss<<idtowrite;

But when I changed unsigned short tempid to unsigned char tempid, it read it as type 0, x 1, y 0, z 0 and id 51, what is not correct, but I expect it from writed file. I wouldn't solve it if it read correctly through full stream, but for some reason until 0;8;0;0;51 all is correct, and from 0;9;0;0;51, which is next to it fails, with x readed as 0, y as 0 and z as 51 and EOF is set. I am thinking if reading haven't missed a byte, but I don't see a reason to do it. Can you please recommend me some effective and working way how to store values in stringstream?

Thanks in advance!

  • 4
    `reader>>data` reads up to the first occurrence of a whitespace character. Try this: `std::stringstream ss; ss << reader.rdbuf();` - this should slurp the whole contents of the file into the string stream. – Igor Tandetnik Dec 30 '17 at 21:44
  • what is with the "reader.close();" immediately after you create the sreader fstream? – SoronelHaetir Dec 30 '17 at 21:44
  • Try this: https://stackoverflow.com/questions/17584784/read-a-binary-file-jpg-to-a-string-using-c – Avishai Y Dec 30 '17 at 21:48
  • All your reading assumes that the content is text, but it isn't. – molbdnilo Dec 30 '17 at 23:13
  • Oh, thanks! Rdbuf variant worked perfectly. Now however I have problems with second part of the code, reading from stringstream. I had problems adding extension to this post, and only after creating newone question I found out why. Sorry, this is my first question on stackoverflow. You can find new question on: https://stackoverflow.com/questions/48053097/saving-and-reading-values-from-stringstream, thanks for understanding. – Rastislav Kish Jan 01 '18 at 20:54
  • Okay, so I was announced that spreading to the two questions is not the best idea, so finally putting it here. – Rastislav Kish Jan 01 '18 at 21:07
  • I don't see what you want to accomplish with the `stringstream` instead of processing the file directly. In any way, using formatted input (like operator `>>`) for processing binary data is wrong. Only use `istream::read()` method. It will require casting though, e. g. `reader.read(reinterpret_cast(&tempid), sizeof(tempid));`. – zett42 Jan 01 '18 at 21:38

1 Answers1

2

std::ios::binary only has the effect of suppressing end-of-line conversion (so that e.g. \r\n in file is not converted to just \n in memory). It is certainly correct to supply this when dealing with binary files.

However, >> is still a formatted input function, which skips leading whitespace, terminates at whitespace and so on.

If you want to actually read the file as binary data, you must use the read function on the stream object.

Angew is no longer proud of SO
  • 156,801
  • 13
  • 318
  • 412