QDataStream and Flush

Question

This is a QT question about using QDataStream and QTemporaryFile in C++ and Linux.

I am having some issues getting a QDataStream to flush. QTextStream has a flush function, however QDataStream apparently doesn't need one. (Citation from 2013: http://www.qtcentre.org/threads/53042-QDataStream-and-flush()). My question is, is this actually/still the case, and is there anyway to force a QDataStream to flush?

When I process the file that I have written using QDataStream the last number of writes are missing (112 Bytes when writing 5 bytes at a time, 22 bytes when writing 1 Byte at a time). However, if I write a large quantity of padding to the end of the file then all of the contents is present (except for the last few writes of the padding). This is why I believe the QDataStream is not being flushed to the file.

The files I am processing are medium size raw binary files (Around 2MB).

Here a minimal example using some of the code I have for processing the files:

void read_and_process_file(QString &filename) {
  QFile inputFile(filename);
  if (!inputFile.open(QIODevice::ReadOnly)) {
    qDebug() << "Couldn't open: " << filename;
    return;
  }
  QDataStream fstream(&inputFile);
  QTemporaryFile *tempfile = new QTemporaryFile();
  if (!tempfile->open()) {
    qDebug() << "Couldn't open tempfile";
    return;
  }
  QDataStream ostream(tempfile);

  while (!fstream.atEnd()) {
    int block_size = 5;      //The number to read at a time
    char lines[block_size];

    //Read from the input file
    int len = fstream.readRawData(lines,block_size);
    QByteArray data(lines,len);

    //Will process data here once copying works

    //Write to the temporary file
    ostream.writeRawData(data,data.size());
  }
  process_file(tempfile);
  delete tempfile;
}

Try to call `tempfile->close()` before calling `process_file(tempfile)`. — hank, Aug 08 '17 at 15:40
Hank, that seems to remove the issue, thank you. Is there not a possibility that the tempfile gets cleaned up if it is closed? I will run some tests and check this. — Weir_Doe, Aug 08 '17 at 16:03
No, `QTemporaryFile::close` does not delete the file. The file is deleted when the `QTemporaryFile` object is destroyed. Also I would suggest not to allocate object inside a single scope using `new` because that may cause memory leaks. In your example you'll get a leak after `Couldn't open tempfile`. Just allocate your object on stack: `QTemporaryFile tempfile;` — hank, Aug 09 '17 at 07:44

R Sahu · Accepted Answer · 2017-08-08T15:55:20.523

The first part of this answer is independent of the issue of flushing the file to disk.

Use of !fstream.atEnd() as the conditional of while is not a good idea. See http://stackoverflow.com/questions/5605125/why-is-iostreameof-inside-a-loop-condition-considered-wrong. I would change the while loop to:

const int block_size = 5;      //The number to read at a time
char lines[block_size];
int len = 0;
while ( (len = fstream.readRawData(lines,block_size)) > 0) {

    QByteArray data(lines, len);

    //Will process data here once copying works

    //Write to the temporary file
    ostream.writeRawData(data,data.size());
}

However, I don't see the point of using the intermediate QByteArray. That loop can be simplified to:

while ( (len = fstream.readRawData(lines,block_size)) > 0) {
    //Write to the temporary file
    ostream.writeRawData(lines, len);
}

If you need to process the QByteArray for other things, it's fine to construct one and use it but the call to ostream.writeRawData doesn't need to use it.

Re. the issue of the file not getting flushed, I would suggest using a nested scope to open the file. The file should get flushed and closed at the end of the scope.

void read_and_process_file(QString &filename) {

   QFile inputFile(filename);
   if (!inputFile.open(QIODevice::ReadOnly)) {
      qDebug() << "Couldn't open: " << filename;
      return;
   }

   QDataStream fstream(&inputFile);
   QTemporaryFile *tempfile = new QTemporaryFile();
   if (!tempfile->open()) {
      qDebug() << "Couldn't open tempfile";
      return;
   }

   // Create a nested scope for the QDataStream
   // object so it gets flushed and closed when the 
   // scope ends.
   {
      QDataStream ostream(tempfile);

      const int block_size = 5;      //The number to read at a time
      char lines[block_size];
      int len = 0;
      while ( (len = fstream.readRawData(lines,block_size)) > 0) {

         QByteArray data(lines, len);

         //Will process data here once copying works

         //Write to the temporary file
         ostream.writeRawData(lines, len);
      }

      // The QDataStream should be flushed and
      // closed at the end of this scope.
   }

   process_file(tempfile);
   delete tempfile;
}

Thanks, good suggestion. I changed the loop to end only when len <= 0 as your original comment suggested. And you are right the QByteArray isn't needed for a minimal example. Sadly while this is useful it doesn't resolve the issue. — Weir_Doe, Aug 08 '17 at 15:46

QDataStream and Flush

1 Answers1