Appending to an ObjectOutputStream

Question

Is it not possible to append to an ObjectOutputStream?

I am trying to append to a list of objects. Following snippet is a function that is called whenever a job is finished.

FileOutputStream fos = new FileOutputStream
           (preferences.getAppDataLocation() + "history" , true);
ObjectOutputStream out = new ObjectOutputStream(fos);

out.writeObject( new Stuff(stuff) );
out.close();

But when I try to read it I only get the first in the file. Then I get java.io.StreamCorruptedException.

To read I am using

FileInputStream fis = new FileInputStream
        ( preferences.getAppDataLocation() + "history");
ObjectInputStream in = new ObjectInputStream(fis);    

try{
    while(true)
        history.add((Stuff) in.readObject());
}catch( Exception e ) { 
    System.out.println( e.toString() );
}

I do not know how many objects will be present so I am reading while there are no exceptions. From what Google says this is not possible. I was wondering if anyone knows a way?

Do you get anything at all from the stream, or does it throw an exception the first time around the loop? — skaffman, Jul 28 '09 at 14:53
In the code above, I am only seeing one object written to the file, so only one would be read, right? — aperkins, Jul 28 '09 at 15:03
first snippet is in a function it is called multiple times whenever a job is complete. but i can only read the first one i wrote. — Hamza Yerlikaya, Jul 28 '09 at 15:05
Your code here only writes 1 object to the stream, is there anything not shown that writes more than 1 object to that stream ? — nos, Jul 28 '09 at 15:06

score 81 · Accepted Answer · edited May 23 '17 at 12:10

81

Here's the trick: subclass ObjectOutputStream and override the writeStreamHeader method:

public class AppendingObjectOutputStream extends ObjectOutputStream {

  public AppendingObjectOutputStream(OutputStream out) throws IOException {
    super(out);
  }

  @Override
  protected void writeStreamHeader() throws IOException {
    // do not write a header, but reset:
    // this line added after another question
    // showed a problem with the original
    reset();
  }

}

To use it, just check whether the history file exists or not and instantiate either this appendable stream (in case the file exists = we append = we don't want a header) or the original stream (in case the file does not exist = we need a header).

Edit

I wasn't happy with the first naming of the class. This one's better: it describes the 'what it's for' rather then the 'how it's done'

Edit

Changed the name once more, to clarify, that this stream is only for appending to an existing file. It can't be used to create a new file with object data.

Edit

Added a call to reset() after this question showed that the original version that just overrode writeStreamHeader to be a no-op could under some conditions create a stream that couldn't be read.

edited May 23 '17 at 12:10

Community

1
1

answered Jul 28 '09 at 15:58

Andreas Dolk

108,221
16
168
253

Clever! I *think* the stream header is the only problem, in which case this should work beautifully, but have you tested it to be sure? – Michael Myers Jul 28 '09 at 16:05
Thanks for you comments :-) Yes, I have posted tested code (just forgot to mention it in the answer) – Andreas Dolk Jul 28 '09 at 18:26
+1, thanks. *Warning:* writeStreamHeader() is called by the ObjectOutputStream *constructor*. This makes it at least difficult for a single class to handle both cases. You can make a simple factory method that returns fileExists ? new AppendingObjectOutputStream( out ) : new ObjectOutputStream( out ); – Andy Thomas Sep 15 '11 at 15:58
Have you really tried that? Not only writing but reading the stream afterwards as well? It doesn't work correctly because there's other internal state which isn't properly restored if you just append to the OOS. – jrudolph Aug 02 '12 at 14:29
Yes, of course I've tested it in 2009 with Sun's Java. Guess it was JDK5. Don't know, if they've changed the internals since then. – Andreas Dolk Aug 02 '12 at 14:31
4

Everyone please note, that this is *only for* appending to an *existing* file! A *new* file has to be created with the normal ObjectOutputStream because we need the header at the beginning of the file. – Andreas Dolk Aug 02 '12 at 14:37
3

apparently there **is** a problem with this solution, if you can fiddle with it enough. Look at this question: http://stackoverflow.com/questions/12279245/classcastexception-when-appending-object-outputstream/12438141#12438141 – sasidhar Sep 16 '12 at 08:15

score 14 · Answer 2 · edited Nov 24 '09 at 06:29

14

As the API says, the ObjectOutputStream constructor writes the serialization stream header to the underlying stream. And this header is expected to be only once, in the beginning of the file. So calling

new ObjectOutputStream(fos);

multiple times on the FileOutputStream that refers to the same file will write the header multiple times and corrupt the file.

edited Nov 24 '09 at 06:29

abyx

61,118
16
86
113

answered Jul 28 '09 at 15:18

Tadeusz Kopec

11,984
6
51
79

score 7 · Answer 3 · answered Jul 28 '09 at 15:13

Because of the precise format of the serialized file, appending will indeed corrupt it. You have to write all objects to the file as part of the same stream, or else it will crash when it reads the stream metadata when it's expecting an object.

You could read the Serialization Specification for more details, or (easier) read this thread where Roedy Green says basically what I just said.

score 7 · Answer 4 · answered Jul 28 '09 at 15:30

The easiest way to avoid this problem is to keep the OutputStream open when you write the data, instead of closing it after each object. Calling reset() might be advisable to avoid a memory leak.

The alternative would be to read the file as a series of consecutive ObjectInputStreams as well. But this requires you to keep count how many bytes you read (this can be implementd with a FilterInputStream), then close the InputStream, open it again, skip that many bytes and only then wrap it in an ObjectInputStream().

score 3 · Answer 5 · answered Aug 07 '19 at 15:07

I have extended the accepted solution to create a class that can be used for both appending and creating new file.

import java.io.DataOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.OutputStream;

public class AppendableObjectOutputStream extends ObjectOutputStream {

    private boolean append;
    private boolean initialized;
    private DataOutputStream dout;

    protected AppendableObjectOutputStream(boolean append) throws IOException, SecurityException {
        super();
        this.append = append;
        this.initialized = true;
    }

    public AppendableObjectOutputStream(OutputStream out, boolean append) throws IOException {
        super(out);
        this.append = append;
        this.initialized = true;
        this.dout = new DataOutputStream(out);
        this.writeStreamHeader();
    }

    @Override
    protected void writeStreamHeader() throws IOException {
        if (!this.initialized || this.append) return;
        if (dout != null) {
            dout.writeShort(STREAM_MAGIC);
            dout.writeShort(STREAM_VERSION);
        }
    }

}

This class can be used as a direct extended replacement for ObjectOutputStream. We can use the class as follows:

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;

public class ObjectWriter {

    public static void main(String[] args) {

        File file = new File("file.dat");
        boolean append = file.exists(); // if file exists then append, otherwise create new

        try (
            FileOutputStream fout = new FileOutputStream(file, append);
            AppendableObjectOutputStream oout = new AppendableObjectOutputStream(fout, append);
        ) {
            oout.writeObject(...); // replace "..." with serializable object to be written
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

}

score 0 · Answer 6 · answered Nov 09 '14 at 22:12

0

How about before each time you append an object, read and copying all the current data in the file and then overwrite all together to file.

answered Nov 09 '14 at 22:12

user4147874

61
8

Appending to an ObjectOutputStream

6 Answers6

Linked

Related