Manage memory usage during streams usage

Question

I have a program which has a wrapper on the OutputStream and inside the wrapper I a list of streams. Take a look code below

private static final class SomeWrapper extends OutputStream {

        private final List<OutputStream> dest;

        @Override
        public void write(int b) throws IOException {
            for (OutputStream destination : dest) {
                destination.write(b);
            }
        }

        @Override
        public void write(byte[] bytes, int off, int len) throws IOException {
            for (OutputStream destination : dest) {
                destination.write(bytes, off, len);
            }
        }

        @Override
        @SneakyThrows
        public void close() {
            super.close();
            for (OutputStream destination : dest) {
                destination.close();
            }
            dest.clear();
        }
    }

So, if wrapper contains a lot of OutputStreams - application crashes with OOM. What pattern or approach would be better here, to control how much streams I can use based on the system resources(for example Xmx could be 512Mb and 2Gb). dest - is a list of BufferedOutputStream. So inside I have a buffer

score 0 · Answer 1 · answered Aug 17 '19 at 14:54

I don't see why that would use that much memory unless all output streams have a huge buffer or you have a crazy amount of outputs. How many outputs are you using and why?

One way to keep memory under control is to make several passes of the input stream if you can. It will obviously come at the expense of performance. Here's a suggestion:

if the input stream is slow to get (e.g. from network), consider writing it to disk to start with
determine how much memory your JVM has
explicitly set the buffer size for your output streams so that you control how much memory they use
determine how many buffers can fit in the available memory and instantiate about 80% of that number to leave some headroom.
do a first pass where you write to the first N outputs. Read from the file you wrote to disk in the first step to fasten the process.
iterate until you're done
don't forget to cleanup the cached input file

Note that if there is only one batch you can skip the first step altogether and get the input stream once from wherever it comes. Also, this solution assumes you have enough disk space to store the input, if not, you can disregard the caching part.

Manage memory usage during streams usage

1 Answers1