5

Whilst building an wrapper for an console application I came across this weird issue where the Input Stream connected to the output (stdout) of the external process is completely blank until the external process exits.

My code as below:

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;

public class Example{

    public static void main(String args[]) throws IOException{
        File executable = ...

        ProcessBuilder pb = new ProcessBuilder(executable.getCanonicalPath());

        pb.redirectErrorStream(true);

        Process p = pb.start();

        BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream(), Charset.forName("UTF-8")));

        String line;

        while((line = br.readLine()) != null){
            System.out.println(line);
        }
    }
}

I've tried several variants of reading from the input stream and all resulted in the same behavior.

I've tried:

CharBuffer charBuf = CharBuffer.allocate(1000);

InputStreamReader isr = new InputStreamReader(p.getInputStream(), Charset.forName("UTF-8"));

while(isr.read(charBuf) != -1){
    System.out.print(charBuf.flip().toString());
}

and

byte[] buf = new byte[1000];
int r;

while((r = p.getInputStream().read(buf)) != -1){
    System.out.print(new String(buf, 0, r));
}

all to no avail.

Somewhere along the line the output from the external process is being buffered (indefinitely) and I can't really figure out where. Loading the process from the command line seems to work fine where I see output coming out instantaneously. The strangest part is where the fact that the termination of the external process results in a flood of all the "buffered" output at once (a lot of stuff).

Unfortunately I don't have access to the source of the external process but given that it writes to stdout fine when in a console shouldn't really make a difference there (as far as I know).

Any ideas are welcome.

Edit:

One answer recommended me to rewrite the reader for the output and error streams to run on a separate thread. My actual implementation is doing that! And yet the problem still exists. The code posted above is a SSCCE of my actual code condensed for readability purposes, the actual code involves a separate thread for reading from the InputStream.

Edit 2:

User FoggyDay seems to have provided the answer which defines how the behavior of output buffering change when outputting between console and non-consoles. Whilst processes which detect that they are writing to a console use line buffering (buffered flushed every new line), writing to non-consoles (everything that it detects to not be a console) may be fully buffered (to a size of something like 8K). If I make the external process spam (8K of lorem ipsum in a for loop) output does indeed appear. I guess my question now is how to make my java program trigger line buffering on the external process.

initramfs
  • 7,625
  • 2
  • 32
  • 56
  • 1
    Why use a `BufferedReader` if you don't want buffering? I might switch to a `java.util.Scanner` instead. – Elliott Frisch Jan 08 '14 at 18:27
  • @ElliottFrisch Initially I wanted some form of buffering but given that didn't quite work out I started to use `CharBuffer` and sorts to try troubleshoot the issue. my final attempt was reading the raw InputStream directly and still, no output is visible. Given that Scanner builds on top of the raw InputStream I don't see how it'll work but I'll give it a try anyway. – initramfs Jan 08 '14 at 18:31
  • I assume you have already tried your code with other executables and saw that its output is available before the process ends? Any chance this executable is doing something funky like checking if the output is redirected? – Miserable Variable Jan 08 '14 at 18:50
  • @MiserableVariable Yes I have, it works with most processes. But you see if the executable was checking for output redirecting and somehow blocking it, then why the spam of output as the application is shutting down? I understand it may very well be something that the application is doing, what I want to know is what. What exactly _could_ the application be doing to cause this behavior. Isn't stdio meant to be a very simple infrastructure? Why all these issues... – initramfs Jan 08 '14 at 18:54
  • The why question is of course very difficult to answer. It could for example, be because the app does not want to be redirected :) It may be useful to try an alternate implementation, if you are on unixish OS then piping through at `| while read line; do echo `time`: $line; done` might be a quick and dirty test that shows how often it produces output. – Miserable Variable Jan 08 '14 at 19:48
  • This article is about Perl ... but it has several useful insights into the why's and wherefore's of what you're experiencing. Please take a glance at it: http://perl.plover.com/FAQs/Buffering.html – FoggyDay Jan 08 '14 at 19:48
  • @MiserableVariable Output is produced at a fast and constant rate, the process happens to write to a log file at the same time and the log is timed-stamped. During initial startup, it may write up to 50 lines in 1 second. – initramfs Jan 08 '14 at 19:57
  • @FoggyDay The article states that stdio is line buffered if attached to a terminal. So I'm assuming my program isn't be counted as a terminal. And it seems that the article is correct, if I make the external process spam 8K of lorem ipsum, output does indeed appear... Now I just need a way to make my java program appear as a console which essentially is an unanswered [question](https://stackoverflow.com/questions/14145481/creating-a-gui-application-that-emulates-as-consolejava) I posted nearly exactly a year ago... How ironic. – initramfs Jan 08 '14 at 20:11
  • I am stumped. I wonder if there is something in the process output that trips `Process` class. I would try different jvms, another test is to capture the output in a file and your program with a different executable that produces the exact same characters, including end of line and other special characters. – Miserable Variable Jan 08 '14 at 20:12
  • @MiserableVariable If you read my response to FoggyDay it seems to be some strange and annoying behavior of applications doing massive (8K) buffering on things that are not consoles. I guess they expect something thats not-a-console to be a file... Apparently fast "fast and constant rate of printing" still isn't good enough for 8K. – initramfs Jan 08 '14 at 20:14

2 Answers2

1

To your question "how to make my java program trigger line buffering on the external process":

On Linux you can use the "stdbuf" program (coreutils package): stdbuf -oL your_program program_args

You only need to change stdout since stderr is unbuffered by default. The man page of setlinebuf gives additional background information if you're interested: http://linux.die.net/man/3/setlinebuf

user3669782
  • 697
  • 7
  • 6
  • My application is meant to be cross-platform so a linux-specific solution is not good enough. I'll have a look at the source for stdbuf though to see if there is anything I can do with it. – initramfs May 24 '14 at 03:36
  • Tried this in windows (git comes with stdbuf.exe) and it does not seem to work, I guess JDK is buffereing? – teknopaul Feb 13 '18 at 11:38
-1

Some software checks if it is writing to a terminal and switches behavior to unbuffered output. This could be a reason, why it works in the terminal. Pipe the output to cat and see if the output still appears immediately.

Another reason could be that the program is waiting for input or a close of its stdin before it does something, although this does not really match the symptoms described so far.

Harald
  • 3,632
  • 1
  • 26
  • 55