Console redirection not working until application closed (Java)

Question

First off, I only do this for fun and I'm not a pro by any means. So I wouldn't be surprised if my code is a bit sloppy!

I'm attempting to write a GUI wrapper in Java 11 for a console application. My plan was to use a BufferedReader to capture stdOut and stdErr from the process and display it in a JTextArea. I'm running this thread from my main GUI thread after populating an ArrayList with the command line parameters. It works perfectly on Ubuntu or Fedora but I just can't get it right on Windows. When I attempt to run a cross-compiled Windows version of the console application, my application only displays its output after the console application has closed. I also tried substituting in a very simple Hello World application in C (which normally displays Hello, waits 5 seconds and then displays World) and this does the same thing. But, if I change my ArrayList to run ping.exe -t 8.8.8.8, this works fine.

What I suspect is happening is that the while loop is blocking the thread but I don't understand how it works on Linux and if I use ping.exe on Windows. I also tried the code in Redirect stdin and stdout in Java and inheritIO mentioned in ProcessBuilder: Forwarding stdout and stderr of started processes without blocking the main thread but am having the same problem with those too. Any ideas?

    public class RunThread extends Thread {
    @Override
    public void run(){
        // Create process with the ArrayList we populated above
        ProcessBuilder pb = new ProcessBuilder(allArgs);
        pb.redirectErrorStream(true);

        // Clear the console
        txtConsoleOutput.setText("");

        // Try to start the process
        try {
            Process p = pb.start();                

            // Get the PID of the process we just started
            pid = p.pid(); 

            // Capture the output
            String cmdOutput;
            BufferedReader inputStream = new BufferedReader(new InputStreamReader(p.getInputStream()));

            // Get stdOut/stdErr of the process and display in the console
            while ((cmdOutput = inputStream.readLine()) != null) {
                txtConsoleOutput.append(cmdOutput + "\n");
            }
            inputStream.close();
        }
        catch (IOException ex) {
            JOptionPane.showMessageDialog(null,
                    "An error (" + ex + ") occurred while attempting to run.", AppName, JOptionPane.ERROR_MESSAGE);
        }

        // Clear the ArrayList so we can run again with a fresh set
        allArgs.clear();

    }
}

Update Based on the code provided by @ControlAltDel and the advice by @Holger, I've rewritten this to be thread safe (hopefully!), but the end result is the same.

        SwingWorker <Void, String> RunTV = new SwingWorker <Void, String> () {
        @Override
        protected Void doInBackground() {
            // Create process with the ArrayList we populated above
            ProcessBuilder pb = new ProcessBuilder(allArgs);
            pb.directory(new File(hacktvDirectory));
            pb.redirectErrorStream(true); 
            // Try to start the process
            try {
                Process p = pb.start();                
                // Get the PID of the process we just started
                pid = p.pid();
                // Capture the output
                DataFetcher df = new DataFetcher(p.getInputStream(), new byte[1024], 0);
                FetcherListener fl = new FetcherListener() {
                    @Override
                    public void fetchedAll(byte[] bytes) {}
                    @Override
                    public void fetchedMore(byte[] bytes, int start, int end) {
                        publish(new String (bytes, start, end-start));
                    }
                };
                df.addFetcherListener(fl);
                new Thread(df).start();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
            return null;
        } // End doInBackground

        // Update the GUI from this method.
        @Override
        protected void done() {
            // Revert button to say Run instead of Stop
            changeStopToRun();
            // Clear the ArrayList so we can run again with a fresh set
            allArgs.clear();                
        }
        // Update the GUI from this method.
        @Override
        protected void process(List<String> chunks) {
            // Here we receive the values from publish() and display
            // them in the console
            for (String o : chunks) {
                txtConsoleOutput.append(o);
                txtConsoleOutput.repaint();
            }
        }
    };
    RunTV.execute();
}

Update 10/11/2020 Following the posts by kriegaex I took another look at this. The sample code did the same thing unfortunately, but their comment "If for example your sample program uses System.out.print() instead of println(), you will never see anything on the console because the output will be buffered." rang a bell with me.

I have access to the source for the program I'm wrapping and it's written in C. It has the following code to print the video resolution to the console:

void vid_info(vid_t *s)
{
        fprintf(stderr, "Video: %dx%d %.2f fps (full frame %dx%d)\n",
                s->active_width, s->conf.active_lines,
                (double) s->conf.frame_rate_num / s->conf.frame_rate_den,
                s->width, s->conf.lines
        );

        fprintf(stderr, "Sample rate: %d\n", s->sample_rate);
}

If I add fflush(stderr); underneath the second fprintf statement, I see these lines on the console, without modifying a thing in my own code. I still don't understand why it works in Linux without this, but at least I know the answer.

Answer 1

Steeviebops: To your last comment "... beginning to wonder if this is even possible"

It is, and here's how you can do it using DataFetcher, a Threadable callback reader: ( you might want to do with System.err also) (https://sourceforge.net/p/tus/code/HEAD/tree/tjacobs/io/)

  DataFetcher df = new DataFetcher(System.in, new byte[1024], 0);
  FetcherListener fl = new FetcherListener() {
    public void gotAll(byte[] bytes) {}
    public void getMore(byte[] bytes, int start, int end) {
      SwingUtilities.invokeLater(new Runnable() {
         txtConsoleOutput.append(new String (bytes, start, end-start) + "\n");
         // You may or may not need/want to do a repaint here
      });
    }
  }
  df.addFetcherListener(fl);
  new Thread(df).start();

Answer 2

Related information to my own comment:

Another thought: Have you tried reproducing this without Swing and just dumping the stuff read from the streams onto the text console of your program? Maybe the problem that it works with ping but not with the other test program is that the latter simply writes into a buffered stream which only gets flushed once in a while (e.g. when exiting) and hence there is nothing to read for your own program. I imagine that writing "Hello" + "world" to a stream with a buffer significantly bigger than those short strings might cause such behaviour. ping however might write and flush directly.

If for example your sample program uses System.out.print() instead of println(), you will never see anything on the console because the output will be buffered. Only after you insert println() - implying a call to BufferedWriter.flushBuffer() - or directly flush the writer's buffer, the other program reading from the first process' console gets to read something.

Target application, writing to the console:

import java.util.Random;

public class TargetApp {
  public static void main(String[] args) throws InterruptedException {
    System.out.print("Hello ");
    Thread.sleep(1500);
    System.out.println("world!");
    Random random = new Random();
    for (int i = 0; i < 250; i++) {
      System.out.print("#");
      if (random.nextInt(20) == 0)
        System.out.println();
      Thread.sleep(50);
    }
  }
}

Controller application, reading the target application's console output:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;

public class ControllerApp extends Thread {
  List<String> allArgs = Arrays.asList(
    //"ping", "-n", "3", "google.de"
    "java", "-cp", "bin", "TargetApp"
  );

  @Override
  public void run() {
    try (
      BufferedReader inputStream = new BufferedReader(
        new InputStreamReader(
          new ProcessBuilder(allArgs)
            .redirectErrorStream(true)
            .start()
            .getInputStream()
        )
      )
    ) {
      String cmdOutput;
      while ((cmdOutput = inputStream.readLine()) != null) {
        System.out.println(cmdOutput);
      }
    }
    catch (IOException ex) {
      throw new RuntimeException(ex);
    }
  }

  public static void main(String[] args) throws InterruptedException {
    new ControllerApp().start();
  }
}

If you run ControllerApp, you will see that it does not read "Hello " and "world!" separately but at once. The same is true for the "#" strings. It reads them in chunks according to the other program's buffer flushing behaviour. You will also notice that it is writing the following protocol in stop-and-go manner, not like a continuous stream of "#" characters every 50 ms.

Hello world!
#####
#################
############
####
#############
##########################################
###############
################
#################
######
##########
######
#######
############################################
#########
#################
##########

So if this is your problem, it is in TargetApp rather than in ControllerApp. Then it would also be unrelated to Swing.

---

Update: I forgot to mention that you can emulate the behaviour that you see the last output only after TargetApp exits by commenting out these two lines:

//      if (random.nextInt(20) == 0)
//        System.out.println();

Then the console log looks like this, the long "#" line only being printed when TargetApp terminates:

Hello world!
##########################################################################################################################################################################################################################################################