redirected STDOUT without ending EOL rests in a buffer and is not shown

Question

I am writing a program (a commandline frontend) which redirects stdout and stdin of arbitrary commandline programs. The problem is platform-independent. To understand the problem, I have written a simple commandline program where the problem occurs:

#include <stdio.h>
main () {
  char Expression[200];
  printf ("Enter first expression: "); scanf ("%s", Expression);
  printf ("You have entered: %s\n", Expression);
  printf ("\n");
  printf ("(Now query with Stderr)\n");
  fprintf (stderr, "Enter second expression: "); scanf ("%s", Expression);
  printf ("You have entered: %s\n", Expression);
}

The stdout "Enter the first expression" rests in the output buffer and is not sent via redirected stdout pipe to my program. So the first question is "Enter second expression" because the buffer problem only occurs with stdout, not with stderr. The buffer content (first query) is sent if the user has typed the input and presses RETURN. So stdout only runs through the pipe after EOL, stderr shows the output immediately.

If you run Info-Zip UNZIP and you are unzipping files which already exist, UNZIP sends a query:

replace myfile.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename:

and this query is sent via stderr, so the problem does not occur. But other programs run into the problem.

The commandline frontend of Windows 10 has been rewritten. If such a situation occurs (e.g. using the cmd.exe integrated "copy" command when overwriting an existing file), the new commandline frontend waits 3 seconds (!) and then shows the buffered query. So it seems that the Microsoft programmers need to write a "dirty hack" to solve this problem.

My question is: How to force the user program to spit out the stdout buffer if no EOL has been received? I have full access to the session where the user program runs by a special helper program in the same session which enables communication between my graphical frontend program and the text window where cmd.exe runs.

This might answer your question https://stackoverflow.com/questions/12450066/flushing-buffers-in-c — Jerry Jeremiah, Apr 22 '20 at 07:18
Thank you for the "fflush" hint. I found the correct corresponding OS-specific command for flushing the buffer and I called it from my helper app which runs in the same session as cmd.exe and the user program. But nothing happens. I also tried now to close and to re-open the named pipe which redirects stdout, also no effect. I really wonder how other developers of a command-line frontend have solved this problem. — Martin V, Apr 22 '20 at 17:26

score 0 · Answer 1 · answered May 12 '20 at 10:02

I realized now that the "fflush" command has to be placed in the user program. Stdout is always buffered when redirecting, Stderr is not. That is the reason that stderr output can happen before the stdout output. I wrote a test sample where the stderr output is shown 4 lines before it should be shown.

So that means that all programmers which are writing to stdout and stderr or write user queries have to flush stdout:

//first case: query and user input
char Expression[200];
printf ("Enter an expression: "); fflush (stdout); scanf ("%s", Expression);
printf ("You have entered: %s\n", Expression);

//second case: stderr output
fflush (stdout); fprintf (stderr, "An error has occured.");

I checked Internet postings and it seems that this problem occurs on all platforms.

So if you write programs with stdout and stderr, use fflush(stdout) in these two cases. The reverse case (using stderr and then stdout) is no problem, because stderr does not get buffered. (I hope I am right for all platforms.)

Possible alternate technique in this answer: [How to turn off buffering of stdout in C](https://stackoverflow.com/a/7876756/6610379). — Phil Brubaker, May 12 '20 at 10:08
I found a post here that flushing stdout and not flushing stderr is ISO defined behaviour. See [link](https://stackoverflow.com/questions/1716296) — Martin V, May 12 '20 at 13:59

score 0 · Answer 2 · answered May 18 '20 at 14:55

No, I have no option to override the programs the users execute in my commandline frontend program.

Everyone can redirect stdout and stderr to a single text file:

[C:\] userprog >output.txt 2>&1

That means both stdout and stderr is redirected to one file "output.txt". If stdout is not flushed correctly, the incorrect order is shown in the text file.

I have realized that all programs from professional source are flushing stdout in these cases and generate a proper order.

redirected STDOUT without ending EOL rests in a buffer and is not shown

2 Answers2