Running an external executable in Python interactively

Question

A simple C program is compiled using gcc test.c -o test.exe:

        /* TEST.C */
#include <stdio.h>
#include <string.h>
#define TRUE 1

int main (void)
{
  char p[256];
  strcpy(p, "start\0");
  printf("Welcome!\n");
  while (TRUE)
  {
    printf("Input: ");
    if (fgets(p, 255, stdin) == NULL)  break;
    if (p[0] == 'q')                   break;
    printf("You Entered: %s\n", p);
  }
  return 0;
}

The subprocess module in Python portends to start a program and grab its stdin and stdout. Simple test using the above program:

import subprocess
from subprocess import PIPE, STDOUT

output = open("test.out","w")
ris = subprocess.Popen(executable="test.exe", args="", stdin=PIPE,
                       stdout=output, stderr=STDOUT,
                       universal_newlines=True, shell=True)  
com = ris.communicate(input='bye\n')
com = ris.communicate(input='q\n')
output.close()

appears to loop infinitely on the first communicate command, filling test.out with:

Welcome!
You Entered: bye
You Entered: bye
You Entered: bye
You Entered: bye
.
.
.

until the user interrupts the python process. Using ris.stdin.write('bye\n') instead of communicate does not seem to send input at all even when followed by ris.stdin.flush().

What is the proper way to interactively run an executable with subprocess? The intention is to leave a single instance of the program running in the background while multiple inputs are passed in through stdin and the corresponding outputs are read from stdout or stderr

You invoked *undefined behavior* by using value of uninitialized variable having automatic storage duration `p[0]`, which is indeterminate. — MikeCAT, Mar 16 '16 at 23:50
Before asking how to run a program in Python, you should make sure the program runs correctly from the console. — too honest for this site, Mar 16 '16 at 23:52
You failed to check the return value of `scanf()`. I believe that it is returning `EOF`. — Robᵩ, Mar 16 '16 at 23:53
Aside: One should avoid `void main(void)`. http://stackoverflow.com/a/204483/8747 — Robᵩ, Mar 16 '16 at 23:55
@Robᵩ: As `p[0] is encountered first, there is no use in meditating if `scanf` updates `p`. Once you encounter undefined behaviour, all bets are off. — too honest for this site, Mar 16 '16 at 23:55
Runs fine from the command line and using `subprocess.run("test.exe")` and entering a sequence like hi, bye, q. @Rob, you think characters are being sent other than what are listed as the input in the call to `communicate`? — user2127595, Mar 16 '16 at 23:59
No, I think the stream is being closed after the first four characters are sent. The subsequent `scanf()` calls return EOF and the process never exits. — Robᵩ, Mar 17 '16 at 00:02
@Olaf - you are correct in your "all bets are off" assertion. I still claim it is worth contemplating what happens *after* he fixes that problem. — Robᵩ, Mar 17 '16 at 00:04
@Robᵩ My question is answered by using `ris.stdin.write`, but I am still curious about why checking for `EOF` doesn't seem to prevent the loop when `com = ris.communicate('bye\n')' is called. — user2127595, Mar 17 '16 at 01:04
You're not checking correctly. Re-read the doc for [`scanf`](http://en.cppreference.com/w/c/io/fscanf) and [`fgets`](http://en.cppreference.com/w/c/io/fgets). In neither your original program nor your edited program do you check the return value of either of those calls. — Robᵩ, Mar 17 '16 at 01:49

Robᵩ · Accepted Answer · 2016-03-17T00:08:20.587

In addition to the errors in your C program that have been mentioned, you are calling .communicate() more than once. .communicate() is single-use function that sends all of its data to the subprocess and waits for the subprocess to exit.

If you must use .communicate(), use it this way:

com = ris.communicate(input='bye\nq\n')

Alternatively, you can use ris.stdin.write(), like so:

import subprocess
from subprocess import PIPE, STDOUT

output = open("test.out","w")
ris = subprocess.Popen(executable="/tmp/test.exe", args="", stdin=PIPE,
                       stdout=output, stderr=STDOUT,
                       universal_newlines=True, shell=True)
com = ris.stdin.write('bye\n')
com = ris.stdin.write('q\n')
ris.stdin.flush()
output.close()
ris.wait()

I would rather not use communicate. `ris.stdin.write()` or somoething like this that can send a stream of input would be much preferable. — user2127595, Mar 17 '16 at 00:06

Running an external executable in Python interactively

1 Answers1