Running shell commands from Python and printing the output in real time

Question

I want to write a function that will execute multiple shell commands one at a time and print what the shell returns in real time.

I currently have the following code which does not print the shell (I am using Windows 10 and python 3.6.2):

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
    p.stdin.write((command + "\n").encode("utf-8"))
p.stdin.close()
p.stdout.read()

How can I see what the shell returns in real time ?

Edit : This question is not a duplicate of the two first links in the comments, they do not help printing in real time.

Possible duplicate of [Calling an external command in Python](https://stackoverflow.com/questions/89228/calling-an-external-command-in-python) — Georgy, Feb 13 '18 at 17:43
Possible duplicate of [live output from subprocess command](https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command) — Matt_G, Feb 13 '18 at 17:47

JohanL · Accepted Answer · 2018-02-16T20:04:41.937

It is possible to handle stdin and stdout in different threads. That way one thread can be handling printing the output from stdout and another one writing new commands on stdin. However, since stdin and stdout are independent streams, I do not think this can guarantee the order between the streams. For the current example it seems to work as intended, though.

import subprocess
import threading

def stdout_printer(p):
    for line in p.stdout:
        print(line.rstrip())

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", stdin=subprocess.PIPE, 
                     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                     universal_newlines=True)

t = threading.Thread(target=stdout_printer, args=(p,))
t.start()

for command in commands:
    p.stdin.write((command + "\n"))
    p.stdin.flush()

p.stdin.close()
t.join()

Also, note that I am writing stdout line by line, which is normally OK, since it tends to be buffered and being generated a line (or more) at a time. I guess it is possible to handle an unbuffered stdout stream (or e.g. stderr) character-by-character instead, if that is preferable.

score 0 · Answer 2 · answered Feb 13 '18 at 18:07

I believe you need something like this

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
    p.stdin.write((command + "\n").encode("utf-8"))
out, err = p.communicate()
print("{}".format(out))
print("{}".format(err))

score 0 · Answer 3 · answered Feb 13 '18 at 18:11

Assuming you want control of the output in your python code you might need to do something like this

import subprocess

def run_process(exe):
    'Define a function for running commands and capturing stdout line by line'
    p = subprocess.Popen(exe.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')


if __name__ == '__main__':
    commands = ["foo", "foofoo"]
    for command in commands:
        for line in run_process(command):
            print(line)

Running shell commands from Python and printing the output in real time

3 Answers3

Linked