How can I print and display subprocess stdout and stderr output without distortion?

Question

Maybe there's someone out in the ether that can help me with this one. (I have seen a number of similar questions to this on SO, but none deal with both standard out and standard error or deal with a situation quite like mine, hence this new question.)

I have a python function that opens a subprocess, waits for it to complete, then outputs the return code, as well as the contents of the standard out and standard error pipes. While the process is running, I'd like to also display the output of both pipes as they are populated. My first attempt has resulted in something like this:

process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

stdout = str()
stderr = str()
returnCode = None
while True:
    # collect return code and pipe info
    stdoutPiece = process.stdout.read()
    stdout = stdout + stdoutPiece
    stderrPiece = process.stderr.read()
    stderr = stderr + stderrPiece
    returnCode = process.poll()

    # check for the end of pipes and return code
    if stdoutPiece == '' and stderrPiece == '' and returnCode != None:
        return returnCode, stdout, stderr

    if stdoutPiece != '': print(stdoutPiece)
    if stderrPiece != '': print(stderrPiece)

There's a couple problems with this though. Because read() reads until an EOF, the first line of the while loop will not return until the subprocess closes the pipe.

I could replace the read() in favor of read(int) but the printed output is distorted, cut off at the end of the read characters. I could readline() as a replacement, but the printed output is distorted with alternating lines of output and errors when there are many of both that occur at the same time.

Perhaps there's a read-until-end-of-buffer() variant that I'm not aware of? Or maybe it can be implemented?

Maybe it's best to implement a sys.stdout wrapper as suggested in this answer to another post? I would only want to use the wrapper in this function, however.

Any other ideas from the community?

I appreciate the help! :)

EDIT: The solution really should be cross-platform, but if you have ideas that aren't, please share them away to keep the brainstorming going.

---

For another one of my python subprocess head scratchers, take a look at another of my questions on accounting for subprocess overhead in timing.

Answer 1

Make the pipes non-blocking by using fcntl.fcntl, and use select.select to wait for data to become available in either pipe. For example:

# Helper function to add the O_NONBLOCK flag to a file descriptor
def make_async(fd):
    fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)

# Helper function to read some data from a file descriptor, ignoring EAGAIN errors
def read_async(fd):
    try:
        return fd.read()
    except IOError, e:
        if e.errno != errno.EAGAIN:
            raise e
        else:
            return ''

process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
make_async(process.stdout)
make_async(process.stderr)

stdout = str()
stderr = str()
returnCode = None

while True:
    # Wait for data to become available 
    select.select([process.stdout, process.stderr], [], [])

    # Try reading some data from each
    stdoutPiece = read_async(process.stdout)
    stderrPiece = read_async(process.stderr)

    if stdoutPiece:
        print stdoutPiece,
    if stderrPiece:
        print stderrPiece,

    stdout += stdoutPiece
    stderr += stderrPiece
    returnCode = process.poll()

    if returnCode != None:
        return (returnCode, stdout, stderr)

Note that fcntl is only available on Unix-like platforms, including Cygwin.

If you need it to work on Windows without Cygwin, it's doable, but it's much, much tougher. You'll have to:

• Use the pywin32 library to call through to the native Win32 API
• Use SetNamedPipeHandleState with PIPE_NOWAIT to make the stdout and stderr pipes non-blocking
• Use WaitForMultipleObjects instead of select to wait for data to become available
• Use ReadFile to read the data

Answer 2

Combining this answer with this, the following code works for me:

import subprocess, sys
p = subprocess.Popen(args, stderr=sys.stdout.fileno(), stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ""):
 print line,

Answer 3

When I tested it, it seemed readline() is blocking. However I was able to access stdout and stderr separately using threads. Code sample as follows:

import os
import sys
import subprocess
import threading

class printstd(threading.Thread):
    def __init__(self, std, printstring):
        threading.Thread.__init__(self)
        self.std = std
        self.printstring = printstring
    def run(self):
        while True:
          line = self.std.readline()
          if line != '':
            print self.printstring, line.rstrip()
          else:
            break

pythonfile = os.path.join(os.getcwd(), 'mypythonfile.py')

process = subprocess.Popen([sys.executable,'-u',pythonfile], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

print 'Process ID:', process.pid

thread1 = printstd(process.stdout, 'stdout:')
thread2 = printstd(process.stderr, 'stderr:')

thread1.start()
thread2.start()

threads = []

threads.append(thread1)
threads.append(thread2)

for t in threads:
    t.join()

However, I am not certain that this is thread-safe.