Using python to read command line output while invoked process is still running

Question

There seem to be a lot of ways to invoke some system process via the command line using python, and then to read the output once the process is complete. For example, the subprocess module can accomplish this via subprocess.Popen(['ls']).

Is there a way to read the output from a command while it is still running?

For example, if I invoke the python script multiply.py:

import time

def multiply(a,b):
    newA = a
    newB = b
    while True: #endless loop
        newA = newA+1
        newB = newB+1

        product = newA * newB
        print 'Product: ',product
        time.sleep(1)


multiply(40,2)

using something along the lines of subprocess.Popen(['python', 'multiply.py]), is there a way to

1. Start the process
2. Silently/actively capture all the output while the process is running
3. 'jump in' at any time to check the contents of the entire output?

The above python script is a model of the sort of process from which I'm interested in capturing output, i.e. there is an endless loop, which prints an output every second; it is this output that I'm interested in actively monitoring/capturing.

Answer 1

Here is an implementation that opens the process on another thread (so you won't have to block the main thread) and communicates back lines using a Queue. The thread is killed on demand using an Event:

#!/usr/bin/env python

from subprocess import Popen, PIPE
from threading import Thread, Event
from Queue import Queue
from time import sleep

# communicates output lines from process to main thread
lines = Queue()
# kills thread
kill = Event()


def runner():
    p = Popen(["python", "multiply.py"], stdout=PIPE)
    # to get stream behaviour
    while p.poll() is None and not kill.isSet():
        lines.put(p.stdout.readline())

    p.kill()
    # in your use case this is redundant unless the process
    # is terminated externally - because your process is never done
    # after process finished some lines may have been missed
    if not kill.isSet():
        for line in p.stdout.readlines():
            lines.put(line)

# open process on another thread    
t = Thread(target=runner)
t.start()

# do stuff - lines aggregated on the queue    
print "First run"
sleep(1)
while not lines.empty():
    print lines.get()

# move to more important stuff...
sleep(3)

# now check the output we missed
print "Second run"
while not lines.empty():
    print lines.get()

# done doing stuff

# tell thread to kill itself
kill.set()
# wait for thread
t.join()
print "done"