How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out

Question

I want output from execute Test_Pipe.py, I tried following code on Linux but it did not work.

Test_Pipe.py

import time
while True :
    print "Someting ..."
    time.sleep(.1)

Caller.py

import subprocess as subp
import time

proc = subp.Popen(["python", "Test_Pipe.py"], stdout=subp.PIPE, stdin=subp.PIPE)

while True :
    data = proc.stdout.readline() #block / wait
    print data
    time.sleep(.1)

The line proc.stdout.readline() was blocked, so no data prints out.

Duplicate: http://stackoverflow.com/search?q=%5Bpython%5D+subprocess+output, http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess, http://stackoverflow.com/questions/1277866/python-subprocess-module-looping-over-stdout-of-child-process — S.Lott, Sep 08 '09 at 00:43

score 47 · Answer 1 · answered Sep 07 '09 at 11:04

You obviously can use subprocess.communicate but I think you are looking for real time input and output.

readline was blocked because the process is probably waiting on your input. You can read character by character to overcome this like the following:

import subprocess
import sys

process = subprocess.Popen(
    cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

while True:
    out = process.stdout.read(1)
    if out == '' and process.poll() != None:
        break
    if out != '':
        sys.stdout.write(out)
        sys.stdout.flush()

[you don't need `process.poll()` in this case](http://stackoverflow.com/a/17701672/4279). — jfs, Jul 17 '13 at 14:25

Derrick Petzold · Answer 2 · 2012-02-16T07:28:56.057

21

Nadia's snippet does work but calling read with a 1 byte buffer is highly unrecommended. The better way to do this would be to set the stdout file descriptor to nonblocking using fcntl

fcntl.fcntl(
    proc.stdout.fileno(),
    fcntl.F_SETFL,
    fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL) | os.O_NONBLOCK,
)

and then using select to test if the data is ready

while proc.poll() == None:
    readx = select.select([proc.stdout.fileno()], [], [])[0]
    if readx:
        chunk = proc.stdout.read()
        print chunk

She was correct in that your problem must be different from what you posted as Caller.py and Test_Pipe.py do work as provided.

https://derrickpetzold.com/p/capturing-output-from-ffmpeg-python/

edited Feb 16 '12 at 07:28

answered Aug 21 '10 at 07:03

Derrick Petzold

960
11
13

It won't produce the output any sooner than the code in the question due to [the block-buffering issue. See my answer](http://stackoverflow.com/a/17701672/4279). – jfs Jul 17 '13 at 14:22
The code does produce output in realtime. Did you have a problem with it? – Derrick Petzold Jul 06 '15 at 06:56
follow the link in the comment. – jfs Jul 06 '15 at 11:10
What does it mean for data "to be ready?" Is it newline delineated or just if at least one byte of data is ready (even if the producer of such data is still writing to its stdout)? – sherrellbc Sep 20 '16 at 16:07
@me, J.F.Sebastian's answer below addresses this. – sherrellbc Sep 20 '16 at 16:09
to be clear: neither yours nor Nadia's answers produce the output in "real-time" due to the **internal** buffers in the child process. `os.O_NONBLOCK` in your code (in the *parent* process) has no effect on internal stdio buffers in the *child* process. For example, running your code produce 4K chunks instead of `len(chunk) == 13` (one line). To see the difference, pass `-u` option (run `[sys.executable or 'python2', '-u', 'Test_Pipe.py']` command). – jfs Mar 26 '18 at 15:49

jfs · Answer 3 · 2018-03-26T15:49:50.243

15

Test_Pipe.py buffers its stdout by default so proc in Caller.py doesn't see any output until the child's buffer is full (if the buffer size is 8KB then it takes around a minute to fill Test_Pipe.py's stdout buffer).

To make the output unbuffered (line-buffered for text streams) you could pass -u flag to the child Python script. It allows to read subprocess' output line by line in "real-time":

import sys
from subprocess import Popen, PIPE

proc = Popen([sys.executable, "-u", "Test_Pipe.py"], stdout=PIPE, bufsize=1)
for line in iter(proc.stdout.readline, b''):
    print line,
proc.communicate()

See links in Python: read streaming input from subprocess.communicate() on how to solve the block-buffering issue for non-Python child processes.

edited Mar 26 '18 at 15:49

answered Jul 17 '13 at 13:53

jfs

346,887
152
868
1,518

1

Again you are using 1 byte buffer which is incredibly inefficient. – Derrick Petzold Jul 06 '15 at 06:48
7

@DerrickPetzold: wrong. `bufsize=1` means "line-buffered". It uses the same buffer size as `bufsize=-1`. You could have found out that both of your comments are wrong if you were actually to run the code (compare time performance and measure the time before the first byte is read) – jfs Jul 06 '15 at 11:12
1

i like this way of doing it because you can print directly to screen and also store the results of the popen to do something with, and later on check error code etc... thumbs up =) i run it with out the bufsize and does work fine for me in my code. – pelos Feb 16 '17 at 15:55

score 14 · Answer 4 · edited Sep 29 '12 at 08:39

14

To avoid the many problems that can always arise with buffering for tasks such as "getting the subprocess's output to the main process in real time", I always recommend using pexpect for all non-Windows platform, wexpect on Windows, instead of subprocess, when such tasks are desired.

edited Sep 29 '12 at 08:39

Nils Lindemann

622
10
21

answered Sep 07 '09 at 16:10

Alex Martelli

762,786
156
1,160
1,345

How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out

4 Answers4

Linked

Related