Subprocess displays sys.stdout correctly in PyCharm but not in the console

Question

I have a small Python program that executes a terminal command from a pip package called commandwrapper (which is a wrapper for subprocess.popen: https://pypi.python.org/pypi/commandwrapper/0.7). I am also trying to capture the real-time output to the console and to a file.

I have the code:

class Tee(object):
    def __init__(self, *files):
        self.files = files
    def write(self, obj):
        for f in self.files:
            f.write(obj)
            f.flush()
    def flush(self) :
        for f in self.files:
            f.flush()

# Set the stdout/stderr to Tee()
out = open(stdout_log, 'w')
stdout_reset = sys.stdout
sys.stdout = Tee(sys.stdout, out)

process = commandwrapper.WrapCommand(command, shell=True) # Also tried False with same behaivor.
cmd = process.prepareToRun()

# Stream the output to the console and stdoutfiles
while cmd.poll() is None:
    msg_out = cmd.stdout.readline()
    sys.stdout.write(msg_out)
    sys.stdout.flush()

out.close()

This works perfect when I run it in PyCharm. The output of command is written to the file AND displayed on the terminal console in real-time.

However, when I run the same code in a terminal, no output is displayed on the console. How can this be? The stdout is correctly captured in a file, but nothing is written to the console.

Can anyone see any reason that this code would work well and as expected in PyCharm, but not display any stdout to the terminal? I'm at a loss here. If anything, I could deal with it if the behavior was reversed.

Using OSX Yosemite and running bash.

Its a wrapper for `subprocess.popen`. I added link to the `pip` package above. Thanks for pointing that out. — Brett, Dec 02 '15 at 23:49
OSX and the default terminal in OSX (I think thats `bash`?). — Brett, Dec 02 '15 at 23:52
Might want to add that to your question, I cannot replicate on ubuntu using bash, `echo $0` will show you the shell name on linux, it may work for mac too — Padraic Cunningham, Dec 02 '15 at 23:53
Actually,scratch that it does not work at all from anywhere properly including pycharm, I get a fraction of the files listed running `ls` — Padraic Cunningham, Dec 02 '15 at 23:59
Hmmm...still works ok for me in PyCharm. Any more info I could add that would make it more debuggable? — Brett, Dec 03 '15 at 00:02
Ah, ok. Guess I'll try ditching `commandwrapper`. I thought that might have been a clean way to avoid working with `subprocess` directly, but I guess not. — Brett, Dec 03 '15 at 00:05
It is the poll logic that is the issue, I added the working code, you can use the wrapper if you prefer but I don't see much point really unless I am missing something — Padraic Cunningham, Dec 03 '15 at 00:06

Padraic Cunningham · Accepted Answer · 2015-12-03T00:11:54.607

1

You need to change the logic where you have poll, I used Popen but you can use the wrapper if you prefer:

out = open(stdout_log, 'w')
stdout_reset = sys.stdout
sys.stdout = Tee(sys.stdout, out)
from subprocess import Popen,PIPE,STDOUT
process = Popen([list_of_cmds],stdout=PIPE,stderr=STDOUT)
# Stream the output to the console and stdoutfiles
for line in iter(process.stdout.readline,""):
    sys.stdout.write(line)


out.close()

Applying the same logic works with the commandwrapper lib:

process = commandwrapper.WrapCommand(command, shell=True) # Also tried False with same behaivor.
cmd = process.prepareToRun()
# Stream the output to the console and stdoutfiles
for line in iter(cmd.stdout.readline,""):
    sys.stdout.write(line)

edited Dec 03 '15 at 00:11

answered Dec 03 '15 at 00:06

Padraic Cunningham

160,756
20
201
286

This is fantastic. Thank you :-) – Brett Dec 03 '15 at 00:14
No prob, you're welcome. If you `print(cmd.stdout.readlines())` outside your while loop you will see the rest/all of the output as the data is getting buffered regardless of the flush – Padraic Cunningham Dec 03 '15 at 00:20

score 0 · Answer 2 · edited May 23 '17 at 12:30

Even if the child process has exited i.e., cmd.poll() is not None there could be a buffered output left in the pipe. You could see it if you call cmd.stdout.read() after the while cmd.poll() is not None loop has ended.

To reproduce the error without Tee, commandwrapper:

#!/usr/bin/env python
import sys
import time
from subprocess import Popen, PIPE

#XXX DO NOT USE THE CODE -- ITS PURPOSE IS TO DEMONSTRATE THE ISSUE
p = Popen([sys.executable, '-c', "import os; os.write(1, b'abc')"],
          stdout=PIPE, bufsize=0) # no buffering in the parent
time.sleep(0.3) # allow the child to exit
while p.poll() is None: # poll() should return non-None value
    assert 0, "it is never run unless the computer is slow"
    line = p.stdout.readline()
    print(line)

output = p.stdout.read() #
print("the output left after the child has died: %r" % (output,))
assert len(output) >= 3

See these posts on how to read subprocess' output in real time and how to redirect it to a file and terminal at the same time:

Subprocess displays sys.stdout correctly in PyCharm but not in the console

2 Answers2

Linked