1

I'm writing a program for Windows 7 using Python 2.7.9. This program allows the user to schedule and run a list of scripts and needs to show the output of the scripts in real-time. In order to run the scripts I have the following code:

self.proc = Popen(command, shell=False, stdout=PIPE, stderr=STDOUT)
for line in iter(self.proc.stdout.readline, b''):
    print line

This is working and runs the scripts just fine but the issue is that I only see output after the script has finished running which is not acceptable. For example I have a simple program:

from time import sleep
print "test: Can you see me?"
#sys.stdout.flush() 
sleep(10)
print "ending"

I do not see any of the print statements until after the sleep time at which point the script ends and everything is printed to the console at once. I've tried a few different approaches but nothing gets it to print in real time. The only thing I have found to work is to make the script I'm running unbuffered by adding the sys.stdout.flush() or adding the following to the beginning of each python script I want to run:

unbuffered = os.fdopen(sys.stdout.fileno(), 'w', 0)
sys.stdout = unbuffered

I also need to be able to run other scripts like perl and java so this is not a good fix.

So is there any way to execute scripts from a python program and have the output shown in real time? I'm open to other suggestions as well, maybe there is a better approach than using subprocess that I'm not aware of.

The CodeWriter
  • 81
  • 1
  • 6
  • duplicate of http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess – Dan Sep 24 '15 at 22:13
  • or duplicate of http://stackoverflow.com/q/1606795/868044 or http://stackoverflow.com/q/2082850/868044 or http://stackoverflow.com/q/17411966/868044 – Dan Sep 24 '15 at 22:14
  • if `command` is a python script then use `-u` parameter. Or (better) [import the modules and runs corresponding functions (perhaps using `multiprocessing` instead of spawning a subprocess.](http://stackoverflow.com/q/30076185/4279) – jfs Sep 24 '15 at 22:47
  • you could [provide a pseudo-tty or use `stdbuf` on Unix](http://stackoverflow.com/a/17698359/4279). On Windows, you could try to run the commands in a console. – jfs Sep 24 '15 at 22:54
  • @Dan: the only question from your links that has answers that are dedicated to the bufferring issue is [real time subprocess.Popen via stdout and PIPE](http://stackoverflow.com/questions/2082850/real-time-subprocess-popen-via-stdout-and-pipe). I expect none of the existing answers to work on Windows (OP could try [`winpexpect` module](https://bitbucket.org/geertj/winpexpect/overview) but it hasn't been updated in a while) – jfs Sep 24 '15 at 23:14
  • Thank you all for the suggestions but nothing has worked so far. It's hard to believe that this is impossible to do using subprocess, but that seems to be the case. Does anyone have any alternate suggestions that might not use the subprocess module? I'm at a loss. – The CodeWriter Oct 01 '15 at 21:13
  • If `-u` parameter doesn't work for `python` script then your code is likely broken. [Provide complete minimal code example that demonstrates the issue](http://stackoverflow.com/help/mcve). – jfs Oct 02 '15 at 08:05
  • Actually the -u parameter does fix the issue! However it's only good for python scripts, I need a solution that will work for Perl and Java scripts as well. – The CodeWriter Oct 02 '15 at 14:55
  • @TheCodeWriter: the buffers are inside the child processes: you can't get the data sooner unless they cooperate: find out how to disable buffering for perl, Java programs. It is not related to Python. – jfs Oct 03 '15 at 03:36
  • But if that's the case then can you tell me why these programs run from the command line in real time with no buffering, but they do buffer when they are opened from a subprocess in python? – The CodeWriter Oct 03 '15 at 16:53
  • @TheCodeWriter: use @ syntax if you want me to be notified about your comments. Here's the explanation for Unix (you might have already encountered it if you have followed the links I've provided above): [Q: Why not just use a pipe (popen())?](http://pexpect.readthedocs.org/en/latest/FAQ.html#whynotpipe) (I don't know what is `pty` analog on Windows) – jfs Oct 04 '15 at 04:35

0 Answers0