0

I am starting a subprocess in python through this command:

result = subprocess.Popen([sys.executable, "/subscript.py"] + parameter,stdout=subprocess.PIPE )
result.wait()
out, err = result.communicate()
for line in out.splitlines():
    print line

The problem that I am facing with this approach is that all the print statements in subscript.py are kind of buffered till the end and are printed on the terminal at the end of the execution. So if my subscript.py takes about 15 minutes in execution then for 15 minutes the terminal does not shows anything and once the subscript.py execution ends then all the print statements are printed together. I know this is because of the use of stdout=subprocess.PIPE and if I remove it then I will get what I want but I cannot remove it as I am using it to get the value of a variable from the subscript.py (I am printing that variable in subscript.py and then in parent script I am going through each print to get the print with variable value).

So, what I want is that all the print statements should be printed as they are encountered in the subscript.py without any buffering till the end of the execution. Is there a way to do that?

UPDATE: I tried to follow J.F. Sebastian's method but I still get the same output. The prints from subprocess are buffered and once the subprocess ends then they are printed to the terminal. The updated code I have is:

result = subprocess.Popen([sys.executable, "/subscript.py"] + parameter,stdout=subprocess.PIPE,bufsize=1)

with result.stdout:
        for line in iter(result.stdout.readline, b''):
            print line,
result.wait()
Community
  • 1
  • 1
user2966197
  • 2,193
  • 7
  • 33
  • 62
  • Have you tried using `result.stdout.readline`? You can wrap it in `iter()` and you should be able to iterate through the stdout lines as they are generated. – russdot Jul 09 '15 at 15:10
  • @russdot No I haven't tried that. At which point should I do that? I mean should i do it before `result.wait()` or after it? – user2966197 Jul 09 '15 at 15:11
  • @russdot can you show how to wrap it in `iter()` and where should I place it in my code? – user2966197 Jul 09 '15 at 15:14
  • Check the [docs](https://docs.python.org/2/library/subprocess.html#popen-objects): `wait()` waits until the child process terminates, and should not be used with `stdout=PIPE`. Instead, try removing calls to `wait()` and `communicate()` and instead try: `for line in iter(result.stdout.readline): print line` – russdot Jul 09 '15 at 15:19
  • @russdot the reason I am using `wait()` is so that my parent script waits for the subprocess to finish before moving ahead with next parts of the code. The code that I have posted above is just a small part of the parent script. The parts after the above code depend on the successful execution of above script so I have to wait till the above script finishes – user2966197 Jul 09 '15 at 15:23
  • The purpose of `iter()` is that it will iterate through the `stdout` lines until EOF is reached. So the `for line in iter()` loop should keep going until the output from the subprocess has been exhausted. To be honest I haven't had a chance to test the code, though. – russdot Jul 09 '15 at 15:27
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/82841/discussion-between-russdot-and-user2966197). – russdot Jul 09 '15 at 15:31
  • @russdot I executed the above and got `TypeError: 'builtin_function_or_method' object is not iterable error at for line in iter(result.stdout.readline):` – user2966197 Jul 09 '15 at 15:38
  • 1
    Maybe `sys.stdout.flush()` somewhere? – Zah Jul 09 '15 at 16:25

1 Answers1

2

The answer that you've linked prints anything only after the stdout buffer is flushed in the child as it is explicitly said in the answer itself.

Use -u parameter, to unbuffer stdout in the Python child process:

#!/usr/bin/env python2
import sys
from subprocess import Popen, PIPE

result = Popen([sys.executable, '-u', '/path/to/subscript.py'] + parameters, 
               stdout=PIPE, bufsize=1)
with result.stdout:
    for line in iter(result.stdout.readline, b''):
        print line,
result.wait()
Community
  • 1
  • 1
jfs
  • 346,887
  • 152
  • 868
  • 1,518