Receive return data from subprocess in python

Question

I'm spawning a process from a script using subprocess. My subprocess takes a JSON input and performs some operations and should return some real time data to the main process. How can I do this from subprocess? I'm trying something like this. But it is throwing an error.

Following is may main process "main.py"

p = subprocess.Popen(['python','handler.py'],
                            stdin=subprocess.PIPE,stdout=subprocess.PIPE)

p.communicate(JSONEncoder().encode(data))
while True:
     out = process.stdout.read(1)
     if out == '' and process.poll() != None:
        break
     if out != '':
        sys.stdout.write(out)
        sys.stdout.flush()

Below is my subprocess "handler.py"

if __name__ == '__main__' :

   command = json.load(sys.stdin)
   os.environ["PYTHONPATH"] = "../../"

   if command["cmd"] == "archive" :
      print "command recieved:",command["cmd"]
      file_ids, count = archive(command["files"])
      sys.stdout.write(JSONEncoder().encode(file_ids))

But it throws an error.

Traceback (most recent call last):
  File "./core/main.py", line 46, in <module>
  out = p.stdout.read(1)
ValueError: I/O operation on closed file

Am I doing something wrong here??

Where does `process` come from? You create `p` with `subprocessPopen`. — Mike Müller, Dec 11 '15 at 08:33

score 2 · Answer 1 · edited Dec 19 '18 at 21:07

2

communicate reads all the output from a subprocess and closes it. If you want to be able to read from the process after writing, you have to use something other than communicate, such as p.stdin.write. Alternatively, just use the output of communicate; it should have what you want https://docs.python.org/3/library/subprocess.html#popen-objects.

edited Dec 19 '18 at 21:07

Community

1
1

answered Dec 11 '15 at 08:36

FerHai

106
4

score 2 · Answer 2 · edited May 23 '17 at 11:51

Popen.communicate() does not return until the process is dead and it returns all the output. You can't read subprocess' stdout after it. Look at the top of the .communicate() docs:

Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate._{emphasis is mine}

If you want to send data and then read the output line by line as text while the child process is still running:

#!/usr/bin/env python3
import json
from subprocess import Popen, PIPE

with Popen(command, stdin=PIPE, stdout=PIPE, universal_newline=True) as process:
    with process.stdin as pipe:
        pipe.write(json.dumps(data))
    for line in process.stdout:
        print(line, end='')
        process(line)

If you need code for older python versions or you have buffering issues, see Python: read streaming input from subprocess.communicate().

If all you want is to pass data to the child process and to print the output to terminal:

#!/usr/bin/env python3.5
import json
import subprocess

subprocess.run(command, input=json.dumps(data).encode())

If your actual child process is a Python script then consider importing it as a module and running the corresponding functions instead, see Call python script with input with in a python script using subprocess.

Receive return data from subprocess in python

2 Answers2