I'm writing a wrapper class for use with a workflow manager. I would like to log output from an application (child process executed via subprocess.Popen) in a certain way:
stdoutof the child should go to a log file and tostdoutof the parent,stderrof the child should go to a different logfile, but also tostdoutof the parent.
I.e. all output from the child should end up merged on stdout (like with subprocess.Popen(..., stderr=subprocess.STDOUT), so I can reserve stderr for log messages from the wrapper itself. On the other hand, the child's streams should go to different files to allow separate validation.
I've tried using a "Tee" helper class to tie two streams (stdout and the log file) together, so that Tee.write writes to both streams. However, this cannot be passed to Popen because "subprocess" uses OS-level functions for writing (see here: http://bugs.python.org/issue1631).
The problem with my current solution (code snippet below, adapted mostly from here) is that output on stdout may not appear in the right order.
How can I overcome this? Or should I use an altogether different approach?
(If I stick with the code below, how do I choose a value for the number of bytes in os.read?)
import subprocess, select, sys, os
call = ... # set this
process = subprocess.Popen(call, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
logs = {process.stdout: open("out.log", "w"), process.stderr: open("err.log", "w")}
done = {process.stdout: False, process.stderr: False}
while (process.poll() is None) or (not all(done.values())):
ready = select.select([process.stdout, process.stderr], [], [])[0]
for stream in ready:
data = os.read(stream.fileno(), 1)
if data:
sys.stdout.write(data)
logs[stream].write(data)
else:
done[stream] = True
logs[process.stdout].close()
logs[process.stderr].close()
By the way, this solution using "fcntl" has not worked for me. And I couldn't quite figure out how to adapt this solution to my case yet, so I haven't tried it.
