Displaying output of shell commands with shared environments

Question

Is there any way to display the output of a shell command in Python, as the command runs?

I have the following code to send commands to a specific shell (in this case, /bin/tcsh):

import subprocess
import select

cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

poll = select.poll()
poll.register(cmd.stdout.fileno(),select.POLLIN)

# The list "commands" holds a list of shell commands
for command in commands:
  cmd.stdin.write(command)

  # Must include this to ensure data is passed to child process
  cmd.stdin.flush() 

  ready = poll.poll()
  if ready:
     result = cmd.stdout.readline()
     print result

Also, I got the code above from this thread, but I am not sure I understand how the polling mechanism works.

What exactly is registered above?
Why do I need the variable ready if I don't pass any timeout to poll.poll()?

possible duplicate of [catching stdout in realtime from subprocess](http://stackoverflow.com/questions/1606795/catching-stdout-in-realtime-from-subprocess) — Robᵩ, Sep 16 '13 at 16:05
The post you reference addresses, I think, a different problem than what you have. What command are *you* trying to run? It is a single program? Or are you trying to send multiple commands to a subshell? — Robᵩ, Sep 16 '13 at 16:07
@Robᵩ I need to run multiple commands that share an environment (i.e. keep the shell open, and run multiple commands iteratively). For each command, I would like my program to print stdout and stder on the terminal as the shell commands spit their output — Amelio Vazquez-Reina, Sep 16 '13 at 16:12
For example, the first would source a specific shell script, setting up several environment variables (e.g. `LD_LIBRARY_PATH` etc), and a second command runs a program that relies on those environment variables. — Amelio Vazquez-Reina, Sep 16 '13 at 16:13
FYI, you might want to use https://pypi.python.org/pypi/pexpect/ — Robᵩ, Sep 16 '13 at 18:52

score 1 · Accepted Answer · answered Sep 16 '13 at 16:25

Yes, it is entirely possible to display the output of a shell comamand as the command runs. There are two requirements:

1) The command must flush its output.

Many programs buffer their output differently according to whether the output is connected to a terminal, a pipe, or a file. If they are connected to a pipe, they might write their output in much bigger chunks much less often. For each program that you execute, consult its documentation. Some versions of /bin/cat', for example, have the -u switch.

2) You must read it piecemeal, and not all at once.

Your program must be structured to one piece at a time from the output stream. This means that you ought not do these, which each read the entire stream at one go:

cmd.stdout.read()
for i in cmd.stdout:
list(cmd.stdout.readline())

But instead, you could do one of these:

while not_dead_yet:
    line = cmd.stdout.readline()

for line in iter(cmd.stdout.readline, b''):
    pass

Now, for your three specific questions:

Is there any way to display the output of a shell command in Python, as the command runs?

Yes, but only if the command you are running outputs as it runs and doesn't save it up for the end.

What exactly is registered above?

The file descriptor which, when read, makes available the output of the subprocess.

Why do I need the variable ready if I don't pass any timeout to poll.poll()?

You don't. You also don't need the poll(). It is possible, if your commands list is fairly large, that might need to poll() both the stdin and stdout streams to avoid a deadlock. But if your commands list is fairly modest (less than 5Kbytes), then you will be OK just writing them at the beginning.

Here is one possible solution:

#! /usr/bin/python

import subprocess
import select

# Critical: all of this must fit inside ONE pipe() buffer
commands = ['echo Start\n', 'date\n', 'sleep 10\n', 'date\n', 'exit\n']

cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

# The list "commands" holds a list of shell commands
for command in commands:
  cmd.stdin.write(command)

  # Must include this to ensure data is passed to child process
  cmd.stdin.flush()

for line in iter(cmd.stdout.readline, b''):
    print line

Displaying output of shell commands with shared environments

1 Answers1