Python: subprocess.popen: read each line of output

Question

I'm having trouble reading my subprocess output by line. The subprocess simply greps the contents of a file against another file. The output, which should be a two column file, prints to stdout just fine. But when I try to read each line, it reads each char followed by \n:

#!/usr/bin/python    

import sys
import getopt
import os
import subprocess
from subprocess import Popen, PIPE, STDOUT

inputfile = ''
target = ''

try:
        opts, args = getopt.getopt(sys.argv[1:],"s:t:",['servers=', 'target='])
except getopt.GetoptError:
        print 'getopt failure: exit'
        sys.exit()

for opt, arg in opts:
        if opt in ("-s", "--servers"):
                inputfile = arg
        if opt in ("-t", "--target"):
                boxwin = arg

p1 = subprocess.Popen(["grep -f " + inputfile + " " + target + " | awk '{print $2, $1}'"], stdout=subprocess.PIPE, shell=True)

output, error = p1.communicate()

print output # this prints normally

for line in output:
        print line # this prints each char of output followed by \n???

Expected output after reading by line:

abc 123
def 456
ghi 789

^^ this will print if I just "print output"

Actual output when using for loop to read each line:

a
b
c

1
2
3

d
e
f

...

Any ideas? Thanks.

Answer 1

Try the following :

for line in output.split(os.linesep):

instead of:

for line in output:

Answer 2

for c in s: reads one character at a time from a string s (as it should). To get a list of lines from a string instead, you could use .splitlines() method:

lines = output.splitlines()

You don't need to call .communicate() to read the output line by line:

p = subprocess.Popen(cmd, stdout=PIPE)
for line in p.stdout:
    # do something with a line

You could modify the code to handle buffering differently or to enable universal newlines support.