Use StringIO as stdin with Popen

Question

I have the following shell script that I would like to write in Python (of course grep . is actually a much more complex command):

#!/bin/bash

(cat somefile 2>/dev/null || (echo 'somefile not found'; cat logfile)) \
| grep .

I tried this (which lacks an equivalent to cat logfile anyway):

#!/usr/bin/env python

import StringIO
import subprocess

try:
    myfile = open('somefile')
except:
    myfile = StringIO.StringIO('somefile not found')

subprocess.call(['grep', '.'], stdin = myfile)

But I get the error AttributeError: StringIO instance has no attribute 'fileno'.

I know I should use subprocess.communicate() instead of StringIO to send strings to the grep process, but I don't know how to mix both strings and files.

You cannot use `StringIO` objects to provide process input; use `subprocess.PIPE` instead. — Martijn Pieters, Dec 13 '13 at 13:51
@MartijnPieters as I said (last sentence), "I know I should use subprocess.communicate() instead of StringIO to send strings to the grep process, but I don't know how to mix both strings and files." — Suzanne Soy, Dec 13 '13 at 13:56
Why not read from the open file object, write to the pipe? If there is no open file, write the alternative text. — Martijn Pieters, Dec 13 '13 at 13:58
@Faust because in my case, the command is `gpg -weird -options -r recipients | mail -s subject recipients`, not just `grep`, and I want to send the e-mail in all cases, with a fallback when the body isn't found. — Suzanne Soy, Dec 13 '13 at 14:01
Oh, ok. You could do it in a full-pythonic way using some library. But I understand your point. — smeso, Dec 13 '13 at 14:03

score 10 · Accepted Answer · answered Dec 13 '13 at 14:19

10

p = subprocess.Popen(['grep', '...'], stdin=subprocess.PIPE, 
                                      stdout=subprocess.PIPE)
output, output_err = p.communicate(myfile.read())

answered Dec 13 '13 at 14:19

Ski

13,009
3
47
59

6

Doesn't this read the whole contents of `myfile` into memory, allocate a string for it, etc.? Shouldn't there be a way to pass the file handle directly to the next process? – ShreevatsaR Apr 04 '17 at 16:35

score 4 · Answer 2 · answered Nov 02 '15 at 20:08

Don't use bare except, it may catch too much. In Python 3:

#!/usr/bin/env python3
from subprocess import check_output

try:
    file = open('somefile', 'rb', 0)
except FileNotFoundError:
    output = check_output(cmd, input=b'somefile not found')
else:
    with file:
        output = check_output(cmd, stdin=file)

It works for large files (the file is redirected at the file descriptor level -- no need to load it into the memory).

If you have a file-like object (without a real .fileno()); you could write to the pipe directly using .write() method:

#!/usr/bin/env python3
import io
from shutil import copyfileobj
from subprocess import Popen, PIPE
from threading import Thread

try:
    file = open('somefile', 'rb', 0)
except FileNotFoundError:
    file = io.BytesIO(b'somefile not found')

def write_input(source, sink):
    with source, sink:
        copyfileobj(source, sink)

cmd = ['grep', 'o']
with Popen(cmd, stdin=PIPE, stdout=PIPE) as process:
    Thread(target=write_input, args=(file, process.stdin), daemon=True).start()
    output = process.stdout.read()

Neat suggestion with `copyfileobj()`. Could you edit your answer to clarify, why a thread is required? I suppose it's required to avoid deadlocks while reading `process`es output. — smbear, Apr 16 '19 at 09:06
@smbear what do you think would happen if a thread (or async. io) is not used? — jfs, Apr 16 '19 at 17:04

Use StringIO as stdin with Popen

2 Answers2

Linked