Python: Respond to Command Line Prompts

Question

I am trying to use Python to interact with another program via the command line. The main problem I am having is a specific call that has multiple follow-up prompts. Initially the command line call asks for the name of a project and then proceeds to ask if I would like to view any of the subfolders of the project. I need to answer y/n to each of these in order and the answer to each is unfortunately not all y or n. Additionally, I cannot know the answer to the question without reading the individual prompts so I am incapable of sending a block of 'y's or 'n's all at once.

This is the command line call:

si viewproject

After entering the command, the command line prompts:

Enter the project name:

And an example response would be:

Enter the project name: c:/test.pj

After entering the project, it prompts the following:

Do you want to recurse into the subproject test_subprj.pj? [ynYN](n)

At which point I need to respond with either a y or n depending on if I need that subproject. Again, the response to this question is dependent on the subproject. I need to be able to read the subproject in this prompt in order to respond to it with a 'y' or 'n'

Currently I need to manually enter in the project and each of the y's and n's respectively. My goal is to automate this process using Python.

Is there a way to respond to these command line prompts automatically?

Current Progress

Subprocess Strategy

 project_path = "c:/test.pj"

 with Popen(["si", "viewproject", "--project=" + project_path], 
             stdin=PIPE, stdout=PIPE, universal_newlines=True) as p:
     for line in p.stdout: 
         if line.startswith("Do you want"):
             answer = 'n'
         else:
             continue # skip it
         print(answer, file=p.stdin) # provide answer
         p.stdin.flush()

This method is hanging after the with Popen statement. It never errors, but it never enters or exits the for statement and never completes. Currently I am defaulting all answers to "n", but that will be replaced with logic later.

Winpexpect Strategy

 import re
 import sys
 from functools import partial
 import winpexpect

 project_path = "c:/test.pj"

 p = winpexpect.winspawn('si viewproject --project=' + project_path)
 p.logfile = sys.stdout
 patterns = [re.compile('ynYN'), winpexpect.EOF]

 for found in iter(partial(p.expect, patterns), 1): # until EOF
     if found == 0:
         answer = 'n'
         p.sendline(answer)

Returns the following error message:

 Traceback (most recent call last):
   File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 541, in read_nonblocking
     handle, status, data = self.child_output.get(timeout=timeout)
   File "C:\Python33\lib\queue.py", line 175, in get
     raise Empty
 queue.Empty

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1378, in expect_loop
     c = self.read_nonblocking (self.maxread, timeout)
   File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 543, in read_nonblocking
     raise TIMEOUT('Timeout exceeded in read_nonblocking().')
 pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "K:\eclipse_3.6.0\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str
     mod = __import__(modname)
   File "C:\workspace\Test_prj\Test_prj.py", line 19, in <module>
     for found in iter(partial(p.expect, patterns), 1): # until EOF
   File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1311, in expect
     return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
   File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1325, in expect_list
     return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
   File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1409, in expect_loop
     raise TIMEOUT (str(e) + '\n' + str(self))
 pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
 <winpexpect.winspawn object at 0x0144AE50>
 version: 2.3 ($Revision: 399 $)
 command: si
 args: ['si', 'viewproject', '--project=c:/test.pj']
 searcher: searcher_re:
     0: re.compile("ynYN")
     1: EOF
 buffer (last 100 chars): 
 before (last 100 chars): 
 after: <class 'pexpect.TIMEOUT'>
 match: None
 match_index: None
 exitstatus: None
 flag_eof: False
 pid: 6448
 child_fd: 4
 closed: False
 timeout: 30
 delimiter: <class 'pexpect.EOF'>
 logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='Cp1252'>
 logfile_read: None
 logfile_send: None
 maxread: 2000
 ignorecase: False
 searchwindowsize: None
 delaybeforesend: 0.05
 delayafterclose: 0.1
 delayafterterminate: 0.1
 ERROR: Module: Test_prj could not be imported (file: C:\workspace\Test_prj\Test_prj.py).

Installing Winpexpect

Lazy Persons Way

Install Distribute

Do This

Install PyWin32

Install Winpexpect

Optional: Install Nose

Optional: Install Pip

First-World Problems

Python is a new language for me, and I had never installed a package before for Python. Additionally, Python 3.x is a little different than the other versions of Python making installing modules a little bit more of an adventure.

So, to help others get some sweet sweet module action (and to help those who are more knowledgeable see if I did anything wrong) here's a soon to be success story (hopefully) documenting how I got and installed my first module.

Setup

Python allows third-party groups to develop and distribute modules that extend the abilities of the programming language. Naturally, there is a standard way to help third-party developers make modules as easily available to the end-user as possible.

For Python 3.x, that standard for distributing modules is called Distutils.

Here is how a developer uses Distutils: Distributing Python Modules

And here is how the end-user uses Distutils: Installing Python Modules

Normally, navigating to the folder of your downloaded module in the command line and running "setup.py install" will be enough.

BUT

Sometimes life isn't so easy and you may still have problems with your installation. You may, in fact, need something else. For example, you may get the following error:

"ImportError “No Module named Setuptools”"

Luckily, there is a solution for that: Python 3: ImportError "No Module named Setuptools"

As it turns out, not everything uses distutils. Some packages use setuptools. Unfortunately, there is no setuptools for Python 3.x. Rather, Python 3.x uses distribute which is a branch of setuptools.

So for those who use Python 3.x, here is Distribute: Distribute

For those using Python 2.x, here is Setuptools: Setuptools

In the Installation Instructions for Distribute, it says the following: "Download distribute_setup.py <http://python-distribute.org/distribute_setup.py>_ and execute it, using the Python interpreter of your choice."

It also says: "Notice this file is also provided in the source release."

So I downloaded Distribute and saved it to the computer. Once it was saved to the computer, I ran distribute_setup.py from the source release and got the following error:

Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
Traceback (most recent call last):
  File "C:\Python33\lib\urllib\request.py", line 1252, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)       File "C:\Python33\lib\http\client.py", line 1049, in request
    self._send_request(method, url, body, headers)
  File "C:\Python33\lib\http\client.py", line 1087, in _send_request
    self.endheaders(body)
  File "C:\Python33\lib\http\client.py", line 1045, in endheaders
    self._send_output(message_body)
  File "C:\Python33\lib\http\client.py", line 890, in _send_output
    self.send(msg)
  File "C:\Python33\lib\http\client.py", line 828, in send
    self.connect()
  File "C:\Python33\lib\http\client.py", line 806, in connect
    self.timeout, self.source_address)
  File "C:\Python33\lib\socket.py", line 406, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workspace\PythonTest\distribute_setup.py", line 553, in <module>
    sys.exit(main())
  File "C:\workspace\PythonTest\distribute_setup.py", line 549, in main
    tarball = download_setuptools(download_base=options.download_base)
  File "C:\workspace\PythonTest\distribute_setup.py", line 204, in download_setuptools
    src = urlopen(url)
  File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python33\lib\urllib\request.py", line 473, in open
    response = self._open(req, data)
  File "C:\Python33\lib\urllib\request.py", line 491, in _open
    '_open', req)
  File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
    result = func(*args)
  File "C:\Python33\lib\urllib\request.py", line 1272, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "C:\Python33\lib\urllib\request.py", line 1255, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

Well that is no good! I honestly still do not know where that error is coming from or why it happened.

Regardless, then I found the following site that ran a .exe to install distribute as well as pip.

Install Distribute

Install Pip

So I got those installed and then used the following site to setup my computer to more easily use easy_install: Setting Up Easy Install Made Easy

Once I got this working I then installed nose:Nose

The reason I got nose was because the Winpexpect website says: "WinPexpect includes unit tests. To run the tests, you need nose. Use the following command to run the tests:

$ python setup.py test"

Well that sounds nice :). Now I just wished I knew where to run that test. I know that if you install manually you use the setup.py install command so there will most definitely be a setup.py in the zipped directory online. To see if this was correct, I downloaded and saved the winpexpect file, extracted the information, navigated to it via command-line, and ran setup.py test.

Here was the following result:

running test
running build_py
running egg_info
creating c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info
writing c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\PKG-INFO
writing dependency_links to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\dependency_links.txt
writing top-level names to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\top_level.txt
writing requirements to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\requires.txt
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
reading manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "C:\Documents and Settings\SLZ1FH\Desktop\winpexpect\geertj-winpexpect-76df3cfcb143\setup.py", line 35, in <module>
    use_2to3 = True
  File "C:\Python33\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Python33\lib\distutils\dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "C:\Python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 138, in run
    self.with_project_on_sys_path(self.run_tests)
  File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 118, in with_project_on_sys_path
    func()
  File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 164, in run_tests
    testLoader = cks
  File "C:\Python33\lib\unittest\main.py", line 124, in __init__
    self.parseArgs(argv)
  File "C:\Python33\lib\unittest\main.py", line 168, in parseArgs
    self.createTests()
  File "C:\Python33\lib\unittest\main.py", line 175, in createTests
    self.module)
  File "C:\Python33\lib\unittest\loader.py", line 137, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Python33\lib\unittest\loader.py", line 137, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Python33\lib\unittest\loader.py", line 96, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\__init__.py", line 1, in <module>
    from nose.core import collector, main, run, run_exit, runmodule
  File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\core.py", line 143
    print "%s version %s" % (os.path.basename(sys.argv[0]), __version__)
                    ^
SyntaxError: invalid syntax

Ok, so the Python 3.3 version of Nose contains invalid syntax for Python 3.3?

print "%s version %s" % (os.path.basename(sys.argv[0]), version)...

should definitely have parenthesis around it... This makes me question if nose will actually work here as it clearly looks to be made for earlier versions of Python.

Answer 1

In the comments you mentioned that xx viewproject < answers.txt > output.txt works but you can't use it because answers depend on the output from the subprocess.

In general pexpect-like modules such as winpexpect (for Windows) could be used. Something like:

import re
import sys
from functools import partial
from winpexpect import EOF, winspawn as spawn

p = spawn('xx viewproject')
p.logfile = sys.stdout
patterns = ['the project:', re.escape('? [ynYN](n)'), EOF]
for found in iter(partial(p.expect, patterns), 2): # until EOF
    if found == 0:
        p.sendline(project_name)
    elif found == 1:
        filename = get_filename_from_prompt(p.before) # a regex could be used
        answer = yes_or_no_from_subproject.get(filename, 'no') # a dict
        p.sendline(answer)

If the prompts are terminated with a newline (and the subprocess doesn't buffer them); you could read line by line using subprocess module directly:

from subprocess import Popen, PIPE

with Popen(["xx", "viewproject"], stdin=PIPE, stdout=PIPE, 
           universal_newlines=True) as p:
    for line in p.stdout: 
        if line.startswith("Please enter the name of the project"):
            answer = project_name
        elif line.startswith("Would you like to recurse into the subproject"):
            filename = get_filename_from_prompt(line) # a regex could be used
            answer = yes_or_no_from_subproject.get(filename, 'n') # a dict
        else:
            continue # skip it
        print(answer, file=p.stdin) # provide answer
        p.stdin.flush()

To test that you can read something from the xx using subprocess:

from subprocess import Popen, PIPE, STDOUT

with Popen(["xx", "viewproject"], bufsize=0,
           stdin=PIPE, stdout=PIPE, stderr=STDOUT) as p:
    print(repr(p.stdout.read(1)))

Answer 2

Yes, first of all you may create subprocess as an object by:

p = subprocess.Popen('xx viewproject', shell=True, stdin=subprocess.PIPE, 
                      stdout=subprocess.PIPE, universal_newlines=True)

Then you'll have methods like communicate() available, for instance:

newline = os.linesep # [1]
commands = ['y', 'n', 'y', 'n', 'y']
p.communicate( newline.join( commands))

^{1 - os.linesep}

Which will send all the answers at once (and hopefully it'll be enough) relying on the same order of question every time.

You may also try parsing p.stdout and then writing to p.stdin, but this may cause deadlock when one buffer will get full while waiting for another, so be careful with this. Luckily there are some complex examples on google.

Simple version would be:

p = Popen(...)
line = p.stdout.readline() # At this point, if child process will wait for stdin
                           # you have a deadlock on your hands
parse_line( line)
p.stdin.write( newline.join( commands).encode( 'utf-8'))

I would also consider rewriting:

p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell=True, 
                      stdin=subprocess.PIPE, stdout=subprocess.PIPE) 

To:

p = subprocess.Popen( ['si', 'viewproject', '--project=d:/Projects/test.pj'],
                      shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)

Unless you explicitly need Shell invocation.