Subprocess hangs / delayed with popen communicate() or wait() (Python 2.7.9)

Question

After a lot of different tests, searching and trying different approaches by others, I am currently a bit lost here, so hopefully anyone can help me...

I am trying to spawn subprocesses (calling external powershell-scripts in my case) and I want to get stdout and the return code. In the past my code already worked under win32 with an older version of Python 2.7 but now on my new development machine (64Bit/Win7) I cannot get it running completely... :-/ (I think my old machine was 32Bit, if that may be an issue)

The strange thing is: Executing in my Eclipse Environment (4.4.1) the code basically works, but it just "hangs" after the stdout-lines. The output is "delayed" before 'DEBUG1' at the point where proc.communicate() is called (same happens with proc.wait()). (I don't see 'DEBUG1' or 'DEBUG2' but I see 'DEBUG0' and the logger.info(stdout) lines)

To explain this in more detail: If I just wait some time nothing happens after 'DEBUG0' and stdout-lines. BUT - if I press 'enter', then the output resumes, I see 'DEBUG1' and then about a second later 'suddenly' the subprocess is finished, all output is given ('DEBUG2' also) and everything is fine... (as it should have been already before)

I just do not understand how I can resolve this automatically when I am not there to hit the Enter-key... :-/

And as I said, that code already did work with my old development environment in the past, without hitting 'enter' of course ;) Thought maybe flushing would help, but that did not work as you can see.

Any ideas? Hints? Many thanks in advance!

Here is the deciding code snippet:

    try:
        proc = subprocess.Popen([powershell_bin,
                               '-ExecutionPolicy',
                               'Unrestricted',
                               '-NonInteractive',
                               '-NoProfile',
                               '-Command',
                               settings['ext_ps_commands'][self.cmd],
                               usr,
                               grp],
                               stdout=subprocess.PIPE)
    except Exception as e:
        warnmsg = "..."
        logger.warn(warnmsg)
        return -1

    print "DEBUG0"
    sys.stdout.flush()

    while True:
        line = proc.stdout.readline()
        if line != '':
            stdout = "  stdout:", line.rstrip()
            logger.info(stdout)
        else:
            break

    print "DEBUG1"
    sys.stdout.flush()

    exitcode = proc.wait()
    # stdout, stderr = proc.communicate()
    # exitcode = proc.returncode

    print "DEBUG2"
    sys.stdout.flush()

    return exitcode

Unrelated: don't use bare `except:` (e.g., you don't want to catch `KeyboardInterrupt` as a rule); you could use `except Exception as e: logger.error(...)` instead. Does it hang if you run your Python script from console (cmd.exe) instead of Eclipse? Have you tried to set `stdin=open(os.devnull, 'rb', 0)`? You could [use `for line in iter(proc.stdout.readline, b''):` instead of the `while`-loop](http://stackoverflow.com/a/17698359/4279). — jfs, Mar 25 '15 at 18:35

score 1 · Answer 1 · edited Feb 18 '17 at 16:42

@J.F. Sebastian: Many thanks for your ideas and suggestions (included the improved exception handling), actually you brought the solution: It is an Eclipse/PyDev issue. In the raw cmd.exe console the code works just fine! :) (both versions with .communicate() and .wait())

It is the first time I encountered a difference in behaviour between console and Eclipse, so I just did not think of that. Seems there is some bug or configuration issue in my Eclipse. It is Eclipse SDK Luna SR1 (4.4.1) and PyDev 3.9.1, probably I should test Eclipse 4.4.2 and PyDev 3.9.2.

Many thanks again!

Subprocess hangs / delayed with popen communicate() or wait() (Python 2.7.9)

1 Answers1