Write command stdout to console and file (live) -- Python 3.5 + subprocess.run()

Question

Platform: Python 3.5 on rhel6 (64-bit)

Scenario: Execute a Bash command which runs a job. This job returns several lines of output to stdout, every few seconds.

Command: ./run_job --name 'myjob' --config_file ./myconfig.conf

Goal: Using Python's subprocess.run(), I am trying to run the above command and capture the stdout of the process, print it to the console and also save it to a file. I need the stdout to be printed as it becomes available (live).

What I've Tried: I have searched extensively for this, and every solution I found was using subprocess.Popen(). This method somewhat worked, but implementing it resulted in breaking the return logic I currently have. Reading through the Python documentation, the subprocess.run() method is the recommended way as of Python 3.5, so that's why I am going this route.

My Setup: So far, I have one common file with the logging and running the shell command below.

def setup_logging(log_lvl="INFO"):
    script_name = path.splitext(path.basename(__file__))[0]
    log_path = environ["HOME"] + "/logs/" + script_name + ".log"

    logging.basicConfig(
        level=getattr(logging, log_lvl.upper()),
        format="%(asctime)s: [%(levelname)s] %(message)s",
        handlers=[
            logging.FileHandler(filename=log_path, mode="w", encoding="utf-8"),
            logging.StreamHandler()
        ]
    )

def run_shell(cmd_str, print_stdout=True, fail_msg=""):
    logger = logging.getLogger()
    result = run(cmd_str, universal_newlines=True, shell=True, stderr=STDOUT, stdout=PIPE)

    cmd_stdout = result.stdout.strip()
    cmd_code = result.returncode
    if print_stdout and cmd_stdout != "":
        logger.info("[OUT] " + cmd_stdout)
    if cmd_code != 0 and fail_msg != "":
        logger.error(fail_msg)
        exit(cmd_code)
    return cmd_code, cmd_stdout

So I would use the following code to run my script:

run_shell("./run_job --name 'myjob' --config_file ./myconfig.conf", fail_msg="Job failed.")

This partially works, but the full stdout is printed only when the process has completed. So the terminal will hang until that happens. I need to print the stdout line by line, in a live manner, so that it can be written by the logger.

Is there any way to do this without overhauling my existing code? Any help would be appreciated.

I recently worked with another SO user to interact with another subprocess: https://stackoverflow.com/a/60680949/7915759 The example can be applied in this scenario. You can read back stdout from the python script. — Todd, Mar 19 '20 at 21:17
In that example, the thread could be modified to write every line it captures to a file, or you can do that in the main loop. Or just make the main loop the same algo as the thread and get rid of the thread and queue. — Todd, Mar 19 '20 at 21:18
With `subprocess.run()`, I believe you have to wait until it completes and get its output all at once. With the solution I linked, you can capture the output as it comes in. So what you describe as your live capture requirements won't be achievable with `run()`. — Todd, Mar 19 '20 at 21:32
@Todd: Thanks for the link! I will take a look. I also had a feeling that it was not achievable with ```subprocess.run()```. If I can get it to work with what you provided, I will post back here. — Tiki, Mar 20 '20 at 17:56

score 0 · Accepted Answer · answered Mar 23 '20 at 16:23

After trying for some time using threads and polling, I couldn't quite get it to work. However, from that same thread linked by Todd, I found another way to achieve what I need by using Popen:

def run_shell(cmd_str, print_stdout=True, fail_msg=""):
    """Run a Linux shell command and return the result code."""

    p = Popen(cmd_str,
              shell=True,
              stderr=STDOUT,
              stdout=PIPE,
              bufsize=1,
              universal_newlines=True)

    logger = logging.getLogger()

    output_lines = list()
    while True:
        output = p.stdout.readline().strip()
        if len(output) == 0 and p.poll() is not None:
            break
        output_lines.append(output)
        if print_stdout:
            logger.info("[cmd] " + output)

    cmd_code = p.returncode
    cmd_stdout = "\n".join(output_lines)
    if cmd_code != 0 and fail_msg != "":
        logger.error(fail_msg)
        exit(-1)

    return cmd_code, cmd_stdout

What are your dependencies? Did you import things in unusual ways? For example it should have been subprocess.Popen. Also afterwards it says STDOUT is not defined. — Kvothe, Feb 24 '21 at 14:50

Write command stdout to console and file (live) -- Python 3.5 + subprocess.run()

1 Answers1