0

I am trying to read a live sensor stream using subprocess in python. In particular, subprocess runs a command line program that outputs the data. This data is input into STDOUT and then read line by line. here is the code:

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=0)

while True:
    line_out =  proc.stdout.readline()
    sys.stdout.flush()

This code produces sensor data output, but incurs a time lag as it runs. So initially the sensor and subprocess are in sync but within an hour, the subprocess is as much as 45 minutes behind.

As you can see I tried "bufsize=0". I also tried running python with the -u option. Pexpect is not available to me since I am running Windows 8 and python 2.7.

How can I get streaming output from subprocess that does not incur a time delay?

MadProgrammer
  • 373
  • 2
  • 17
  • Unbuffered is expensive - every character is read/written independently. For your use, consider bufsize=1 [line buffered]. This would mean you get buffering that matches your example usage, which is far more efficient... – F1Rumors Jan 25 '16 at 18:33
  • Are you sure [`cmd` flushes its output in time](http://stackoverflow.com/q/20503671/4279)? Why do you use `sys.stdout.flush()` here? (It is pointless. There is no code that prints to stdout in your example). Where do you pass `-u` option? Why do you think it should have any effect on `cmd`? Is `cmd` a Python script? If it is a time performance issue; [try `bufsize=1` (line-buffered mode)](http://stackoverflow.com/a/17698359/4279). – jfs Jan 26 '16 at 10:03
  • The -u option is passed when you start python: python -u script.py – MadProgrammer Jan 26 '16 at 14:51
  • Since the -u parameter specifies that stdout and stdin are unbuffered it will absolutely have an effect on a subprocess that is called. The subprocess is an executable, but this doesnt make a difference in the discussion here. – MadProgrammer Jan 26 '16 at 14:52

1 Answers1

0

Please refer to this SO query that should address your problem. Based on the way you are creating the subprocess; you are using un-buffered input. This has poor performance not only in Python but in another runtime as well.

Set the bufsize to -1 - this will make sure that the full buffering (system default mode) is used to read your input.

Please check the performance after making this change and let us know.

Community
  • 1
  • 1
Prahalad Deshpande
  • 3,993
  • 17
  • 20
  • Thanks for the detailed answer. In the end, I used -1 to enable full buffering and dropped the -u parameter when calling the script. It looks like my intake is not lagging anymore. I will keep testing this and let you know if it incurs a delay over time. – MadProgrammer Jan 26 '16 at 14:53