Using textfile as stdin in python under windows 7

Question

I'm a win7-user.

I accidentally read about redirections (like command1 < infile > outfile) in *nix systems, and then I discovered that something similar can be done in Windows (link). And python is also can do something like this with pipes(?) or stdin/stdout(?).

I do not understand how this happens in Windows, so I have a question.

I use some kind of proprietary windows-program (.exe). This program is able to append data to a file. For simplicity, let's assume that it is the equivalent of something like

while True:
    f = open('textfile.txt','a')
    f.write(repr(ctime()) + '\n')
    f.close()
    sleep(100)

The question: Can I use this file (textfile.txt) as stdin? I mean that the script (while it runs) should always (not once) handle all new data, ie

In the "never-ending cycle":

• The program (.exe) writes something.

• Python script captures the data and processes.

Could you please write how to do this in python, or maybe in win cmd/.bat or somehow else.

This is insanely cool thing. I want to learn how to do it! :D

Answer 1

If I am reading your question correctly then you want to pipe output from one command to another.

This is normally done as such:

cmd1 | cmd2

However, you say that your program only writes to files. I would double check the documentation to see if their isn't a way to get the command to write to stdout instead of a file.

If this is not possible then you can create what is known as a named pipe. It appears as a file on your filesystem, but is really just a buffer of data that can be written to and read from (the data is a stream and can only be read once). Meaning your program reading it will not finish until the program writing to the pipe stops writing and closes the "file". I don't have experience with named pipes on windows so you'll need to ask a new question for that. One down side of pipes is that they have a limited buffer size. So if there isn't a program reading data from the pipe then once the buffer is full the writing program won't be able to continue and just wait indefinitely until a program starts reading from the pipe.

An alternative is that on Unix there is a program called tail which can be set up to continuously monitor a file for changes and output any data as it is appended to the file (with a short delay.

tail --follow=textfile.txt --retry | mycmd 
# wait for data to be appended to the file and output new data to mycmd

cmd1 >> textfile.txt # append output to file

One thing to note about this is that tail won't stop just because the first command has stopped writing to the file. tail will continue to listen to changes on that file forever or until mycmd stops listening to tail, or until tail is killed (or "sigint-ed").

This question has various answers on how to get a version of tail onto a windows machine.

Answer 2

import sys
sys.stdin = open('textfile.txt', 'r')
for line in sys.stdin:
    process(line)

Answer 3

If the program writes to textfile.txt, you can't change that to redirect to stdin of your Python script unless you recompile the program to do so.

If you were to edit the program, you'd need to make it write to stdout, rather than a file on the filesystem. That way you can use the redirection operators to feed it into your Python script (in your case the | operator).

Assuming you can't do that, you could write a program that polls for changes on the text file, and consumes only the newly written data, by keeping track of how much it read the last time it was updated.

Answer 4

When you use < to direct the output of a file to a python script, that script receives that data on it's stdin stream.

Simply read from sys.stdin to get that data:

import sys
for line in sys.stdin:
    # do something with line