Reading multiple files in real time?

Question

I'm trying to listen log files that are constantly updated and work with the lines continuously. The thing is that I have multiple files to listen. The logs are separated by the jboss instances and I have to work with all them together to insert them on a database.

I've got a good example of how to read a file continuously from the question 5419888, but this code only reads one file by time. I've tried the following code to read them all, but it only listen to the first file it finds in the array of files.

How could I multithread this to process all the files at the same time?

import time
from glob import glob

def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line



if __name__ == '__main__':
    for log in glob("/logs/xxx/production/jboss/yyy*/xxx-production-zzzz*/xxx-production-zzzz*-xxx-Metrics.log"):
        logfile = open(log, "r")
        loglines = follow(logfile)

        for line in loglines:
            print line,

http://stackoverflow.com/documentation/python/544/multithreading#t=201607261842139282503 — Aran-Fey, Jul 26 '16 at 18:43
Maybe what you want, is to create a thread for each file and set as target function the follow() function you have defined. — theVoid, Jul 26 '16 at 18:53
@Rawing I've tried to apply this on my situation but I couldn't understand it very well. But thanks. — davis, Jul 26 '16 at 20:52
@theVoid That's pretty much the reply above and it worked for me. Thanks for the assistance! — davis, Jul 26 '16 at 20:53

score 3 · Accepted Answer · answered Jul 26 '16 at 19:11

You can print the lines of each file at the same time using the following code:

lock = threading.Lock()

def printFile(logfile):
    loglines = follow(logfile)
    for line in loglines:
        #only one thread at a time can print to the user
        lock.acquire()
        print line
        lock.release()



if __name__ == '__main__':
    for log in glob("/logs/xxx/production/jboss/yyy*/xxx-production-zzzz*/xxx-production-zzzz*-xxx-Metrics.log"):
        logfile = open(log, "r")
        t = threading.Thread(target = printFile,args = (logfile,))
        t.start()

Reading multiple files in real time?

1 Answers1