Opening files recursively in python from a textfile

Question

The first few lines of a python script I am writing look like this:

#!/usr/bin/env python

import os
import sys
import subprocess
import csv
import itertools

fin=open("filelist.txt",'r')
print(fin)
for line in fin:
        TARG_FILE = open('fin', 'r')
        spamreader = csv.reader(TARG_FILE, delimiter='\t')
        for line in spamreader:
                print(line[0:6])

Where filelist.txt is a text file in the same directory as the script. If I put fin in quotation marks, the function returns:

  File "VCFParser.py", line 13, in <module>
    TARG_FILE = open('fin', 'r')
IOError: [Errno 2] No such file or directory: 'fin'

but if I put in without quotation marks, I get:

Traceback (most recent call last):
  File "VCFParser.py", line 11, in <module>
    TARG_FILE = open(fin, 'r')
TypeError: coercing to Unicode: need string or buffer, file found

I suspect the problem is very similar to what is found here:

However, I am unable to get the script to work using the information there.

If I print(fin), it returns the appropriate filepath for reading:

print(fin)
16.31235000.31366000/vcfBeta-GS000012109-ASM.vcf.gz.summary.txt

What am I missing here?

Could it have anything to do with a newline at the end of every line in the filelist.txt file?

Answer 1

Your problem is that you are passing a file object into your open function, instead of a string from the file you just opened.

for line in fin:
    TARG_FILE = open('fin', 'r')

needs to be replaced with

for line in fin:
    TARG_FILE = open(line, 'r')

Your line variable references each line in filelist.txt, not fin.

Answer 2

The reasons for both errors are very clear.

Error: No such file or directory: 'fin'

You try to open a file named 'fin' and it is missing.

Error: ... need string or buffer, file found

To open you must provide file name. You are passing in file descriptor and this causes the error.

Solution: open really existing file name

In case, your "filelist.txt" has file names in lines, you probably want to open mofied content of the line:

#!/usr/bin/env python

import os
import sys
import subprocess
import csv
import itertools

with open("filelist.txt",'r') as fin:
    for line in fin:
        fname = line.strip("\n")
        with open(fname, "r") as TARG_FILE:
            spamreader = csv.reader(TARG_FILE, delimiter='\t')
            for line in spamreader:
                    print(line[0:6])

The `line.strip("\n") is needed to get rid of newline character, which is present in most of the lines (may be missing in the last one).

Lesson learned: read error messages carefully

Error messages are often explaining the reason very well.

1. start from the bottom
2. go step by step higher, until you reach the error from a statement in the code you have written
3. do your best to understand, what is the error message exactly telling.
4. if needed, try to imagine, how the situation happened logically connecting the bottom most error with the line of code you wrote.

If we get used to their a bit strange way of talking, we may find the problems much faster.