OSError 24 (Too many open files) when reading bunch of FITS with astropy.io

Question

I’m trying to load into memory a few 2 000 FITS using astropy.io.fits:

def readfits(filename):
    with fits.open(filename) as ft:
        # the fits contain a single HDU
        data = ft[0].data
    return data

data_sci = []
for i in range(2000):
    data_sci.append(readfits("filename_{}.fits".format(i)))

However, when reaching the 1015th file, OSError: [Errno 24] Too many open files is raised.

I have the same issue with:

def readfits(filename):
    ft = fits.open(filename) as ft:
    data = ft[0].data
    ft.close()
    return data

I suspect that astropy.io.fits does not properly close the file. Is there a way I can force the files to be closed?

score 3 · Answer 1 · answered Oct 05 '15 at 15:44

3

After taking a look at the astropy documentation i found this: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

answered Oct 05 '15 at 15:44

bkaf

426
6
16

1

Thanks, this was the info I was searching for! The solution I used was a bit different though: as I needed to keep the data in memory, I couldn’t blindly use `del`. I simply replaced the line `data = ft[0].data` with `data = ft[0].data.copy()` instead. – Arcturus B Oct 05 '15 at 16:02
1

If you do `data.copy()` then you're loading the entire data into memory anyways, so you might as well just open the files with `memmap=False` and then there's no issue. Also if the data consists of tables there was an issue, now fixed, that resulted in a resource leak. – Iguananaut Oct 06 '15 at 15:07
@Iguananaut, definitely buying that `memmap=False`. I should have read more carefully the doc of `open`… – Arcturus B Oct 06 '15 at 22:38
It's a tricky thing; easy to miss. Opening lots of files simultaneously is not the most common case, but it's common enough that there should probably be a better interface for that. – Iguananaut Oct 07 '15 at 03:56

score 3 · Accepted Answer · answered Oct 06 '15 at 15:10

Your readfits function actually leaves the file handle open in order to keep access to the data, because by default it creates a mmap to the data and does not read it entirely into physical memory, as explained: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

Incidentally, if you just want a function that reads the data out of the first HDU this is already built in: http://docs.astropy.org/en/v1.0.5/io/fits/api/files.html#astropy.io.fits.getdata

It's not necessary to reinvent the wheel.

score 0 · Answer 3 · edited May 23 '17 at 11:46

0

You can call this function and store its output as long as you have memory. I thought it worths mentioning the answer explicitly but the credit goes to Iguananaut, bkaf, and this page.

def get_single_fits_data(fits_dir):

    hdul  = fits.open(fits_dir)
    for hdu in hdul:
        image_data = hdu.data.copy()
    hdul.close()
    gc.collect()

    return image_data

edited May 23 '17 at 11:46

Community

1
1

answered Mar 23 '17 at 03:57

Miladiouss

2,451
19
25

OSError 24 (Too many open files) when reading bunch of FITS with astropy.io

3 Answers3

Linked