Memory Error : Training large dataset

Question

memory error

My dataset have 70k images which i want to train through Conv2D but it is throwing memory error when i tried to load the dataset. I just have 4GB RAM, how i can resolve this issue through HDF5 matrix by creating a dataset in HDF5? and then loading it to train, i guess it will take less memory then. I tried some tutorial to create HDF5 dataset but this processes comes after where the error is occurring. What i am doing wrong? Please ask if question is not clear.

datagen=ImageDataGenerator(rotation_range=40,
                            width_shift_range=0.2,
                            height_shift_range=0.2,
                          rescale=1./255,
                          shear_range=0.2,
                          zoom_range=0.2,
                          horizontal_flip=True)

batch_size=28
num_classes=37
epochs=100

os.chdir("E:");
path="Dataset/One";
classes=os.listdir(path)
x=[]#Datapoints 
y=[]#labels 
for fol in classes:
    imgfiles=os.listdir(path+u'\\'+fol);
    for img in imgfiles:
        im=Image.open(path+u'\\'+fol+u'\\'+img);
        im=numpy.asarray(im)/255;
        x.append(im)
        y.append(fol)
x=numpy.array(x)
y=numpy.array(y)
#memory error####################################################
x=x.reshape((-1,100,100,1))

n=x.shape[0]
randomize=numpy.arange(n)
numpy.random.shuffle(randomize)
randomize
x=x[randomize]
y=y[randomize]

How large is the total size of the 70k images? – John Zwinck Mar 07 '18 at 12:42 — John Zwinck, Mar 07 '18 at 12:42

score 4 · Accepted Answer · answered Mar 07 '18 at 12:50

4

Your problem is that you try to load all the data at once, and it is much larger than your RAM. You need to load just one batch and process it, then discard that batch and move on. A natural way to do this might be inside the for fol in classes loop--treat each fol value as one batch, and fit one batch at a time.

answered Mar 07 '18 at 12:50

John Zwinck

207,363
31
261
371

but i am splitting my dataset after loading all of it if i just take one folder how its going to split – user9456630 Mar 07 '18 at 12:55
The problem is you load 100% of the files before processing any of them. You should load 1%, process them, then load the next 1%.... – John Zwinck Mar 07 '18 at 12:59
how i can load each image or batch of dataset can you show me some simple steps – user9456630 Mar 08 '18 at 16:19

score 0 · Answer 2 · answered Apr 30 '18 at 05:49

If you don't need to access or process all of the data at once then you can load it in chunks.

If it's a csv file and if you can use pandas then maybe you can do it like this:

import pandas as pd
for chunk in pd.read_csv('dataset/movies.csv', chunksize=1000):
    # use this chunk for processing and/or training

Hope it helps!

Memory Error : Training large dataset

2 Answers2