Python, Memory Error in making dataframe

Question

When I use pandas DataFrame, occuring the Memory Error.

data's row is 200000 and column is 30.(type: list) fieldnames1 has columns name.(type:list)

Error occured in:

df = pd.DataFrame(data,columns=[fieldnames1])

what should I do? (python version 2.7 32bit)

You ran out of RAM. Try to use less data for testing and a combination of more RAM, a 64bit OS and 64bit Python. — Klaus D., Jul 06 '15 at 09:03
Any reason you don't use 64-bit version of python? Also storing lists in a df is a bit strange — EdChum, Jul 06 '15 at 09:13

score 3 · Answer 1 · edited May 23 '17 at 10:30

3

As indicated by Klaus, you're running out of memory. The problem occurs when you try to pull the entire text to memory in one go.

As pointed out in this post by Wes McKinney, "a solution is to read the file in smaller pieces (use iterator=True, chunksize=1000) then concatenate then with pd.concat".

edited May 23 '17 at 10:30

Community

1
1

answered Jul 06 '15 at 10:11

Julien Marrec

9,670
4
36
57

score -1 · Answer 2 · answered Dec 26 '16 at 07:26

-1

You can try this line of code:

data=pd.DataFrame.from_csv("train.csv")

This is an alternate of read.csv but it returns Data frame object without giving any memory error P.S the size of the training data is around 73 mb

answered Dec 26 '16 at 07:26

Javeria

41
1
7

Python, Memory Error in making dataframe

2 Answers2

Linked