How to deal with 'Memory Quota Vastly Exceeded' errors in Flask and Heroku when loading in postgreSQL databases into dataframes?

Question

So, basically, I have a website that will be used by people to modify filters and then click 'download' and the resulting Excel file will have the data (specified by their filters). There are about 125,000+ data-points in my postgreSQL database, and I currently have it being loaded in the background using a simple

df = pd.read_sql_query('select * from real_final_sale_data', con = engine)

The only problem is that this quickly overwhelms Heroku's memory allowance on 1 dyno (512 MB), but I do not understand why this is happening or what the solution is.

For instance, when I run this on my computer and do 'df.info()' it shows that it's only using about 30 MB of space, so how come when I read it, it suddenly is sucking up so much MB?

Thank you so much for your time!

Does specifying a `chunksize` help at all? https://stackoverflow.com/a/29522443/13808319 Be sure to read the comment under the answer specific to PostgreSQL. — Mike Organek, Jul 31 '20 at 18:34

score 0 · Answer 1 · answered Aug 01 '20 at 00:10

So, the solution that ended up working was to just use some of the filters as queries to SQL. I.e., I had been just doing a select * without filtering anything from SQL, so my database which has like 120,000 rows and 30 columns caused a bit of strain on Heroku's dyno so it's definitely recommended to either use chunking or do some filtering when querying the DB.

How to deal with 'Memory Quota Vastly Exceeded' errors in Flask and Heroku when loading in postgreSQL databases into dataframes?

1 Answers1