1

My nodejs program runs out of memory...I know you can set/change the allocated amount of memory in nodejs - but this is not enough... if i set it to 1,2,4 gb...

My program runs though 1,5, 10 or more millions iteration in a loop - where it saves to my db also... and but after a while it crashes...

I wonder why - the program actually crashes and not just runs slower / swap data to disk ect... like 'normal'...

I have tried to install and run a program in windows which helps GC / free up memory ... to see if it made any changes ... but it didnt...

How do i solve this? How do i get a better idea of what actually uses the amount of memory - so i can change the code ect?

Screenshot: http://screencast.com/t/Vucqa60F1

PabloDK
  • 1,501
  • 1
  • 7
  • 26
  • You might want to add a piece of code to show what you are doing, it'll be easier to see what is going on. This sounds like a memory leak problem and not a Node.js memory issue. – Juan Stiza Jun 07 '16 at 12:41
  • I realized its because i made like 5 million seperate inserts to the db... when i make 5.000 inserts with 1.000 objects at a time... no exception is thrown... But the execution time went up a bit...? What is the 'best practice' to insert millions of rows via Node to PostgreSQL? To make it all perform as fast as possible... i have coded the DB layer myself... so no overhead via fancy frameworks ect... – PabloDK Jun 07 '16 at 15:21
  • I'd say do it in batches, see how many inserts yo want to do in a single batch (I don't know how PSQL performs here) and split all your inserts in pages. With node you can even decide hown many of those inserts can be done at the same time. – Juan Stiza Jun 07 '16 at 15:31
  • I construct 5.000 statements... each of those contains 1.000 insert-statements inside (one string) ... which then is executed with all 1.000x17(column values) = 17.000 parameters...but it is WAAY to slow this way... 5 million single inserts are MUCH faster...but it makes my program crash because of too much memory usage...How would you do it? – PabloDK Jun 07 '16 at 16:25
  • Perhaps you should look for [this](http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql) and [this](http://stackoverflow.com/questions/758945/whats-the-fastest-way-to-do-a-bulk-insert-into-postgres) first. Speed will depend on how much data we are talking about. – Juan Stiza Jun 07 '16 at 17:59

0 Answers0