JAVA For Loop running out of memory when running though image files

Question

I am loading about 100 FITS images of 3000x2000 pixels and then converting the pixel values for each image into a matrix and rescaling the matrix and adding each one to a SET.

But I am quickly running out of memory in my heap. The Set of int matrices shouldn't take too much memory should it? (The heap is about 2GB I believe, at least eclipse uses 2GB before giving error.)

So my thinking is that each new Fits object is being stored in memory after each loop. but it isn't needed again after the loop finishes so I don't know why that would be the case.

Is there a way to do it without running out of memory? There would be a different number of Fits files each time the program is run.

Public Set<int[][]> rescaleFitsList(File[] fitsFileList){

    Set<int[][]> rescaledFitsSet = new HashSet();

     for(File fits: fitsFileList){
         Fits f = new Fits(fits);
         double bScale = f.getHDU(0).getBScale;
         double bLinear = f.getHDU(0).getBLinear;
         short[][] counts = (short[][])f.getHDU(0).getKernel();
         int[][] rescaledFits = new int[counts.length][counts[0].length];

            for(int i =0, i<counts.length, i++){
                for(int j =0, j<counts[0].length, j++){
                    rescaledFits[i][j] = (int)(bScale * counts[i][j] + bLinear); 
                 }
             }
        rescaledFitsSet.add(rescaledFits);
      }

}

What matters in this case: how you invoke the JVM that runs this code. You have to pay close attention to such details. — GhostCat, Aug 18 '17 at 09:00
'Try doing all logic one by one. Then you shouldn't hit memory limit. (First do the counting and then transform one by one.) — John Tribe, Aug 18 '17 at 09:16
Did you set `-Xmx` in your launch/run configuration: in the _Run > Run Configurations...: Arguments_ tab: _VM arguments_ field? — howlger, Aug 18 '17 at 10:12
@JohnTribe Not quite sure how to do that. I could try saving each matrix to a .csv file instead of storing to a Set. Not sure if that will help me though. — Ray, Aug 18 '17 at 15:23
@Ray It depends on the `-Xmx` default (see: https://stackoverflow.com/q/28272923/6505250). A Java application that is launched from inside of Eclipse has its own JRE. — howlger, Aug 18 '17 at 15:33
Hint: when you do not understand something then try to do some research. A person trying to pull up such features should really know a bit about such important details :-) — GhostCat, Aug 18 '17 at 16:14
Try working not on files[] but string[] representing path to file and in foreach do new File(path) — John Tribe, Aug 19 '17 at 08:08

score -1 · Accepted Answer · answered Aug 18 '17 at 21:32

An application that is started within Eclipse has its own JRE (otherwise a System.exit(0) would kill also the Eclipse IDE) with its own VM settings (other than that of the Eclipse IDE). So make sure you set -Xmx in your launch/run configuration: Run > Run Configurations...: Arguments tab: VM arguments field.

100 images of 3000x2000 with one int (4 bytes) per pixel = 100 x 3,000 x 2,000 x 4 bytes = 2,400,000,000 bytes ≈ 2.2 GB. 2 GB are probably not enough memory, but with 3 GB it should work: -Xms3g -Xmx3g

JAVA For Loop running out of memory when running though image files

1 Answers1