0

after searching the web for a while I decided to ask you for help with my problem.

My program should analyze logfiles, which are really big. They are about 100mb up to 2gb. I want to read the files using NIO-classes like FileChannel.

I don't want to save the files in memory, but I want to process the lines immediately. The code works.

Now my problem: I analyzed the Memory usage with the Eclipse MAT plugin and it says about 18mb of data is saved (that fits). But TaskManager in Windows says that about 180mb are used by the JVM.

Can you tell me WHY this is?

I don't want to save the data reading with the FileChannel, i just want to process it. I am closing the Channel afterwards - I thought every data would be deleted then?

I hope you guys can help me with the difference between the used space is shown in MAT and the used space is shown in TaskManager.

user2090321
  • 15
  • 3

1 Answers1

1

MAT will only show objects that are actively references by your program. The JVM uses more memory than that:

  • Its own code
  • Non-object data (classes, compiled bytecode e.t.c.)
  • Heap space that is not currently in use, but has already been allocated.

The last case is probably the most major one. Depending on how much physical memory there is on a computer, the JVM will set a default maximum size for its heap. To improve performance it will keep using up to that amount of memory with minimal garbage collection activity. That means that objects that are no longer referenced will remain in memory, rather than be garbage collected immediately, thus increasing the total amount of memory used.

As a result, the JVM will generally not free any memory it has allocated as part of its heap back to the system. This will show-up as an inordinate amount of used memory in the OS monitoring utilities.

Applications with high object allocation/de-allocation rates will be worse - I have an application that uses 1.8GB of memory, while actually requiring less than 100MB. Reducing the maximum heap size to 120 MB, though, increases the execution time by almost a full order of magnitude.

Community
  • 1
  • 1
thkala
  • 76,870
  • 21
  • 145
  • 185
  • Thank you for your fast answer. So it does not really matter if how many space is used in TaskManager (in case I have enough memory)? The JVM will run the Garbage Collector if the program is about running out of memory? – user2090321 Feb 20 '13 at 09:33
  • Okay thank you. I tried to launch the application with -Xmx50m argument and the space in the TaskMgr was reduced. That answered my question! – user2090321 Feb 20 '13 at 09:43