How do I understand the reason for frequent garbage collection?

Question

jstat -gc 27539
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   
901632.0 468480.0  0.0    0.0   911360.0 911360.0 5464064.0  5463748.3  21632.0 20948.0 2944.0 2777.7    153   33.727  401   782.598  816.325

jstat -gccapacity 27539
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC       MCMN     MCMX      MC     CCSMN    CCSMX     CCSC    YGC    FGC 
171008.0 2732032.0 2714624.0 901632.0 468480.0 911360.0   343040.0  5464064.0  5464064.0  5464064.0      0.0 1069056.0  21632.0      0.0 1048576.0   2944.0    153   404

I added EU and OU to find the total heap used. That looks 6GB is used. I referred this

But there are 400+ FGC happened. It has reached 700+ now. After sometime, it just performs GC. It is 850+ now.

My job:

It is multi threading. 100 reader, 100 writer threads. Each one has it's own connection to the database. Each reader thread reads 100000 records and stores in a LinkedList and sends to writer thread. Writer writes the data to another collection in the same database. LinkedList is not reused means each 1L creates a new LinkedList.

It is akka based multi-threading. So I don't handle thread failure, thread spawning i.e thread management.

But my doubt is that why such huge FGC happening when I have 32gb ram? any pointers to look further?

It ran into GC Overhead LIMIt exceeded error sometimes.

I didn't set any explicit min, max memory for the job.

EDIT:

As per my analysis, it has fixed some EU and OU. It is full, hence it is keep on performing GC. Is it possible and am I correct?

Edit2

Thanks @emotionlessbanans, @Cascader. I have the below.

    uintx ErgoHeapSizeLimit                         = 0                                   {product}
    uintx HeapSizePerGCThread                       = 87241520                            {product}
    uintx InitialHeapSize                          := 526385152                           {product}
    uintx LargePageHeapSizeThreshold                = 134217728                           {product}
    uintx MaxHeapSize                              := 8392802304                          {product}
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Any specific reason to stop at only 6GB when I have 8gb ? Or am I missing something?

I am not sure if this will help, but have you tried using jconsole. It will provide you great help if you will check into thread section — emotionlessbananas, Aug 06 '20 at 07:18
"I didn't set any explicit min, max memory for the job." That means you didn't specify `-Xmx` when running your JVM? Could it be that your JVM uses a small fragment of your 32Gb of RAM? — Tasos P., Aug 06 '20 at 07:23
@Cascader Yes, I suspect that. When 24Gb is free, will it not use as per the need? But after sometime, why does job just performs only GC. it has performed 500+ GC without any actual operation. Is there any max GC limit per job? — Gibbs, Aug 06 '20 at 07:24
use -Xmx and -Xms to set jvm max and min memory, or it will run with defaults — zlaval, Aug 06 '20 at 07:26

score 3 · Accepted Answer · answered Aug 06 '20 at 07:33

3

When no max heap size is specified using -Xmx switch, the JVM chooses a default value. This value is JVM vendor specific and it usually depends on architecture (32/64bit) and total available memory.

It seems like your application uses all allocated memory (the one determined by your JVM) and from some point on it spends too much time on GC until you get "GC Overhead Limit Exceeded" error.

You should explicitly set a maximum heap size (i.e. -Xmx 10g) in order to be sure you utilize all of your available memory.

answered Aug 06 '20 at 07:33

Tasos P.

2,856
1
18
32

Sure, I am trying that. How do I find how much memory allocated by JVM for the job? Is it `EU,OU` or something? – Gibbs Aug 06 '20 at 07:35
2

https://stackoverflow.com/questions/4667483/how-is-the-default-max-java-heap-size-determined – emotionlessbananas Aug 06 '20 at 07:36
Thanks, I tried that earlier. Does `MaxHeapSize` is the max one, `initialHeapSize` is the one min? Am I correct? I am kind of sure about this. My max is `8392802304`. Why job stuck at only 6? +1 from my side for your efforts – Gibbs Aug 06 '20 at 07:38
1

@Gibbs you can try with jconsole to check visually how your threads and heap are working and also there is option to perform GC manually but won't recommend performing GC manually on prod – emotionlessbananas Aug 06 '20 at 07:47
Both Eden (EU/EC) and Old Gen (OU/OC) usage ratios are ~100%. This would definitely put some pressure on GC threads – Tasos P. Aug 06 '20 at 07:56
@Cascader I have a doubt. When I set this, GC reduced. I still wonder why EU/OU was 6g. But GC was more though no another process was running on that server. When I set xmx then also EU/OU values are similar but GC reduced. – Gibbs Aug 07 '20 at 03:08

score 0 · Answer 2 · answered Aug 06 '20 at 07:23

0

Default min memory is not specified and max is 256Mb. It may be problem with not sufficient memory assigned to heap. (It is my suspicion)

Also you could use VisualVM or any other tool that might show more info about what happens there.

answered Aug 06 '20 at 07:23

Filip_Lechowicz

36
4

When I sum `EU,OU`, it has used 6GB. How is that even possible if 256MB is allowed case? Is it per thread? Thanks about `VisualVM`, I have only linux console, have to setup some display to it. – Gibbs Aug 06 '20 at 07:27
Yup, you are right @matt I looked up oracle docs for [java 8](https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gc-ergonomics.html) and [java 11](https://docs.oracle.com/en/java/javase/11/gctuning/ergonomics.html#GUID-DA88B6A6-AF89-4423-95A6-BBCBD9FAE781) . Basicly it says `Initial heap size of 1/64 of physical memory` and `Maximum heap size of 1/4 of physical memory ` – Filip_Lechowicz Aug 06 '20 at 10:16

How do I understand the reason for frequent garbage collection?

2 Answers2

Linked