Out of memory error in java with using HtmlUnit

Question

I built a web bot using htmlUnit and java which does a bunch of stuff but after running for about 10 minutes I always get the following error.

The program processes a few text file but the error below seems like the issue has something to do with HtmlUnit parsing javascript.

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at net.sourceforge.htmlunit.corejs.javascript.Parser.forLoop(Parser.java:1297)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:982)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statements(Parser.java:886)
at net.sourceforge.htmlunit.corejs.javascript.Parser.block(Parser.java:1699)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1031)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)
at net.sourceforge.htmlunit.corejs.javascript.Parser.ifStatement(Parser.java:1102)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:970)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)
at net.sourceforge.htmlunit.corejs.javascript.Parser.parseFunctionBody(Parser.java:662)
at net.sourceforge.htmlunit.corejs.javascript.Parser.function(Parser.java:827)
at net.sourceforge.htmlunit.corejs.javascript.Parser.primaryExpr(Parser.java:2746)
at net.sourceforge.htmlunit.corejs.javascript.Parser.memberExpr(Parser.java:2388)
at net.sourceforge.htmlunit.corejs.javascript.Parser.unaryExpr(Parser.java:2292)
at net.sourceforge.htmlunit.corejs.javascript.Parser.mulExpr(Parser.java:2219)
at net.sourceforge.htmlunit.corejs.javascript.Parser.addExpr(Parser.java:2205)
at net.sourceforge.htmlunit.corejs.javascript.Parser.shiftExpr(Parser.java:2188)
at net.sourceforge.htmlunit.corejs.javascript.Parser.relExpr(Parser.java:2165)
at net.sourceforge.htmlunit.corejs.javascript.Parser.eqExpr(Parser.java:2139)
at net.sourceforge.htmlunit.corejs.javascript.Parser.bitAndExpr(Parser.java:2130)
at net.sourceforge.htmlunit.corejs.javascript.Parser.bitXorExpr(Parser.java:2121)
at net.sourceforge.htmlunit.corejs.javascript.Parser.bitOrExpr(Parser.java:2112)
at net.sourceforge.htmlunit.corejs.javascript.Parser.andExpr(Parser.java:2103)
at net.sourceforge.htmlunit.corejs.javascript.Parser.orExpr(Parser.java:2094)
at net.sourceforge.htmlunit.corejs.javascript.Parser.condExpr(Parser.java:2060)
at net.sourceforge.htmlunit.corejs.javascript.Parser.assignExpr(Parser.java:2033)
at net.sourceforge.htmlunit.corejs.javascript.Parser.assignExpr(Parser.java:2044)
at net.sourceforge.htmlunit.corejs.javascript.Parser.expr(Parser.java:2023)
at net.sourceforge.htmlunit.corejs.javascript.Parser.nameOrLabel(Parser.java:1774)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1053)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)

I get a ton of errors anytime I load a javascript page with HtmlUnit, for example:

net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Expected argument of type object, but instead had type object (http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js#4)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3935)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3919)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3944)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError1(ScriptRuntime.java:3954)
at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.ensureScriptableObject(ScriptableObject.java:2193)
at net.sourceforge.htmlunit.corejs.javascript.NativeObject.execIdCall(NativeObject.java:303)
at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:93)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1536)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:5)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:3)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:3)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3286)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:827)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:939)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:836)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:812)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:997)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:399)
at com.gargoylesoftware.htmlunit.html.HtmlScript$3.execute(HtmlScript.java:277)
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:293)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:799)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:756)
at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1039)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:252)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:198)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:271)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:159)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:478)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:352)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:417)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:402)
at WebBotServicesAdID.replyToAd(WebBotServicesAdID.java:269)
at WebBotServicesAdID.main(WebBotServicesAdID.java:771)
======= EXCEPTION END ========

However these errors seem to be unavoidable and they don't cause the program to stop. The page still gets loaded and I can still do what I need to do so I have always ignored them.

Is this OutOfMemoryError the result of an accumulation of those errors I ignored?

Answer 1

GC Error:

GC stands for Garbage Collector and the error means that the GC is working hard and doing almost nothing. Such error was already discussed here and here, for example. You can try to increase the heap ("-Xmx1024m", "-Xmx512m", etc) or disable the error verification ("-XX:-UseGCOverheadLimit"). I don't think you should use either of those options.

If you're running linux, you can try to use jmap to check the live data in the heap and then optimize your code accordingly.

JS Error:

According to this answer those errors seem to be caused by script errors. Try the given suggestion to disable them.

If those errors are thrown and are not properly handled some objects may be left in the heap causing the GC issue. Or you may have some piece of code (a loop for example) instantiating more objects than it should.