Error Executing javascript using HtmlUnit

Question

I am trying to execute a JavaScript on the following URL: http://steamcommunity.com/id/bobcatchris/inventory#730. I can do it using google chrome by opening console by pressing ctrl+shift+j and pasting the following script:

var list = []; 
var size = Object.keys(g_ActiveInventory.rgInventory).size();    
var counter = 0;
while (counter < size) {
 list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);
 counter +=1;
}

It returns: 150 Then in the next line, i write: >list and it returns an array of length 150.

When i try to do it in HtmlUnit as:

 public static void main(String[] args) throws IOException {
        WebClient webClient=new WebClient(BrowserVersion.FIREFOX_17);
        HtmlPage page=webClient.getPage("http://steamcommunity.com/id/bobcatchris/inventory#730");
        String script="var list = [];\n" +
                "\n" +
                "\n" +
                "var size = Object.keys(g_ActiveInventory.rgInventory).size();\n" +
                "\n" +
                "\n" +
                "\n" +
                "var counter = 0;\n" +
                "\n" +
                "while (counter < size) {\n" +
                " list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);\n" +
                " counter +=1;\n" +
                "}";
        Object result = page.executeJavaScript(script).getJavaScriptResult();
        System.out.println(result);
    }

I am getting the following exception:

Exception in thread "main" ======= EXCEPTION START ========
EcmaError: lineNumber=[4] column=[0] lineSource=[<no source>] name=[TypeError] sourceName=[injected script] message=[TypeError: Expected argument of type object, but instead had type object (injected script#4)]
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Expected argument of type object, but instead had type object (injected script#4)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:684)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:602)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:507)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:570)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:545)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptIfPossible(HtmlPage.java:959)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:927)
    at Scraper.main(Scraper.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Expected argument of type object, but instead had type object (injected script#4)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3603)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3587)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3608)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError1(ScriptRuntime.java:3618)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.ensureScriptable(ScriptableObject.java:2095)
    at net.sourceforge.htmlunit.corejs.javascript.NativeObject.execIdCall(NativeObject.java:287)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:89)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3031)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:561)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:669)
    ... 12 more
Enclosed exception: 
net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Expected argument of type object, but instead had type object (injected script#4)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3603)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3587)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3608)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError1(ScriptRuntime.java:3618)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.ensureScriptable(ScriptableObject.java:2095)
    at net.sourceforge.htmlunit.corejs.javascript.NativeObject.execIdCall(NativeObject.java:287)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:89)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531)
    at script(injected script:4)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3031)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:561)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:669)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:602)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:507)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:570)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:545)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptIfPossible(HtmlPage.java:959)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:927)
    at Scraper.main(Scraper.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
======= EXCEPTION END ========

The line causeing exception is:

Object result = page.executeJavaScript(script).getJavaScriptResult();

So how should I execute this script from HtmlUnit as I do from google chrome console?

@MostyMostacho could u help? – rahulserver Feb 04 '14 at 06:15 — rahulserver, Feb 04 '14 at 06:15

PopoFibo · Accepted Answer · 2014-02-04T08:17:29.987

These are your javascript script errors translated to Java exceptions by your framework.

That's because you have not explicitly set the setThrowExceptionOnScriptError option of your WebClient to false.

Unless there are absolutely no errors in the js, it is useful to always set this value as false unless it interferes with the results your are looking for.

Typically, these are my webclient settings when dealing with js and ajax through HtmlUnit.

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17,
                PROXY_HOST, PROXY_PORT);

WebRequest request = new WebRequest(new URL(
                "http://steamcommunity.com/id/bobcatchris/inventory#730"));

webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.setJavaScriptTimeout(10000);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getOptions().setTimeout(10000);

HtmlPage page = webClient.getPage(request);

          String script="var list = [];\n" +
                    "\n" +
                    "\n" +
                    "var size = Object.keys(g_ActiveInventory.rgInventory).size();\n" +
                    "\n" +
                    "\n" +
                    "\n" +
                    "var counter = 0;\n" +
                    "\n" +
                    "while (counter < size) {\n" +
                    " list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);\n" +
                    " counter +=1;\n" +
                    "}";
            Object result = page.executeJavaScript(script).getJavaScriptResult();
            System.out.println(result);

If I try your code, with the above settings I get 150.0 printed to the console which I assume is working as expected.

EDIT:

To traverse over the complete array list:

String script="var list = [];\n" +
                    "\n" +
                    "\n" +
                    "var size = Object.keys(g_ActiveInventory.rgInventory).size();\n" +
                    "\n" +
                    "\n" +
                    "\n" +
                    "var counter = 0;\n" +
                    "\n" +
                    "while (counter < size) {\n" +
                    " list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);\n" +
                    " counter +=1;\n" +
                    "}"
                    + "list";

            Object result = page.executeJavaScript(script).getJavaScriptResult();
            if (result instanceof NativeArray) {
                for (Object obj : (NativeArray)result) {
                    System.out.println(obj);    
                }

            }

Above, I have change the js to include list as the return parameter and iterate over NativeArray to get each element.

Output:

P2000 | Scorpion (Factory New)
AK-47 | Black Laminate (Field-Tested)
★ StatTrak™ Karambit | Case Hardened (Minimal Wear)
CS:GO Case Key
CS:GO Case Key

You can read much on their ajax settings at their FAQs - here.

Can u please paste your code by which you executed javascript? I can not find something like executeJavascript() with the request object. — rahulserver, Feb 04 '14 at 07:37
@rahulserver updated with the complete code, it's on the `HtmlPage` instance which in turn is created via the request — PopoFibo, Feb 04 '14 at 07:43
@rahulserver you can get the array in the same way as you get 150.0, I get the list as the last item in javascript and work it to get all the values - see the edit. — PopoFibo, Feb 04 '14 at 08:16

Error Executing javascript using HtmlUnit

1 Answers1

Linked