java - 使用 HtmlUnit 执行 javascript 时出错

Question

我正在尝试在以下 URL 上执行 JavaScript:http://steamcommunity.com/id/bobcatchris/inventory#730。我可以通过按 ctrl+shift+j 打开控制台并粘贴以下脚本来使用 google chrome 来完成此操作:

var list = []; 
var size = Object.keys(g_ActiveInventory.rgInventory).size();    
var counter = 0;
while (counter < size) {
 list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);
 counter +=1;
}

它返回:150 然后在下一行，我写:>list 并返回一个长度为 150 的数组。

当我尝试在 HtmlUnit 中这样做时:

 public static void main(String[] args) throws IOException {
        WebClient webClient=new WebClient(BrowserVersion.FIREFOX_17);
        HtmlPage page=webClient.getPage("http://steamcommunity.com/id/bobcatchris/inventory#730");
        String script="var list = [];\n" +
                "\n" +
                "\n" +
                "var size = Object.keys(g_ActiveInventory.rgInventory).size();\n" +
                "\n" +
                "\n" +
                "\n" +
                "var counter = 0;\n" +
                "\n" +
                "while (counter < size) {\n" +
                " list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);\n" +
                " counter +=1;\n" +
                "}";
        Object result = page.executeJavaScript(script).getJavaScriptResult();
        System.out.println(result);
    }

我收到以下异常:

Exception in thread "main" ======= EXCEPTION START ========
EcmaError: lineNumber=[4] column=[0] lineSource=[<no source>] name=[TypeError] sourceName=[injected script] message=[TypeError: Expected argument of type object, but instead had type object (injected script#4)]
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Expected argument of type object, but instead had type object (injected script#4)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:684)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:602)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:507)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:570)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:545)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptIfPossible(HtmlPage.java:959)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:927)
    at Scraper.main(Scraper.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Expected argument of type object, but instead had type object (injected script#4)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3603)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3587)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3608)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError1(ScriptRuntime.java:3618)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.ensureScriptable(ScriptableObject.java:2095)
    at net.sourceforge.htmlunit.corejs.javascript.NativeObject.execIdCall(NativeObject.java:287)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:89)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3031)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:561)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:669)
    ... 12 more
Enclosed exception: 
net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Expected argument of type object, but instead had type object (injected script#4)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3603)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3587)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3608)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError1(ScriptRuntime.java:3618)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.ensureScriptable(ScriptableObject.java:2095)
    at net.sourceforge.htmlunit.corejs.javascript.NativeObject.execIdCall(NativeObject.java:287)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:89)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531)
    at script(injected script:4)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3031)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:561)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:669)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:602)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:507)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:570)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:545)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptIfPossible(HtmlPage.java:959)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:927)
    at Scraper.main(Scraper.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
======= EXCEPTION END ========

导致异常的行是:

Object result = page.executeJavaScript(script).getJavaScriptResult();

那么我应该如何像从 google chrome 控制台一样从 HtmlUnit 执行这个脚本呢？

Answer 1

这些是您的 javascript 脚本错误，由您的框架转换为 Java 异常。

那是因为您没有明确地将 WebClient 的 setThrowExceptionOnScriptError 选项设置为 false。

除非 js 中绝对没有错误，否则始终将此值设置为 false 很有用，除非它会干扰您正在寻找的结果。

通常，这些是我通过 HtmlUnit 处理 js 和 ajax 时的 webclient 设置。

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17,
                PROXY_HOST, PROXY_PORT);

WebRequest request = new WebRequest(new URL(
                "http://steamcommunity.com/id/bobcatchris/inventory#730"));

webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.setJavaScriptTimeout(10000);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getOptions().setTimeout(10000);

HtmlPage page = webClient.getPage(request);

          String script="var list = [];\n" +
                    "\n" +
                    "\n" +
                    "var size = Object.keys(g_ActiveInventory.rgInventory).size();\n" +
                    "\n" +
                    "\n" +
                    "\n" +
                    "var counter = 0;\n" +
                    "\n" +
                    "while (counter < size) {\n" +
                    " list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);\n" +
                    " counter +=1;\n" +
                    "}";
            Object result = page.executeJavaScript(script).getJavaScriptResult();
            System.out.println(result);

如果我尝试您的代码，使用上述设置，我会将 150.0 打印到控制台，我认为它按预期工作。

编辑:

遍历整个数组 list:

String script="var list = [];\n" +
                    "\n" +
                    "\n" +
                    "var size = Object.keys(g_ActiveInventory.rgInventory).size();\n" +
                    "\n" +
                    "\n" +
                    "\n" +
                    "var counter = 0;\n" +
                    "\n" +
                    "while (counter < size) {\n" +
                    " list.push(g_ActiveInventory.rgInventory[Object.keys(g_ActiveInventory.rgInventory)[counter]].market_name);\n" +
                    " counter +=1;\n" +
                    "}"
                    + "list";

            Object result = page.executeJavaScript(script).getJavaScriptResult();
            if (result instanceof NativeArray) {
                for (Object obj : (NativeArray)result) {
                    System.out.println(obj);    
                }

            }

在上面，我更改了 js 以将列表作为返回参数并遍历 NativeArray 以获取每个元素。

输出:

P2000 | Scorpion (Factory New)
AK-47 | Black Laminate (Field-Tested)
★ StatTrak™ Karambit | Case Hardened (Minimal Wear)
CS:GO Case Key
CS:GO Case Key

您可以在他们的常见问题解答 - here 中阅读很多关于他们的 ajax 设置的信息.