12

I am crawling data using HtmlUnit from a dynamic webpage, which uses infinite scrolling to fetch data dynamically, just like facebook's newsfeed. I used the following sentence to simulate the scrolling down event:

webclient.setJavaScriptEnabled(true);
webclient.setAjaxController(new NicelyResynchronizingAjaxController());
ScriptResult sr=myHtmlPage.executeJavaScript("window.scrollBy(0,600)");
webclient.waitForBackgroundJavaScript(10000);
myHtmlPage=(HtmlPage)sr.getNewPage();

But it seems myHtmlPage stays the same with the previous one, i.e., new data is not appended in myHtmlPage, as a result I can only crawl the first few data on the web page. Thanks for your help!

Marcopolo Soc
  • 121
  • 1
  • 4

2 Answers2

0

I was searching the same thing. I was only able to find that it is not scroll event (90% sure). There is link on JS wich is responsilbe for loading the page and could maybe help you.

Community
  • 1
  • 1
Srneczek
  • 1,945
  • 1
  • 17
  • 25
-1

I had similiar problem where the content were post-loaded during page scrolling. I solved it using:

webClient.getCurrentWindow().setInnerHeight(Integer.MAX_VALUE);

pitschr
  • 479
  • 1
  • 6
  • 12