0

I'm trying to get values from a table in a webpage that has only divelements. No tr td. Below is an example (not the real one) how the HTML look like.

<div class="table" style="style="transform: translate3d(0px, 0px, 0px); opacity: 1;">
<div class="row heading">
<div class="row" style="opacity: 1;">
 <div class="cell">
  <div class="row1">value1</div>
  <span class="Tooltip" style="style="position: relative;">...</span>
 </div>
 <div class="cell1">
  <div class="classname"></div>
  <div class="row1">value2</div>
 </div>
 <div class="cell1">
  <div class="classname"></div>
  <div class="row1">value3</div>
 </div>
 <div class="cell1">
  <div class="classname"></div>
  <div class="row1">value4</div>
 </div>
 <div class="cell1">
  <div class="classname"></div>
  <div class="row1">value5</div>
 </div>

The table has 50 rows in it 5 columns each. I tried 2 approaches to get all the values from the table.

  1. Using CSSSelector

     for(int i=2;i<=38;i++) {
         WebDriverWait wait = new WebDriverWait(driver, 60);
         wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("#root > div > div.Home > div.home-left > div.table > div:nth-child("+i+") > div:nth-child(1) > div")));            
         String valueone = driver.findElement(By.cssSelector("#root > div > div.Home > div.home-left > div.table > div:nth-child("+i+") > div:nth-child(1) > div")).getText();
         clist.add(valueone);
         for(int j=2;j<=5;j++) {
             val = driver.findElement(By.cssSelector("#root > div > div.Home > div.home-left > div.table > div:nth-child("+i+") > div:nth-child("+j+") > div.total")).getText();
             clist.add(val);
         }   
     }

  2. Using Xpath:

     for(int i=2;i<=38;i++) {
 String c;
     WebDriverWait wait = new WebDriverWait(driver, 30);
     wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//*[@id=\"root\"]/div/div[3]/div[1]/div[5]/div["+i+"]/div[1]/div")));
     valueone = driver.findElement(By.xpath("//*[@id=\"root\"]/div/div[3]/div[1]/div[5]/div["+i+"]/div[1]/div")).getText();
         clist.add(valueone);
 }
 for(int j=2;j<=38;j++) {
     c = driver.findElement(By.xpath("//*[@id=\"root\"]/div/div[3]/div[1]/div[5]/div["+j+"]/div[2]/div[2]")).getText();
     clist.add(c);
 }   
 for(int k=2;k<=38;k++) {
     c = driver.findElement(By.xpath("//*[@id=\"root\"]/div/div[3]/div[1]/div[5]/div["+k+"]/div[3]/div")).getText();
     clist.add(c);
 }
 for(int l=2;l<=38;l++) {
     c = driver.findElement(By.xpath("//*[@id=\"root\"]/div/div[3]/div[1]/div[5]/div["+l+"]/div[4]/div[2]")).getText();
     clist.add(c);
 }

Issue: When I try to iterate using css selector, I get this exception (couldn't find 12th row) -

    Exception in thread "main" org.openqa.selenium.TimeoutException: Expected condition failed: waiting for visibility of element located by By.cssSelector: #root > div > div.Home > div.home-left > div.table > div:nth-child(12) > div:nth-child(1) > div (tried for 30 second(s) with 500 milliseconds interval)
at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:95)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:272)
at App.GetData.main(GetData.java:46)
    Caused by: org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"#root > div > div.Home > div.home-left > div.table > div:nth-child(12) > div:nth-child(1) > div"}

And when I try to iterate using X Path, I get this exception (couldn't find 13th row) -

    Exception in thread "main" org.openqa.selenium.TimeoutException: Expected condition failed: waiting for visibility of element located by By.xpath: //*[@id="root"]/div/div[3]/div[1]/div[5]/div[13]/div[1]/div (tried for 30 second(s) with 500 milliseconds interval)
at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:95)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:272)
at App.GetData.main(GetData.java:72)
    Caused by: org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="root"]/div/div[3]/div[1]/div[5]/div[13]/div[1]/div"}

I tried ImplicitWait, ExplicitWait and even Thread.sleep (which I know is not recommended). Nothing seems to be working. Irony is if I try 20 times, Selenium identifies all the elements without issue in 1 try.

  1. Why is that css selector is finding hard to find an element and xpath is finds the element but finds hard to find another element?

  2. I have faced this issue often in Selenium, when it is able to find the same element without issue on one run and struggles to find it on other runs? Is there a permanent solution/approach for this issue?

justcurious
  • 829
  • 3
  • 11
  • 28

2 Answers2

1

To get all the values instead of using visibilityOfElementLocated() you need to induce WebDriverWait for the visibilityOfAllElementsLocatedBy() and you can use the following based Locator Strategy:

  • Code Block:

    List<WebElement> values = new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//div[@class='table']//div[@class='cell' or @class='cell1']/div[@class='row1']")));
for(WebElement value:values) { System.out.println(value.getText()); }

Alternative using Java 8 stream() and map()

As an alternative you can use Java8 stream() and map() as follows:

  • Code Block:

    System.out.println(new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//div[@class='table']//div[@class='cell' or @class='cell1']/div[@class='row1']"))).stream().map(element->element.getAttribute("innerHTML")).collect(Collectors.toList()));
DebanjanB
  • 118,661
  • 30
  • 168
  • 217
  • Hi, Thanks for your answer. I tried both the methods suggested by you using css and xpath. Still no luck. Same errors. – justcurious Jun 20 '20 at 13:37
  • Exception in thread "main" org.openqa.selenium.TimeoutException: Expected condition failed: waiting for visibility of all elements located by By.cssSelector: #root > div > div.Home > div.home-left > div.table > div:nth-child(12) > div:nth-child(1) > div (tried for 30 second(s) with 500 milliseconds interval) – justcurious Jun 20 '20 at 13:38
  • Exception in thread "main" org.openqa.selenium.TimeoutException: Expected condition failed: waiting for visibility of all elements located by By.xpath: //*[@id="root"]/div/div[3]/div[1]/div[5]/div[13]/div[1]/div (tried for 30 second(s) with 500 milliseconds interval) – justcurious Jun 20 '20 at 13:39
  • @justcurious Those cssSelector and the xpath were never a part of my answer. Can you just copy paste my code and let me know the status please? – DebanjanB Jun 20 '20 at 13:41
  • Your cssSelector and the xpath were for the HTML I have mentioned, I believe. As I have mentioned in my question that is not the real HTML. I just copy pasted your code and added the real xpath and cssSelector – justcurious Jun 20 '20 at 13:44
  • Also, I tried using your exact XPath - List values = new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//div[@class='table']//div[@class='cell' or @class='cell1']/div[@class='row1']"))); This time the code didn't throw any error but terminated after identifying elements till 12th row. – justcurious Jun 20 '20 at 13:56
  • @justcurious My answer was based on the information you provided within the question. Of coarse, I don't have access to your webpage or the application neither I can read your mind :) – DebanjanB Jun 20 '20 at 19:37
  • @justcurious If this, or any other answer, was useful please [upvote](https://stackoverflow.com/help/why-vote) it. Once you find the answer to your question, please mark it as accepted so the question isn't left unanswered. – DebanjanB Jun 20 '20 at 19:39
0

Found out what is causing the issue, finally! I was playing around with the HTML and found out that, upon initial loading only the initial 12 rows are getting loaded (even though all the rows are displayed) and only upon further scrolling, rest of the rows are getting loaded. So did the following before traversing through the rows and now all the rows are now getting identified both by cssSelector and Xpath.

((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight)");
justcurious
  • 829
  • 3
  • 11
  • 28