Trying to help someone who works for a nonprofit. Currently trying to pull info from the STL County Boards/Commissions website(https://boards.stlouisco.com/).
Having trouble for a few reasons:
Was going to attempt to use BeautifulSoup, but the actual data isn't even shown until you choose a Board/Commission from a dropdown bar above, so I have switched to Selenium, which I am new at.
Is this task possible? When I look at the html code for the site, I see that the info isn't stored in the page, but pulled from another location and just displayed on the site based on the option chosen from the dropdown menu.
function ShowMemberList(selectedBoard) {
ClearMeetingsAndMembers();
var htmlString = "";
var boardsList = [{"id":407,"name":"Aging Ahead","isActive":true,"description":"... ...1.","totalSeats":14}];
var totalMembers = boardsList[$("select[name='BoardsList'] option:selected").index() - 1].totalSeats;
$.get("/api/boards/" + selectedBoard + "/members", function (data) {
if (data.length > 0) {
htmlString += "<table id=\"MemberTable\" class=\"table table-hover\">";
htmlString += "<thead><th>Member Name</th><th>Title</th><th>Position</th><th>Expiration Date</th></thead><tbody>";
for (var i = 0; i < totalMembers; i++) {
if (i < data.length) {
htmlString += "<tr><td>" + FormatString(data[i].firstName) + " " + FormatString(data[i].lastName) + "</td><td>" + FormatString(data[i].title) + "</td><td>" + FormatString(data[i].position) + "</td><td>" + FormatString(data[i].expirationDate) + "</td></tr>";
} else {
htmlString += "<tr><td colspan=\"4\">---Vacant Seat---</td></tr>"
}
}
htmlString += "</tbody></table>";
} else {
htmlString = "<span id=\"MemberTable\">There was no data found for this board.</span>";
}
$("#Results").append(htmlString);
});
}
So far, I have this (not a lot), which goes to the page and selects every board from the list:
driver = webdriver.Chrome()
driver.get("https://boards.stlouisco.com/")
select = Select(wait(driver, 10).until(EC.presence_of_element_located((By.ID, 'BoardsList'))))
options = select.options
for board in options:
select.select_by_visible_text(board.text)
From here I would like to be able to scrape the info from the MemberTable but I don't know how to move forward/if it is something in the scope of my abilities, or even if it is something possible with Selenium.
I've tried using find_by a few different elements to click on the members table but am met with errors. I have also tried calling for the memberstable after my select, but it is not able to find that element. Any tips/pointers/advice is appreciated!
