C# Selenium WebDriver - Solution to Wait for Outstanding HTTP Requests

Question

Background

When writing front-end tests, we often need to wait until the web-application is done fetching data and updating the DOM before we want to interact with the page. With Selenium C#, this means a lot of explicit waits on the state of the page that is tailored to the specific scenario (maybe waiting for a loading indicator or a specific element to appear). However, most of the time this visual indicator is just a proxy for an async task like an HTTP request. Other solutions such as Protractor and Cypress have easy solutions for waiting for HTTP requests (this is the default in Protractor).

Question

One of the frameworks I maintain is written in C#, and I'm trying to find a solution to easily wait for any outstanding HTTP requests, rather than writing custom explicit waits against the DOM. Is there a solution for this? I'm open to using an additional open-source solution if needed.

I assumed I might need to set up a proxy so that I can manipulate and hook into HTTP requests. I looked into BrowserUp (continuation of the BrowserMobProxy project, which seems to no longer be maintained), but can't tell from docs if this sort of use case is possible or intended.

Answer 1

I remember years ago trying to solve this in Ruby. We settled on a hybrid JavaScript and Ruby solution. Each time an Ajax request got sent, we set a global JavaScript variable to true. When all pending requests had finished, we set it to false. We still had flaky tests. They were brittle and inconsistent even with some JavaScript gymnastics going on behind the scenes.

Even though Ajax (or background HTTP requests) might be complete, JavaScript still needs additional processing time to do something with the response. It was mere milliseconds, but remember that Selenium and your browser run in different threads — everything is a race condition. We kept getting intermittent test failures because the HTTP requests were done, but the browser was still in the process of evaluating element.innerHTML = response.responseText when Selenium would attempt to interact with an element that was supposed to be on screen after the request was complete. We still had to use explicit waits.

Basically, you are stuck with explicit waits in order to achieve stable tests. I've jumped through a lot of hoops over the years to get things working any other way. The only saving grace I've found is the Page Object Model Pattern, which at least centralizes this ugly code in one place for any particular use case.

So, yeah. The code is ugly. You need to use explicit waits. It turns out test code needs to be just as purposefully architected as the application code it tests.