Using a .net core 3.1 XUnit test project and the C# bindings for the Selenium WebDriver API, my tests succeed when I run them locally on my development machine but always fail when run in an Azure DevOps release pipeline.
Here is the Yaml definition for the test task
steps:
- task: VSTest@2
displayName: 'VsTest - testAssemblies'
inputs:
testAssemblyVer2: '**\--REDACTED--.dll'
uiTests: true
runInParallel: false
rerunFailedTests: true
I have followed the microsoft guidelines and activated uiTest
and ensured runInParallel
is false
in the pipeline definition. I have also tried running the tests with every possible permutation of these two
The ChromeDriver version is 80.0.3987.106
, which is the version pre-installed on the agent I'm using. I target the ChromeDriver binaries pre-installed on the agent using the ChromeWebDriver
environment variable (its value is C:\\SeleniumWebDrivers\\ChromeDriver
). I have tried using the windows-2019
and the vs2017-win2016
agents. This did not change anything.
The web driver is running in headless mode.
Here is a relevant sample of the output from the Azure DevOps pipeline. I believe the test does not actually fail because an element could not be located (since the tests consistently work in local), rather because the driver itself fails.
[...]
2020-03-06T14:38:02.2076961Z Starting ChromeDriver 80.0.3987.106 (f68069574609230cf9b635cd784cfb1bf81bb53a-refs/branch-heads/3987@{#882}) on port 49976
2020-03-06T14:38:02.2078676Z Only local connections are allowed.
2020-03-06T14:38:02.2079642Z Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
2020-03-06T14:38:28.0289610Z [xUnit.net 00:00:31.64] OpenQA.Selenium.NoSuchElementException : no such element: Unable to locate element: {"method":"css selector","selector":"#autocomplete"}
2020-03-06T14:38:28.0296747Z (Session info: headless chrome=80.0.3987.122)
2020-03-06T14:38:28.0297355Z [xUnit.net 00:00:31.64] Stack Trace:
2020-03-06T14:38:28.0328125Z ##[error][xUnit.net 00:00:31.64] --REDACTED--Test.ArticleSearchTest.SearchJuraSaturdaySaturdayWithoutDatesAndNoOsmSelectionResultIsPageUfi [FAIL]
2020-03-06T14:38:28.0338184Z [xUnit.net 00:00:31.64] at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
2020-03-06T14:38:28.0339005Z [xUnit.net 00:00:31.64] at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
2020-03-06T14:38:28.0339833Z [xUnit.net 00:00:31.64] at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(String mechanism, String value)
2020-03-06T14:38:28.0340530Z [xUnit.net 00:00:31.64] at OpenQA.Selenium.Remote.RemoteWebDriver.FindElementById(String id)
2020-03-06T14:38:28.0341218Z [xUnit.net 00:00:31.64] at OpenQA.Selenium.By.<>c__DisplayClass16_0.<Id>b__0(ISearchContext context)
2020-03-06T14:38:28.0341907Z [xUnit.net 00:00:31.64] at OpenQA.Selenium.By.FindElement(ISearchContext context)
2020-03-06T14:38:28.0342538Z [xUnit.net 00:00:31.64] at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(By by)
2020-03-06T14:38:28.0343418Z [xUnit.net 00:00:31.64] C:\agent\_work\25\s\Tests\--REDACTED--Test\SeleniumSearchUtils.cs(16,0): at --REDACTED--Test.SeleniumSearchUtils.SetDestinationToJuraWithoutSelectingDestinationInDropdown(IWebDriver driver)
2020-03-06T14:38:28.0344566Z [xUnit.net 00:00:31.64] C:\agent\_work\25\s\Tests\--REDACTED--Test\Tests\ArticleSearchTest.cs(118,0): at --REDACTED--Test.ArticleSearchTest.SearchJuraSaturdaySaturdayWithoutDatesAndNoOsmSelectionResultIsPageUfi()
2020-03-06T14:38:28.0345417Z [xUnit.net 00:00:31.65] Output:
2020-03-06T14:38:28.0346024Z [xUnit.net 00:00:31.65] Selenium target url: http://test.--REDACTED--.com/de/frankreich/camping-bretagne
2020-03-06T14:38:28.0375453Z [xUnit.net 00:00:31.65] Finished: --REDACTED--Test
[...]
As advised by Greg, I increased the allowed wait time before an element not found exception is thrown. I set it to 120 seconds, which lead to a timeout error Relevant stack trace
---- System.Net.WebException : The operation has timed out.
2020-03-06T15:39:37.4581244Z [xUnit.net 00:02:24.81] Stack Trace:
2020-03-06T15:39:37.4582083Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.HttpCommandExecutor.MakeHttpRequest(HttpRequestInfo requestInfo)
2020-03-06T15:39:37.4582955Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.HttpCommandExecutor.Execute(Command commandToExecute)
2020-03-06T15:39:37.4583816Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.DriverServiceCommandExecutor.Execute(Command commandToExecute)
2020-03-06T15:39:37.4584863Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
2020-03-06T15:39:37.4585662Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(String mechanism, String value)
2020-03-06T15:39:37.4586387Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.RemoteWebDriver.FindElementById(String id)
2020-03-06T15:39:37.4587146Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.By.<>c__DisplayClass16_0.<Id>b__0(ISearchContext context)
2020-03-06T15:39:37.4587838Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.By.FindElement(ISearchContext context)
2020-03-06T15:39:37.4588507Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(By by)
2020-03-06T15:39:37.4589490Z [xUnit.net 00:02:24.81] C:\agent\_work\25\s\Tests\--REDACTED--Test\SeleniumSearchUtils.cs(16,0): at --REDACTED--Test.SeleniumSearchUtils.SetDestinationToJuraWithoutSelectingDestinationInDropdown(IWebDriver driver)
2020-03-06T15:39:37.4590637Z [xUnit.net 00:02:24.81] C:\agent\_work\25\s\Tests\--REDACTED--Test\Tests\ArticleSearchTest.cs(118,0): at --REDACTED--Test.ArticleSearchTest.SearchJuraSaturdaySaturdayWithoutDatesAndNoOsmSelectionResultIsPageUfi()
2020-03-06T15:39:37.4613895Z [xUnit.net 00:02:24.81] ----- Inner Stack Trace -----
2020-03-06T15:39:37.4614741Z [xUnit.net 00:02:24.81] at System.Net.HttpWebRequest.GetResponse()
2020-03-06T15:39:37.4615470Z [xUnit.net 00:02:24.81] at OpenQA.Selenium.Remote.HttpCommandExecutor.MakeHttpRequest(HttpRequestInfo requestInfo)
2020-03-06T15:39:37.4616428Z [xUnit.net 00:02:24.81] Output:
2020-03-06T15:39:37.4617060Z [xUnit.net 00:02:24.81] Selenium target url: http://test.--REDACTED--.com/de/frankreich/camping-bretagne
2020-03-06T15:39:37.4617775Z [xUnit.net 00:02:24.82] Finished: --REDACTED--Test
Note: I tried adding the no-sandbox
option to the ChromeDriver, to no avail.
UPDATE:
After going through the logs in details again, I stumbled upon this line:
OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:49946/session/a0dac255bd41cc2bb8c6882761408f12/element timed out after 60 seconds.
I am absolutely positive the URL passed to the GoToUrl
method is http://test.--REDACTED--.com/de/frankreich/camping-bretagne
, valid URL (which is on a remote server, DNS resolution and all)
Could the localhost:49946
reference some kind of local selenium component that hasn't been launched properly ? I could not find any more information about this "back-end" in the error logs or on the internet ...