3

Our team is trying to implement Azure DevOps pipeline but we are having some issues. For background, we are running the tests on our self-hosted windows server. We got the front-end tests to all work but now we are having issues getting the backend to pass.

While the majority of the tests are passing just fine we have a few instances where it is not working as expected. Although they should be able to run in parallel, I turned that off to verify that wasn't the issue. I've also tried running them in isolation without any noticeable change.

The errors we get are just simple assert failures like the one below that don't give us much information.

 Error Message:
   Assert.AreEqual failed. Expected:<True>. Actual:<False>.

But the line numbers point to a mock setup like the one below which I don't understand.

mockQueryHandler.Setup(x => x.Handle(It.IsAny<FindQuery>())).Returns(info);

Currently the test class that is giving us the most problems is one that tests our event alert functionality. It seems that the first test will pass but the subsequent ones fail. However, we have the pipeline setup to rerun any failed tests three times until it gives up. So on every rerun the next test will pass. The only thing I can think of is that our auto mocks are giving us trouble for some reason. We set them up in accordance with AutoMock - How to unit test with Keyed Registrations?

We mock it out as follows:

  var IAbstractEventFactoryMock = new Mock<IAbstractEventFactory>();
  using (var mock = AutoMock.GetLoose(builder => builder.RegisterInstance(IAbstractEventFactoryMock.Object).Keyed<IAbstractEventFactory>("EventLogFactory").Keyed<IAbstractEventFactory>("ArpEventLogFactory"))) {
...
}

Below is our .runsettings file

<RunSettings>  
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
  </RunConfiguration>

  <MSTest>
  </MSTest>
</RunSettings>

Below is the YAML we are using to run the tests

steps:
- task: VSTest@2
  displayName: 'Run All Tests'
  inputs:
    testAssemblyVer2: |
     **\*ID_Test*.dll
     !**\*TestAdapter.dll
     !**\obj\**
    searchFolder: '$(System.DefaultWorkingDirectory)/ID_Test'
    runSettingsFile: 'ID_Test/.runsettings'
    runInParallel: false
    runTestsInIsolation: true
    codeCoverageEnabled: false
    testRunTitle: 'Unit Tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    rerunFailedTests: true
  continueOnError: true

Again, these tests all pass without problems locally and they pass eventually if I run them enough times on the pipeline. So I don't think the problem is in the code itself. It seems to be that the tests aren't getting setup or cleaned properly. I am primarily looking for any ideas on other ways I can configure my yaml or runsettings to try to fix this issue. Thank you in advance for any help and let me know if I can provide any additional information to help.

Andrew
  • 165
  • 8
  • 1
    Do you mean you are using self-hosted build agent and the tests pass without issue when you run locally? Do you run the tests on the self-hosted build agent machine? Could you try to clean the `\_work` folder of the build agent and try again? Also, try to configure a new agent to see how's the result. – Cece Dong - MSFT Oct 26 '20 at 09:23
  • Yes, when I run the test locally on my computer using Visual Studio's test runner they all pass. And we are using a self-hosted Windows build agent which is our server computer. That is a different computer than my own though, so I haven't been able to manually run the local tests on the server. I will look into testing that as well as the other solutions you mentioned, thanks for the ideas. – Andrew Oct 26 '20 at 12:59
  • Kindly let us know your update. – Cece Dong - MSFT Oct 27 '20 at 02:24
  • Sorry for the delay, I was able to clean the \_work folder but that didn't seem to have any effect. I also looked into running the tests locally on the server computer but my management said I can't have direct access to it. I will look into downloading an agent on my own computer to test that. – Andrew Oct 27 '20 at 16:27
  • I did find out that the server doesn't actually have Visual Studio installed. Could that create any problems? I feel like that shouldn't be an issue though because all the other tests work fine. – Andrew Oct 27 '20 at 16:29
  • You can use the `Visual Studio Test Platform Installer` task to run tests without needing a full Visual Studio installation. You may try to add this task before `VSTest` task and select appropriate version. Then in `VSTest` task, choose `Installed by Tools Installer` under `Test platform version`. – Cece Dong - MSFT Oct 28 '20 at 07:46

2 Answers2

0

You can use the Visual Studio Test Platform Installer task to run tests without needing a full Visual Studio installation. You may try to add this task before VSTest task and select appropriate version. Then in VSTest task, choose Installed by Tools Installer under Test platform version:

enter image description here

Cece Dong - MSFT
  • 25,734
  • 1
  • 13
  • 30
  • Thanks for the idea. Unfortunately it didn't seem to have any affect on the tests passing. But I'll still keep it implemented in the project since it seems like best practices in my case. – Andrew Oct 28 '20 at 14:14
  • How's the result when you run tests on your own computer? Do you also have this issue? – Cece Dong - MSFT Oct 29 '20 at 07:49
  • Well the tests all pass locally. I am looking into hosting the agent on my local computer to test if it is just an issue with the server computer itself rather than the pipeline configuration. – Andrew Oct 29 '20 at 15:13
  • Kindly let us know your result. – Cece Dong - MSFT Oct 30 '20 at 01:19
  • After testing it some more it looks like it is just an issue with our own tests. There were a lot of instances where we got the current time and used that. Normally it was fine but on a faster computer it created a few places where it would create errors depending on execution time. Thank you for your assistance and suggestions. – Andrew Oct 30 '20 at 18:06
0

Turned out this was just an issue with our tests. In a few places we would use the current time which was fine locally, but the server had a much higher execution time that created a few errors. Testing the pipeline on an agent that had a similar execution time as running the tests locally helped me discover that it wasn't an issue with the pipeline.

Andrew
  • 165
  • 8