How do you run NUnit tests from Jenkins?

Question

I'm looking to run automated NUnit tests for a C# application, nightly and on each commit to svn.

Is this something that Jenkins-CI can do?
Is there an online tutorial or how-to document which documents a similar setup that I can look at?

Answer 1

I needed to do exactly what you do, here's how I setup Jenkins to do this:

1. Add the NUnit Plugin to Jenkins
2. In your project go to Configure -> Build -> Add a build step
3. In the dropdown scroll down to -> Execute Windows Batch Command
4. Ensure this step is placed after your MSBuild step
5. Add the following, replacing the variables:

Single dll test:

[PathToNUnit]\bin\nunit-console.exe [PathToTestDll]\Selenium.Tests.dll /xml=nunit-result.xml

Multiple dll test using NUnit test projects:

[PathToNUnit]\bin\nunit-console.exe [PathToTests]\Selenium.Tests.nunit /xml=nunit-result.xml

1. Under Post-build Actions, tick Publish NUnit test result report
2. For the textbox Test report XMLs, enter nunit-result.xml

Once you project has been built, NUNit will now run and the results will be viewable either on the Dashboard(if you hover over the Weather report icon) or on the project page under Last Test Result.

You could also run the command from within Visual Studio or as part of you local build process.

Here's two blog posts I used for reference. I didn't find any that fitted my requirements exactly:
1-Hour Guide to Continuous Integration Setup: Jenkins meets .Net (2011)
Guide to building .NET projects using Hudson (2008)

Answer 2

If you don't want to hardcode your unit test projects, you are better off writing a script to grab all of your Unit Test project dll's. We do it with Powershell and follow a specific convention for naming our Unit Testing Projects. Here is the content of the powershell file that runs our unit tests:

param(
[string] $sourceDirectory = $env:WORKSPACE
, $fileFilters = @("*.UnitTests.dll", "*_UnitTests.dll", "*UnitTests.dll")
, [string]$filterText = "*\bin\Debug*"
)

#script that executes all unit tests available.
$nUnitLog = Join-Path $sourceDirectory "UnitTestResults.txt"
$nUnitErrorLog = Join-Path $sourceDirectory "UnitTestErrors.txt"

Write-Host "Source: $sourceDirectory"
Write-Host "NUnit Results: $nUnitLog"
Write-Host "NUnit Error Log: $nUnitErrorLog"
Write-Host "File Filters: $fileFilters"
Write-Host "Filter Text: $filterText"

$cFiles = ""
$nUnitExecutable = "C:\Program Files (x86)\NUnit 2.6.3\bin\nunit-console-x86.exe"

# look through all subdirectories of the source folder and get any unit test assemblies. To avoid duplicates, only use the assemblies in the Debug folder
[array]$files = get-childitem $sourceDirectory -include $fileFilters -recurse | select -expand FullName | where {$_ -like $filterText}

foreach ($file in $files)
{
    $cFiles = $cFiles + $file + " "
}

# set all arguments and execute the unit console
$argumentList = @("$cFiles", "/framework:net-4.5", "/xml=UnitTestResults.xml")

$unitTestProcess = start-process -filepath $nUnitExecutable -argumentlist $argumentList -wait -nonewwindow -passthru -RedirectStandardOutput $nUnitLog -RedirectStandardError $nUnitErrorLog

if ($unitTestProcess.ExitCode -ne 0)
{
    "Unit Test Process Exit Code: " + $unitTestProcess.ExitCode
    "See $nUnitLog for more information or $nUnitErrorLog for any possible errors."
    "Errors from NUnit Log File ($nUnitLog):"
    Get-Content $nUnitLog | Write-Host
}

$exitCode = $unitTestProcess.ExitCode

exit $exitCode

The script is robust enough that we are reusing for all of our build jobs. If you don't like the full path to NUnit console, you could always put that location in your PATH environment variable.

Then we put the RunUnitTests.ps1 file on our build server and use this batch command:

powershell.exe -file "{full-path-to-script-direcory}\RunUnitTests.ps1"

Answer 3

For Nunit 3 or above farmework:

1. Building Step (Windows command line) "c:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" c:\AutomationTraining\CSharpSelenium\bin\Debug\test.dll --result=TestR.xml;format=nunit2

2. Post step for Nunit report publishing, it shows only test results file in Jenkins workspace directory, not in your project: TestR.xml

We need to make test results in nunit2 format because now Jenkins Nunit plugin doesn't recognize Nunit3 results format. Also options string format is different: --result=TestR.xml;format=nunit2 NOT /xml=nunit-result.xml

Answer 4

This works nicely, I've set this up before.

Configure NUnit to output the results to an XML file and configure the NUnit Jenkins Plugin to consume this XML file. The results will be available on the dashboard.

Now, how you invoke NUnit is up to you. The way we did it was: Jenkins job executes NAnt target executes NUnit test suite.

You can configure Jenkins jobs to run on commit and/or scheduled at a certain time.

Answer 5

The solution from Ralph Willgoss is working good, but i changed 2 things to make it great:

a) I used a NUnit project instead of the DLL file directly. This makes it more easy to add more assemblies or configure the test in the NUnit GUI.

b) I added one more line to the batch to prevent the build from failing when a test fails:

[PathToNUnit]\bin\nunit-console.exe [PathToTestProject]\UnitTests.nunit /xml=nunit-result.xm
exit 0

The NUnit Plugin mentioned marks the build UNSTABLE automatically, which is exactly what i want, whenever a test fails. It shows with a yellow dot.

Answer 6

I think it's better to fail the build when it doesn't pass so you don't deploy it. Do something like this:

C:\YourNUnitDir\nunit-console.exe C:\YourOutDir\YourLib.dll /noshadow
if defined ERRORLEVEL if %ERRORLEVEL% neq 0 goto fail_build

:: any other command

: fail_build
endlocal
exit %ERRORLEVEL%

Reference: http://www.greengingerwine.com/index.php/2013/01/tip-check-errorlevel-in-your-post-build-steps-when-using-nunit/

Answer 7

This is my solution for running OpenCover with vstest in Jenkins:

param(
[string] $sourceDirectory = $env:WORKSPACE
, $includedFiles = @("*Test.dll")
, $excludedFiles = @("*.IGNORE.dll")
, [string]$filterFolder = "*\bin\Debug*"
)

# Executables
$openCoverExecutable = "C:\Users\tfsbuild\AppData\Local\Apps\OpenCover\OpenCover.Console.exe"
$unitExecutable = "F:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"

# Logs
$openCoverReport = Join-Path $sourceDirectory "opencover.xml"
$openCoverFilter = "+[*]* -[*Test]*"

Write-Host "`r`n==== Configuration for executing tests ===="
Write-Host "Source: `"$sourceDirectory`""
Write-Host "Included files: `"$includedFiles`""
Write-Host "Excluded files: `"$excludedFiles`""
Write-Host "Folder filter: `"$filterFolder`""
Write-Host ""
Write-Host "OpenCover Report: `"$openCoverReport`""
Write-Host "OpenCover filter: `"$openCoverFilter`""

# look through all subdirectories of the source folder and get any unit test assemblies. To avoid duplicates, only use the assemblies in the Debug folder
[array]$files = get-childitem $sourceDirectory -include $includedFiles -exclude $excludedFiles -recurse | select -expand FullName | where {$_ -like $filterFolder} | Resolve-Path -Relative

$exitCode = 0
$failedTestDlls = ""

foreach ($file in $files)
{
    Write-Host "`r`nCurrent test dll: $file"

    # set all arguments and execute OpenCover
    $argumentList = @("-target:`"$unitExecutable`"", "-targetargs:`"$file /UseVsixExtensions:false /Logger:trx`"", "-register:user -filter:`"$openCoverFilter`" -mergeoutput -mergebyhash -skipautoprops -returntargetcode -output:`"$openCoverReport`"")

    $unitTestProcess = start-process -filepath $openCoverExecutable -argumentlist $argumentList -wait -nonewwindow -passthru -WorkingDirectory $sourceDirectory

    if ($unitTestProcess.ExitCode -ne 0)
    {
        $failedTestDlls = $failedTestDlls + $file + "`r`n"
        $exitCode = $unitTestProcess.ExitCode
    }
}

if ($exitCode -ne 0)
{
    Write-Host "`r`n==== Executing tests in following dlls failed ===="
    Write-Host "$failedTestDlls"
}

exit $exitCode

Each test dll is executed in an own process because we had troubles to execute all test dlls in a single procress (probmels with assembly loading).

Answer 8

Jenkins does have plugins that will support that. The exact configuration is going to depend quite a bit on your project setup. There are specific plugins for nUnit, MSBuild,nAnt etc. Start by looking at the plugins page, but it shouldn't be terribly difficult to figure out.

Answer 9

For .Net Core it suffices to add "execute shell" build step with following script:

#!bash -x

cd $my_project_dir
rm -rf TestResults   # Remove old test results.
dotnet test -l trx

After that add "Publish MSTest test result report" post-build action to make test results visible.

Default test reports path should be **/*.trx and will publish all produced .trx files.