Build Status from Matlab using Drone

Question

I am using Drone as a continuous integration (CI) server.

The tests are started using this script:

image: drone/matlab:R2014a
script:
    - cd test
    - matlab -nodesktop -nosplash -r ci_run_tests

notify:
    email:
        on_failure: blame

The function ci_run_tests is based on this answer: https://stackoverflow.com/a/23347768

As for Jenkins, the author suggests to write the test results into a *.tap-file, in my case looking like this:

1..4
ok 1 - test_annotation_to_pitch/test_with_systematic_scale
ok 2 - test_audio_to_pitch/test_120_vs_360
not ok 3 - test_pitch_to_CENS/test_12_vs_36
ok 4 - test_pitch_to_chroma/test_12_vs_36

Test 3 was failing. Drone is not aware of this information since it does not interpret those *.tap files, it only registered that Matlab exited properly - hence saying that the build itself worked.

My Question: Does Drone support some kind of functionality like the *.tap file in Jenkins

Thanks!

Answer 1

Most continuous integration systems don't parse results or know about the tests being used but check the exit status of the programs called.

To signal an error the program needs to exit with something else but 0.

While the test script mentioned has an exit(1) it seems the testrunner will not raise an exception when there was a failed test. So to check for failed tests you need to count the number of them:

function runAllMyTests

import matlab.unittest.TestSuite;
import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
    % Create the suite and runner
    suite = TestSuite.fromPackage('packageThatContainsTests', 'IncludingSubpackages', true);
    runner = TestRunner.withTextOutput;

    % Add the TAPPlugin directed to a file in the Jenkins workspace
    tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
    runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile)));

    results = runner.run(suite);

    % Count number of failed tests. Exit(1) if greater than 0
    if nnz([results.Failed])
        exit(1);
    end
catch e;
    disp(e.getReport);
    exit(1);
end;
exit force;

When you think about it, that's the behaviour you actually want: An exception always stops the execution of whatever is throwing it. So your testsuite would stop at the first encountered error, not showing any of the other ones.

Answer 2

Does drone support a JUnit style XML artifact? If it does then another solution is to use the XMLPlugin for the MATLAB Unit TestRunner instead of the TAPPlugin.