Why does SauceLabs say my QUnit Test Fails when its clearly passing?

Question

We are trying to use SauceLabs to verify that our browser-based QUnit tests are passing in popular device/browser combinations ...

The tests PASS when we look at them in the browser: https://ordenado.herokuapp.com/

But for some reason SauceLabs is telling us they "failed"...

See: https://saucelabs.com/tests/5b0f07813a7f4934bb44b07606ea2fd5

cURL Command Used

For reference, we used the following curl command:

curl https://saucelabs.com/rest/v1/ordem/js-tests \
-X POST \
-u ordem:SECRET_KEY \
-H 'Content-Type: application/json' \
--data '{
    "platforms": [
      ["Windows 8.1", "internet explorer", "11"],
      ["Windows 8", "internet explorer", "10"],
      ["Windows 8.1", "firefox", "beta"],
      ["Windows 8", "firefox", "37"],
      ["Windows 7", "firefox", "32"],
      ["OS X 10.8", "safari", "6"],
      ["OS X 10.8", "chrome", "37"],
      ["Linux", "chrome", "30"],
      ["Linux", "firefox", "dev"],
      ["OS X 10.10","iphone", "7.0"],
      ["OS X 10.10","iphone", "8.2"],
      ["OS X 10.10","ipad", "7.0"],
      ["OS X 10.10","ipad", "8.2"]
    ],
    "url": "https://qunit.herokuapp.com/test/test.html?coverage=true",
    "framework": "qunit",
    "name":"ordem",
    "public": "public",
    "build": "build-007"
}'

Also, does anyone else ever experience the following error:

"The Sauce VMs failed to start the browser or device"

Visit: https://saucelabs.com/u/ordem for complete list of tests. click on any of the ones Sauce claims "failed" and watch the video to see the tests passing!!

Any insight much appreciated!

Answer 1

You should add several hooks to report qunit test results from SauceLabs.

There is an article how to get started with qunit. There mentioned repository with example where is following code snippet:

var log = [];
QUnit.done = function (test_results) {
  var tests = log.map(function(details){
    return {
      name: details.name,
      result: details.result,
      expected: details.expected,
      actual: details.actual,
      source: details.source
    }
  });
  test_results.tests = tests;

  // delaying results a bit cause in real-world
  // scenario you won't get them immediately
  setTimeout(function () { window.global_test_results = test_results; }, 2000);
};
QUnit.testStart(function(testDetails){
  QUnit.log = function(details){
    if (!details.result) {
     details.name = testDetails.name;
     log.push(details);
    }
 }
});

Add this code before your tests and results will be reported properly