Automatically detect test coupling in Protractor (randomizing test execution order)

Question

The Problem:

We have a rather large test codebase. From time to time, instead of executing all the tests, we execute them individually or in packs. But, sometimes, we see the unexpected test failures because of the tests being interconnected, coupled. For example, one test assumes there is some data created by a previous test - running this kind of test individually will fail.

The Question:

Is it possible to automatically detect which Protractor tests are coupled in the project?

Our current idea is to somehow randomize the test execution order or randomly pick up a pack of tests from all the available tests and check if there are no failures. Hence, the other question: is it possible to change/randomize the Protractor test discovery and change the order of test execution?

Inspired by the Ned Batchelder's "Finding test coupling" blogpost and the Python nose test runner's nose-randomly plugin:

Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.

By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced - a technique used in many places, for example Google’s C++ test runner googletest.

Have you considered creating the necessary data in the tests setup? this will remove the dependency in other tests. — Guy, Dec 25 '16 at 05:28
@Guy yeah, ideally, the tests should be independent from each other, properly set up and cleaned up, but unfortunately, we have this problem already. Plus, if there is a way to randomize the test execution order, that would be a great advantage to us in the future. We'll have this as a sort of "isolation test" for our test. Thanks. — alecxe, Dec 25 '16 at 05:31

score 12 · Accepted Answer · answered Dec 25 '16 at 13:06

12

You can run tests randomly (at the file level) by setting the random property in your config . You can also set your salt/seed so it's reproducibly random.

/**
 * If true, run specs in semi-random order
 */
random?: boolean,
/**
 * Set the randomization seed if randomization is turned on
 */
seed?: string,

You could also turn on shardTestFiles (parallel test runs), which should also be very telling in how coupled your tests are.

answered Dec 25 '16 at 13:06

Brine

3,635
1
17
37

Sounds very close to solve the part of the problem with randomization! Thanks. Running tests in random blocks (not all tests, but random parts) is an open problem though. – alecxe Dec 25 '16 at 15:10
It seems like this randomizing test blocks should be doable... probably at the test-runner level. So maybe a jasmine or mocha thing. Hmmm.... I try to always run my new tests, or tests I'm fixing, to run each test in isolation. I just decoupled a test today in fact... by adding an `afterEach` cleanup block. This _can_ seem time consuming... but certainly no more time consuming that debugging an entire test run :) – Brine Jan 05 '17 at 23:37
Yes, I understand the theory of test setups and teardowns, here I've tried to get more a perspective of: "okay, we already have the tests coupled, now, since there are a lot of tests, is there a way to find the groups of already coupled tests". Then, as a bonus, the same mechanism can be applied to run tests randomly in the future which would help to maintain the isolated tests. I think the `random` flag is the solution to this. The other part is a bonus and can be programmed through dynamically preparing specs or suites inside the protractor config. Thanks again! – alecxe Jan 06 '17 at 20:26
Yeah, I'm with you. Didn't mean to come off as lecturing... I was just expanding on my response (and kind of rambling :) ). My intent would have been clearer if I'd also added "... but having a way to help _detect_ these couplings would certainly be better yet". It's a great question, Alecxe... and I dug the link to the article. Thanks for posting. – Brine Jan 07 '17 at 18:30

Vishal Aggarwal · Answer 2 · 2017-10-22T23:46:05.950

Did you try shuffling "it" blocks like below:

var shuffle = function (items) {
  var item, randomIndex;      
  for(var i = 0; i < items.length; i++){
    randomIndex= (Math.random() * items.length) | 0;
    item = items[i];
    items[i] = items[randomIndex];
    items[randomIndex] = item;
  }
}

describe('Suite', function() {

  it("should a", function () {
      console.log("execute a");
  });

  it("should b", function () {
      console.log("execute b");
  });

  it("should c", function () {
      console.log("execute c");
  });

  shuffle(this.children);    // shuffle the 'it' blocks

});

Source: Can protractor tests be run in a random order?

score 4 · Answer 3 · answered Dec 25 '16 at 05:33

One problem is you likely have no idea how tests might be coupled. If one test referenced some variables from another test, you might be able to find those automatically but that's only one way tests might be coupled and probably not a likely scenario.

My first thought was to just run them individually and see which ones fail. The problem is that if you aren't cleaning state between tests, you might change the order (randomizing them, as you suggested) but if test 50 expects the data that test 20 set up but in the new order, test 20 still runs before test 50... test 50 will still pass. You will find some but probably not all until you run all of them in a random order several times.

You don't describe your application but my second thought was if there was a way to get back to a clean slate between tests, you should be able to find the tests that rely on other tests to set up data. I'm kinda surprised you aren't doing that already but if there's a long setup process that has to run to install a clean slate, etc. that might be an issue. Depending on your system, you might be able to snapshot a VM after a clean install and restore it to "quickly" get back to clean or you may be able to roll back SQL tables, etc. It really depends on your system and without more details on the system, it's hard to offer advice.

Another option is to go to those that wrote or maintain the tests and have them self-identify the tests they own that are coupled and fix them. This likely won't find them all but it might be a semi-quick start.

Oh... I just thought of one more thing. If you could reverse the order of test execution, that should be better than randomizing the execution. With reverse order, NO test would run after it's former predecessor and you should be able to find them all in one go.

Good advices in general, thanks! Reversing the order is a nice idea, it won't catch all of the connected tests, but might reveal some of them for sure. I had to keep the question generic since the interconnections between the tests vary so widely and they are pretty much application specific. Mostly, here I'm trying to see how can we randomize the protractor execution order, but it's not only about the order. Ideally, we'd like to also pick random "samples", block of tests and run them separately as well. All of it should increase/check the value and quality of our tests.. — alecxe, Dec 25 '16 at 05:43

Automatically detect test coupling in Protractor (randomizing test execution order)

3 Answers3