How to sort unittest TestCases properly?

Question

For instance, I want these to run in the order they appear in the file.

import unittest

class Test_MyTests(unittest.TestCase):

    def test_run_me_first(self): pass
    def test_2nd_run_me(self):   pass
    def test_and_me_last(self):  pass

class Test_AnotherClass(unittest.TestCase):

    def test_first(self):        pass
    def test_after_first(self):  pass
    def test_de_last_ding(self): pass

if __name__ == "__main__":
    unittest.main(verbosity=2)

Running this gives

test_after_first (__main__.Test_AnotherClass) ... ok
test_de_last_ding (__main__.Test_AnotherClass) ... ok
test_first (__main__.Test_AnotherClass) ... ok
test_2nd_run_me (__main__.Test_MyTests) ... ok
test_and_me_last (__main__.Test_MyTests) ... ok
test_run_me_first (__main__.Test_MyTests) ... ok

Not what I want.

The answers I've found here on SO after searching all either dance around and avoid the issue by calling / grouping tests or saying

• "don't do it, just write your tests differently" or

• "name your tests lexicographically!"

...except for one, which does what I want:

loader = unittest.TestLoader()
ln = lambda f: getattr(MyTestCase, f).im_func.func_code.co_firstlineno
lncmp = lambda a, b: cmp(ln(a), ln(b))
loader.sortTestMethodsUsing = lncmp
unittest.main(testLoader=loader, verbosity=2)

but only for one TestCase class, and only in Python 2.

I want to run all my Python 3 unittest.TestCase subclasses, and I'd like to be able to specify the exact algorithm to use for ordering.

Can I do this in unittest?

Answer 1

^{After doing a lot of research, aided greatly by SO and Python's help and not at all by unittest's documentation, I got the answer I initially wanted so I figured I'd write this up to help others, because this is a fair (and apparently common) request.}

---

To run a specific test case, we need to make a new TestSuite for that TestCase. Let's do that for any number of TestCases:

def suiteFactory(*testcases):

    ln    = lambda f: getattr(tc, f).__code__.co_firstlineno
    lncmp = lambda a, b: ln(a) - ln(b)

    test_suite = unittest.TestSuite()
    for tc in testcases:
        test_suite.addTest(unittest.makeSuite(tc, sortUsing=lncmp))

    return test_suite

It's pleasingly simple:

1. Define a function to get a function's line number. In Python 3, the attribute we're after changed from func.im_func.func_code.co_firstlineno to func.__code__.co_firstlineno, which you can see using dir(anyFunction).

2. Define a function to sort two arguments based on cmping their line numbers. cmp isn't in Python 3 on account of People Can Do Math, so I've just done exactly what it did in the interest of readability.

3. Make a new blank TestSuite(), and give it a TestCase or ten, then tell it to sort that TestCase's methods using point #2: cmping their line numbers.

---

Now we need to sort the file's TestCase subclasses.

To do this, we can look at the globals() and their attributes.

def caseFactory():

    from inspect import findsource

    g = globals().copy()

    cases = [
        g[obj] for obj in g
            if obj.startswith("Test")
            and issubclass(g[obj], unittest.TestCase)
    ]

    ordered_cases = sorted(cases, key=lambda f: findsource(f)[1])

    return ordered_cases

This will just get all the subclasses of unittest.TestCase that begin with Test, or any naming convention you prefer, and then sort them by their line number: findsource(object) returns source code, and line number as index 1, which is what we care about.

---

To wrap it into something we can use:

if __name__ == "__main__":
    cases = suiteFactory(*caseFactory())
    runner = unittest.TextTestRunner(verbosity=2)
    runner.run(cases)

This does compartmentalise the output, but probably a good thing if the lowest-level tests are at the top (or bottom, or somewhere) in the file, and should run before the higher-level tests.

So the full output is then:

test_run_me_first (__main__.Test_MyTests) ... ok
test_2nd_run_me (__main__.Test_MyTests) ... ok
test_and_me_last (__main__.Test_MyTests) ... ok
test_first (__main__.Test_AnotherClass) ... ok
test_after_first (__main__.Test_AnotherClass) ... ok
test_de_last_ding (__main__.Test_AnotherClass) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.000s

OK

Success!

---

You can find a more interesting version of this on Github gist.