3

I have a kind of general question I wrote several unit tests for a bunch of connected classes and all unit tests are succeeding.

However, after implementing my first integration tests I got some errors because I didn't take some shared behavior between the classes into account.

My question is: Am I supposed to fix my unit tests because they failed to find the bugs? Are unit tests not supposed to take the other classes into account and I just bugfix my productive code because it is sufficient that the integration test cover such things.

EDIT:

due to the responses it seems necessary to specify my problem further. I am still not sure which test is responsible for what.

Lets assume I have a class(lets call it ListClass) which contains a list of objects, an index, a getCurrentValue, and a incrementIndex function. I guess the function names speak for themselves.

The index starts with -1 and if you call getCurrentValue with -1 it returns you null

I also have a reset function which sets the start index back to -1.

Okay my tests have 100% code coverage, I created mutants which were detected by my tests, so everthing seems fine.

My second class is a handler which is responsible to set all Permutations everytime i call Handler.Next(). 

// listContainer1.index = -1, listContainer2.index = -1,

handler.add(listContainer1) //has 2 values
handler.add(listContainer2) //has 2 values

handler.next() // listContainer1.index = 0, listContainer2.index = 0
handler.next() // listContainer1.index = 0, listContainer2.index = 1
handler.next() // listContainer1.index = 1, listContainer2.index = 0
handler.next() // listContainer1.index = 1, listContainer2.index = 1

Again 100% code coverage, mutants were detected (I used mocking to define the behaviour of the listContainer)

Here is the Integration test which broke the code: Calling reset on a listContainer, after several handler.next() calls Because the container index was -1 the next handler.next() call resulted in an not expected state.

How can I predict such an outcome in an unit test in my handler without being depent on my ListClass? Or as I asked in my original post .. Is it sufficient if the integration test catches the error.

For me it seems both unit test classes covered their own responsibilities ...

Silvio Marcovic
  • 425
  • 4
  • 18

3 Answers3

3

First you have to determine where the problems are at.

Are your unit tests relevant? This includes:

  • Do they have meaningful asserts?
  • How much of your code is covered by unit tests?
  • Do you test if it works in valid conditions and fails in invalid?

You can only fix your unit tests if there is actually something wrong with them. Typically though you built your tests in such a manner that you should be able to depend on them. Considering you feel like you can't, you should make that your first priority (if you can't trust them, then they are essentially worthless).

Considering the problems occurred when you started with integration tests your first train of thought (assuming sufficient unit tests) should be to verify the integrating works well. If an external dependency goes wrong somewhere then it is not necessarily a programmatic error but perhaps a configuration that isn't properly setup (like a database connection).

Since your situation seems to deal with classes interrupting eachothers behaviour, I believe you'll have to revisit your test design because you have likely made some mistakes there. Even though the unit tests themselves work, you have to make sure that they are isolated but you should also be aware that a unit might be more than just one method or class (more on that here).

One example of something you should look out for are state changes, typically something that happens as a result of a void method but certainly possible as a side-effect of any type of action. This might interfere with assumptions you make in other tests.

Because of the general nature of the question I have kept the answer pretty general as well but if you'd share some specifics then I could provide a more specific answer.

In short: bad unit tests are unreliable and thus useless unit tests; make sure they are sufficiently relevant to your application.

Community
  • 1
  • 1
Jeroen Vannevel
  • 41,258
  • 21
  • 92
  • 157
1

Unit tests check that each part is working properly. Integration tests check that the parts fit together. If either one of them could cover everything you wouldn't need the other.

Still - eventually you would need to change your unit tests, not because they are wrong but because if the components of your program don't fit together you'll need to alter the interface behavior(or code-that-using-other-interface) of some of them, which means you'll need to change some unit tests.

For example, if Foo depends on Bar, both pass unit tests, but an integration test that checks them both fails, that means there is something wrong in the way Foo uses Bar. This means that either(or both!):

  • You'll have to change the behavior of Bar. This means some Bar unit tests are no longer valid(since they check the old behavior) and you need to change them too.

  • You'll need to change the way Foo uses Bar. This can break some Foo unit tests where you send to Foo a mock of Bar. That mock was created with the old way Foo was using Bar in mind - you'll need to change it to match the new, correct way.

Still - this does not mean the new unit tests will be able to catch integration problems!

Idan Arye
  • 11,682
  • 3
  • 40
  • 61
0

You called "reset" on the listContainer, then the handler was in a not expected state?

When you say "not expected state" this could mean your test is wrong (the state was just externally changed by the test, but the test expects something else, so it is clearly a bad test).

If the handler was corrupted by outside code changing its data, making it out of sync with another internal index maintained by the handler, then the integration test violated encapsulation. You can't unit test if they are not units.

If the handler is one of many classes accessing the listContainer then it has to use the listContainer's interface. It can't have a bad state if it just asks the listContainer what its state is. In this case the handler is entirely at fault, and you need to add a new unit test to the handler. You can't anticipate everything but when you find a problem, you can add a unit test which detects it, then fix the code so the test passes.

Kenny Ostrom
  • 4,598
  • 2
  • 16
  • 26
  • Why did i violate encapsulation with the integration test? why are they no units? Is it also the responsibilty of the handler unit test to ask even if the interface has no reset... how can you anticipate this from the view of a handler unit test? – Silvio Marcovic Jun 13 '14 at 18:51
  • IF the integration test changed the lists, when the handler is supposed to have exclusive use of them THEN it violated encapsulation. – Kenny Ostrom Jun 13 '14 at 18:59
  • I think it is not possible for a test to break encapsulation (if you are not doing stuff on purpose like reflection or deriving your test from the class under test etc..) ... If something breaks encapsulation than it is the Reset function of the class itself. And this breaks encapsulation just as much as a setter function does. And the test does not change the list it changes the index, which is responsible for, well, indexing the list – Silvio Marcovic Jun 13 '14 at 20:02
  • So the reset function is provided by the "handler" class, and using it causes the class to fail? Then you are in case three -- the problem must be entirely in the handler and you must address it in handler's unit tests. – Kenny Ostrom Jun 17 '14 at 15:14
  • I promise I read your question, although I may have misunderstood something. You said: 1) handler has lists 2) list.reset 3) handler is in a "not expected state" So perfectly legal actions which you can do in the handler unit test resulted in the handler being bad, therefore the handler is entirely at fault. Additionally, in the original post, your lists don't go through all possible permutations. If it is not supposed to return -1, then you have to check every list for -1 on every call to next. – Kenny Ostrom Jun 19 '14 at 16:00