38

A Rails/tool specific version of: How deep are your unit tests?

Right now, I currently write:

  • Cucumber features (integration tests) - these test against the HTML/JS that is returned by our app, but sometimes also tests other things, like calls to third-party services.
  • RSpec controller tests (functional tests), originally only if the controllers have any meaningful logic, but now more and more.
  • RSpec model tests (unit tests)

Sometimes this is entirely necessary; it is necessary to test behavior in the model that is not entirely obvious or visible to the end-user. When models are complex, they should definitely be tested. But other times, it seems to me the tests are redundant. For instance, do you test method foo if it is only called by bar, and bar is tested? What if bar is a simple helper method on a model that is used by and easily testable in a Cucumber feature? Do you test the method in rspec as well as Cucumber? I find myself struggling with this, as writing more tests take time and maintaining multiple "versions" of what is effectively the same behaviors, which makes maintaining the test suite more time intensive, which in turn makes changes more expensive.

In short, do you believe there is there a time when writing only Cucumber features is enough? Or should you always test at every level? If you think there is a grey area, what is your threshold for "this needs a functional/unit test." In practical terms, what do you do currently, and why (or why not) do you think it's sufficient?

EDIT: Here's an example of what might be "test overkill." Admittedly, I was able to write this pretty quickly, but it was completely hypothetical.

Community
  • 1
  • 1
wuputah
  • 11,044
  • 1
  • 40
  • 59
  • been struggling with *exactly* the same problem. Managing the time/benefit ratio is hard to do sometimes. – brad Sep 17 '10 at 16:35
  • I'm also curious about this. I'm just trying to figure out how many controller/view tests are really necessary if I've got a layer of cukes tests running through my app. – wesgarrison Oct 28 '10 at 20:20

6 Answers6

27

Good question, one I've grappled with recently while working on a Rails app, also using Cucumber/RSpec. I try to test as much as possible at every level, however, I've also found that as the codebase grows, I sometimes feel I'm repeating myself needlessly.

Using "Outside-in" testing, my process usually goes something like: Cucumber Scenario -> Controller Spec -> Model Spec. More and more I find myself skipping over the controller specs as the cucumber scenarios cover much of their functionality. I usually go back and add the controller specs, but it can feel like a bit of a chore.

One step I take regularly is to run rcov on my cucumber features with rake cucumber:rcov and look for notable gaps in coverage. These are areas of the code I make sure to focus on so they have decent coverage, be it unit or integration tests.

I believe models/libs should be unit tested extensively, right off the bat, as it is the core business logic. It needs to work in isolation, outside of the normal web request/response process. For example, if I'm interacting with my app through the Rails console, I'm working directly with the business logic and I want the reassurance that methods I call on my models/classes are well tested.

At the end of the day, every app is different and I think it's down to the developer(s) to determine how much test coverage should be devoted to different parts of the codebase and find the right balance so that your test suite doesn't bog you down as your app grows.

Here's an interesting article I dug up from my bookmarks that is worth reading: http://webmozarts.com/2010/03/15/why-not-to-want-100-code-coverage/

Sidane
  • 2,091
  • 17
  • 20
  • Good link + didn't know about `rake cucumber:rcov`. Out of curiosity, do you unit test every model method, even simple things that have cucumber coverage? (See my example in the question) Do you consider it essential to the point where you won't consider code "finished" (i.e. not allowed to be committed, or merged to master) until it is tested at that level? – wuputah Sep 23 '10 at 14:58
  • I usually aim for unit test coverage of most models/libs. It doesn't always happen though, when I'm in full flow, writing code, sometimes I end up taking short-cuts. I try to regularly check for noticeable gaps in my test coverage, be it manually or with tools like rcov, and then plug those gaps. And I don't stick to a hard and fast rule like not committing code untested at the unit level. Simple boolean methods like in your example I would not be concerned about if they didn't have unit test coverage, but methods with any sort of complexity I try to test. – Sidane Sep 23 '10 at 16:00
2

Rails has a well-tested codebase, so I'd avoid re-testing stuff that is covered in those steps.

For example, unless it is custom code, it is pointless to test the results of validations at unit and functional levels. I'd test them at the integration level though. Cucumber features act as specifications for your project, so it is good to specify that you need a validation for x and y, even if the implementation is a single line declaration in the model.

edgerunner
  • 14,394
  • 2
  • 57
  • 68
  • Yes, but: while it's easy enough with Shoulda to test validations in rspec, do you think you should test validations in both Cucumber and rspec? The point is not that things should be or not be tested, it's whether things should be tested at multiple levels. Validations aren't exactly what I was thinking of (particularly since Shoulda makes it so easy), but it's a reasonable example. Pretend that you had to write those rspec tests by hand! – wuputah Sep 18 '10 at 21:24
  • 2
    To put it another way, what I meant was that you should test your original code at all levels, and test everything at the top level. Consider ActiveRecord lifecycle callbacks for example. You need not worry about if the callbacks are called on save. You need to worry about if the result you expect from that callback is on the record or not. – edgerunner Sep 18 '10 at 23:30
1

You usually don't want to have both Cucumber stories and RSpec controller specs/integration tests. Pick one (generally Cucumber is the better choice, except for certain special cases). Then use RSpec for your models, and that's all you need.

Marnen Laibow-Koser
  • 5,182
  • 1
  • 23
  • 30
0

I test complex model/lib methods with rspec then the main business logic in web with cucumber, so I'm sure that the main features of the web will work 100%, then if I got more time and resources I test everything else.

Gacha
  • 1,028
  • 8
  • 17
  • I have the same feelings, but I would like to hear more about your approach. How would you convince others that this is the right way to do it? – wuputah Sep 18 '10 at 21:32
  • The main reason I do that is because I can't test it all, so I need to prioritize and test from more important to less. To cover more of web page I suggest to create simple specs for all models to see if everything saves/validates, then specs for all complex methods, and at last cucumber for all forms, main views. And then if you have more time/resources you can test not so important things. – Gacha Sep 27 '10 at 12:27
0

Its easier to write simple specs for simple methods. Its much easier than writing cukes.

If you keep your methods simple - and keep your specs simple - by testing only the logic inside that method - you will find joy in unit testing.

If anything is redundant its cucumber tests. If you have well tested models and lib your software should work.

Vojto
  • 6,687
  • 4
  • 22
  • 30
  • 3
    Until you realise that you forgot to put a submit button on your form and you can't get from A -> B in your app. The methods in question may work in isolation and all your unit tests pass, but when used in a real world situation, your application will not work. Integration tests are just as important as unit tests imo and should not be considered redundant. – Sidane Sep 22 '10 at 11:46
  • The question is, to a degree, subjective, and one about best practices. I definitely understand the idea that integration tests can seem redundant if you have good compartmentalized tests. I am mostly coming from the other angle, that given integration tests exist, the rest can seem redundant at times. I personally think this is a better approach since, as Sidane points out, you can miss the "big picture" if you dont have integration tests. I think both opinions have merit, though. – wuputah Sep 23 '10 at 14:55
  • I strongly disagree with declaration that having only integration tests is better approach. Integration testing just ensures the whole thing works. Unit testing on the other hand, if done correctly (test-first), ensures high quality of code. – Vojto Sep 24 '10 at 15:11
  • 1
    Not _only_ integration tests, but putting the emphasis there - start at the outside and go in, instead of starting at the inside and going out. – wuputah Sep 27 '10 at 15:34
  • Yup. It's shortsighted to only have unit tests -- that will prove that each unit works, but will not prove that the units talk to each other correctly. That's why you need integration tests. I love having both Cucumber stories (which are integration tests) and unit tests, but if I had to pick one or the other, I'd go with the Cucumber stories nearly every time. – Marnen Laibow-Koser Jun 06 '11 at 14:37
-1

The purpose of Cucumber is not to run integration tests. Cucumber, an in general BDD, works as a communication platform, a way to improve the "talk" inside a big team of developers an non-developers that are developing big and complex software. BDD is very useful to communicate a model an its domain at the same level to everybody in the team, even if they don't know anything about computers.

If that is not your scenario, don't use cucumber, because you don't need it. Use rspec and capybara to test your JS code and your integration tests.

aarkerio
  • 1,638
  • 1
  • 15
  • 29
  • Downvoting because this is a common misconception. Cucumber is useful for anything with a UI, even if only the developers ever look at the Cucumber scenarios. That’s because it forces developers into thinking about *user intent*, not *implementation*. Although you *can* do that without Cucumber, you have to be very disciplined and I’ve never seen it work consistently well; Cucumber makes the process much easier. – Marnen Laibow-Koser Sep 22 '18 at 18:22
  • @MarnenLaibow-Koser that is just an opinion, as valid as mine. And is totally false that "Cucumber makes the process much easier." Cucumber was never thought as a test tool, but as a communication platform. – aarkerio Sep 24 '18 at 01:29
  • I’m aware that Cucumber was originally designed as a communication platform, but it makes a fantastic test tool for anything user-facing. I believe it is a fact, not merely an opinion, that user interface needs to be described—and tested—in user terms. I stand by my statement that Cucumber makes the process of testing *in user terms* much easier, because I know it to be true from many years of my own experience. Functional tests in plain RSpec look easier, but don’t actually test at the level we care about (even if they seem to), and therefore are not suitable. – Marnen Laibow-Koser Sep 24 '18 at 11:48
  • If you are using Cucumber as a testing tool, you don't understand TDD neither BDD. – aarkerio Sep 24 '18 at 18:03
  • I understand BDD and TDD well, and I use Cucumber in exactly the way that BDD advises: I start each feature with an executable scenario and write enough code to make it pass (unit testing as I go). Once it passes, though, I consider the story to be a type of acceptance test, and I treat it as such. (I don’t use Cucumber for unit testing, in case that wasn’t clear.) – Marnen Laibow-Koser Sep 24 '18 at 20:32