21 May 2008

Got Tests?

by mo

A couple of months ago I finished reading…

xUnit Test Patterns: Refactoring Test Code (The Addison-Wesley Signature Series)
by Gerard Meszaros

Read more about this book...

This was a thick book, that discusses unit test smells, unit test refactorings, unit test patterns… and just about anything else related to unit testing. Here’s a little of what I’ve learned from this book…

Defect Localization

Mistakes happen! Of course, some mistakes are much more expensive to prevent than to fix. Suppose a bug does slip through somehow and shows up in the Integration Build. If our unit test are fairly small (i.e., we test only a single behavior in each one), we should be able to pinpoint the bug quickly based on which test fails. This specificity is one of the major advantages that unit tests enjoy over customer tests. The customer tests tell us that some behavior expected by the customer isn’t working; the unit tests tell us why. We call this phenomenon Defect Localization. If a customer test fails but no unit tests fail, it indicates a Missing Unit Test.

Tests as Documentation

Without automated tests, we would need to pore over the SUT code trying to answer the question, ‘What should be the result if …?’ With automated tests, we simply use the corresponding Tests as Documentation; they tell us what the result should be. If we want to know how the system does something, we can turn on the debugger, run the test, and single-step through the code to see how it works. In this sense, the automated tests act as a form of documentation for the SUT.

When it is not important for something to be seen in the test method, it is important that it not be seen in the test method!

Test Doubles

A test double is any object or component that we install in place of the real component for the express purpose of running a test. Depending on the reason why we are using it, a Test Double can behave in one of four ways.

  • Dummy Object: a object that is passed to the SUT as an argument but is never actually used.
  • Test Stub: an object that replaces a real component on that the SUT depends on so that different inputs can by applied against the SUT.
  • Test Spy: an object that can act as an observation point for the indirect outputs of the SUT.
  • Mock Object: an object that replaces a real component that the SUT depends on to test the and verify indirect outputs.
  • Fake Object: an object that replaces the functionality of the real SUT dependency with an alternate implementation that provides the same functionality.

Strict vs Loose

Mock objects come in two basic flavors:

  • Strict Mock: fails the test if incorrect calls are received.
  • Loose (lenient) Mock: fails if expected calls are not received, but is lenient if additional calls are received.

Need-Driven Development

This ‘outside-in’ approach to writing and testing software combines the conceptual elegance of the traditional ‘top-down’ approach to writing code with modern TDD techniques supported by Mock Objects. It allows us to build and test the software layer by layer, starting at the outermost layer before we have implemented the lower layers.

Test Smells

“Developers Not Writing Tests may be caused by an overly aggressive development schedule, supervisors who tell developers not to ‘waste time writing tests,’ or developers who do not have the skills to write tests. Other potential causes might include an imposed design that is not conducive to testing or a test environment that leads to Fragile Tests. Finally, this problem could result from Lost Tests - tests that exist but are not included in the All Tests Suite used by developers during check-in or by the automated build tool.”

“Another productivity-sapping smell is Frequent Debugging. Automated unit tests should obviate the need to use a debugger in all but rare cases, because the set of tests that are failing should make it obvious why the failure is occurring. Frequent Debugging is a sign that the unit tests are lacking in coverage or are trying to test to much functionality at once.”

Fragile Test: “A test fails to compile or run when the SUT is changed in ways that do not affect the part the is exercising… Fragile tests increase the cost of test maintenance by forcing us to visit many more tests each time we modify the functionality of the system or the fixture.”

This headache is typical if you’re working with strict mock objects. I experienced this pain when working on a project using NMock. I couldn’t find a clean separation between strict and loose mocks using NMock. There was only the concept of Strict Mocks and Stubs.

Slow Tests: “The tests take too long to run… They reduce the productivity of the person running the test. Instead, the developers wait until the next coffee break or another interruption before running them. Or, whenever they run the tests, they walk around and chat with other team members…”

The main disadvantages of using Fit are described here:

  • The test scenarios need to be very well understood before we can build the Fit Fixture. We then need to translate each test’s logic into a tabular representation; this isn’t always a good fit.
  • The tests need to employ the same SUT interaction logic in each test. To run several different styles of tests, we would probably have to build one or more different fixtures for each style of test. Building a new fixture is typically more complex than writing a few Test Methods.
  • Fit tests aren’t normally integrated into developers’ regression tests that are run via xUnit. Instead, these test must be run separately - which introduces the possibility that they will not be run at each check-in.


A concept from lean manufacturing that states that things should be produced only once a real demand for them exists. In a ‘pull system,’ upstream assembly lines produce only enough products to replace the items withdrawn from the pool that buffers them from the downstream assembly lines. In software development, this idea can be translated as follows: ‘We should only write methods that have already been called by other software and only handle those cases that the other software actually needs.’ This approach avoids speculation and the writing of unnecessary software, which is one of software development’s key forms of inventory (which is considered waste in lean systems).

books tdd