Code Coverage

We all want to achieve 100% code coverage.  If we have 100% of our code executed in tests, we can feel confident that we are testing our code well. Right?

Not so fast!

We wanted to share some examples of tests that, while they may boost your code coverage, aren’t testing much.  Below are some examples we have actually seen (code has been changed to protect the guilty).

Trust, but verify

This is a code snippet using Mockito.

public void testSomething() {
   classInTest.doStuff();
   verify(classInTest).doStuff();
}

For those of you unfamiliar with what verify does in Mockito…it will pass if the method in question was called.  That’s right…this test is verifying that we called the “doStuff” method…which we did in the previous line.

Unfortunately for us, our code coverage tools will look at this and see that we executed lines inside the doStuff method and count them as covered, even though we didn’t actually test anything!

Preparation is key

public void testStuff() {
  when(thing.doStuff()).thenReturn("hi");
  when(otherThing.doStuff()).thenReturn("there");
  .... lots more setup ....

  classInTest.methodInTest();

}

That’s right…they took the time to do a whole bunch of setup and even called the method in test…but forgot to actually test anything!  There are no asserts or verifies.  Again, our helpful code coverage will show that lines inside of methodInTest were covered since they were executed in a test, even though we didn’t test a single thing.

Green means Go

public void testTest() {
  ... improper setup ...
  ... call method ...
  ... assert the wrong thing ...
}

Sometimes we make mistakes in our setup code and when we run the test, it fails, even though we are sure that the code is working correctly.  That leaves us with two options.  1) Figure out why our test is failing…step through, debug, print stuff out, etc.  See if it is a bug in the production code that we hadn’t noticed or if there is an issue with our test.  OR 2) Change the expected values in our test.

Option 2 is obviously the easier answer.  Yes, we’ve seen this.  No, this is not correct.  Tests are supposed to give you confidence that your code is working correctly.  It can also act as a sort of documentation for future developers.  Do you want future coders to look at the test and think that what you are testing is expected behavior?

What have we learned?

Code coverage is not actually telling the whole story.  As we have seen above, if you have poorly written tests lines of code covered don’t mean anything.  Anyone can be clever and find ways to execute more code in their tests…the smart programmer figures out how to actually test the code.

Please be vigilant in writing your tests and actually ensuring that your code is doing what you expect it to.  When doing code reviews, if you see any blunders like we show above, or other not so great tests, please help the other person write better tests.  You, your team and your code will be better off for it.

photo credit: <a href=”http://www.flickr.com/photos/62975503@N04/8227882239″>Facepalm</a&gt; via <a href=”http://photopin.com”>photopin</a&gt; <a href=”https://creativecommons.org/licenses/by/2.0/”>(license)</a&gt;

 

Mocking – How should I start?

In a previous post, I talked about what Mocking is, and why you might want to use it.  Now that we think we want to mock our objects, where do we start?

There are plenty of libraries available to allow you to create mock versions of your objects, and if you don’t like any of them, you can always create your own.  For many years, my team created our own mock objects rather than using libraries simply because we felt the libraries didn’t offer some of the things we wanted.  (I will be focusing on Java).

Rolling your own

Creating your own mocks is very straightforward, however it does add some additional overhead that may make your code a little harder for newcomers to understand.

When creating your own mocks, every class that you create that is going to interact with other classes (which is pretty much all of them) needs an interface*.  This will allow us to create our production class, as well as the mock version of our class.  So, it will look something like this:

                             Class <<interface>>
   ProductionClass implements Class       MockClass implements Class

Then, inside our MockClass, we create fake versions of the methods that can capture arguments passed in, additional methods that aren’t part of the interface to set return values, set up methods to throw exceptions, and to get information back out about method calls (like whether it was called, the arguments passed in, etc).

The benefits to creating your own mocks are many.  Creating your own mocks is very simple, you just have to create an interface and implement that, everything else is straight up Java.  Because you have complete control over the mock objects, you have lots of flexibility.  Another great advantage I have seen is that you can set your objects up to have default return values (like empty lists instead of null), which is very beneficial.  (When using Mockito, for instance, if you don’t tell a method what to return, it will return null, so you must set it up every time you use it.)

Since you are writing your own mock classes, you can set breakpoints inside the mock object itself, and inspect the state of the mock object easily (this is not true of Mockito).  Lastly, I think creating your own mock objects at some point in your career is a a good exercise as it gives you a firm understanding of mocks and how they work.

The downside is that there is a fair amount of additional overhead to working this way.  You have a bunch of interfaces that are only useful because you need to create mock objects, and a whole bunch of mock classes that you need to write by hand.  While both of these things are easy to create, it is additional overhead that many teams are unwilling to take on.

* Creating interfaces for your objects is very easy, but it can cause annoyances with naming.  Since most of the objects we create from day to day don’t have variations, and wouldn’t strictly require an interface, you are left with the conundrum of what to name the interface and what to name the implementation class.  We went with the convention of naming our implementation classes with the postfix of Impl.   Not the prettiest solution, but it worked for us.

Mockito (as a surrogate for mocking libraries)

I will admit that the only mocking framework that I have tried is Mockito, so some of my issues with mocking things using a framework are likely to be biased.

The biggest drawback I can see to using a framework to do your mocking is that there is a learning curve (and for Mockito it is a little steep), whereas when you roll your own mocks it is really quite simple.  I think there is a slight sacrifice in readability when using a mocking framework (or maybe it’s just mockito) until you are used to it’s syntax.  For me, Mockito seems to have several different modules mashed together to make a framework, and each has it’s own way of thinking about the problem, making the syntax less cohesive than one would like.

There are several nice benefits to using a framework.  First of all, you don’t have to create a bunch of interfaces just to mock out your objects, so there is less code lying around and it is a little easier to navigate.

Secondly, there is a lot of pre-built functionality in the frameworks so you don’t have to have a bunch of near duplicated code.  When you look at hand-created mocks, you will see a lot of similarities between the mock objects which should really be generalized.  The pre-built functionality ends up meaning that you have a lot less code to maintain, which is huge.

 

Recommendation

 

My recommendation would be that you find a small personal project and write your own mocks with that project just to get a deeper understanding of mocks and how you would do some of the things that a framework will provide you.

But, overall I would recommend using a mocking framework in your production code because it will actually make your life easier in the long run (though the going may be a little rough at first).  You will have less code to maintain and you won’t have to worry about debugging your mock objects (though that is usually not an issue).

Mocking – What is it?

As many of you know, we have a couple of interns at our office this summer.  These guys (yes, they are all guys) have never done any testing before (they are in high school still) and I am rather passionate about testing my code.  So, being a good mentor (ok…so being a selfish person who knows that maintaining the software will fall to me and my team), I suggested that they learn about unit testing and begin applying it to the code they are writing.

As part of their learnings, the interns needed to learn about mock objects, what they are and why you need them.

What is a Mock?

Simply put, a mock object is a faked out version of an object.  You have complete control over what values are returned, and can interrogate it to see what values were passed into it.

Why would we want to do that?!

When writing unit tests, we should be constraining our tests to one method and one path through that method.  But, we know that when we are writing code, often our objects and methods need to access other objects and methods.  This leaves us in a conundrum.  How do we isolate our code?

Here’s a simple example.  Let’s say we have an object that can translate from several languages into English:

public class TranslatorFacade {
  private Translator klingonTranslator;
  private Translator spanishTranslator;

  public TranslatorFacade(Translator klingon, Translator spanish) {
    klingonTranslator = klingon;
    spanishTranslator = spanish;
  }

  public String translateFromKlingon(String phrase) {
    return klingonTranslator.translate(phrase);
  }
  public String translateFromSpanish(String phrase)...

In this example, we are creating a simple Facade that can call a bunch of different translators, but we are using the translator object directly.  We already have tests around each translator, so we don’t want to retest the translator here, but we want to make sure our facade is calling the right methods and is returning the right things.

How do we avoid retesting each translator then?  By mocking out the translators, of course!

Let’s give a short snippet of what that might look like if we wrote our own mock objects.  First, let’s assume we have an interface for translators which looks like this:

public interface Translator {
   public String translate(String phrase);
}

Then, in our test we could do something like this:

private class TestTranslator implements Translator {
  private String translatePhrase;
  private String phraseToReturn;
  public String translate(String phrase) {
    this.translatePhrase = phrase;
    return phraseToReturn;
  }
  public String translate_phrase() {
    return translatePhrase;
  }
  public void translate_return(String phraseToReturn) {
    this.phraseToReturn = phraseToReturn;
  }
}

// WARNING!  I DO NOT KNOW KLINGON
public testTranslateFromKlingon() {
  Translator klingon = new TestTranslator();
  klingon.translate_return("HORRAY");
  TranslatorFacade facade = new TranslatorFacade(klingon, new TestTranslator());
  String english = facade.translateFromKlingon("KA PLA");
  assertEquals("KA PLA", klingon.translate_phrase());
  assertEquals("HORRAY", english);
}

I know there is a lot of code up there.  But what I am trying to show is that we can easily make fake versions of our translators to capture the values that were sent into the object, as well as control the values that were returned from it.

This keeps us from re-testing our implementations when we are using the class somewhere else, which means that when we introduce an error, we are directed very precisely to where the issue is, rather than to all of the classes that happen to be using the failing method.

Seems like a lot of work

The above example is very simplistic.  Imagine that we needed to do something extra with the value that was returned from the translator in our method.  Perhaps we need to censor it or quote it.  We want to make sure that our censoring code is working as we expected.  By introducing mock objects, we can be sure that we are testing just that by setting our return values to strings that contain words that need censoring as well as ones that do not without having to know or understand the implementation of the translator class.

Let me state that again…because I think it is important.  By mocking the objects we are using, we don’t have to know or understand the implementation of the objects we are using!  Think about how many libraries you use in your day to day coding.  Being able to test your code without knowing the details of the code you are using is a big deal!

Do I have to write my own mocks?

No.  In fact, there are a lot of frameworks out there that will help you create mocks very easily.  Mockito and PowerMock are two that come to mind off the top of my head, and I know that there are many more out there.