Code Coverage

We all want to achieve 100% code coverage.  If we have 100% of our code executed in tests, we can feel confident that we are testing our code well. Right?

Not so fast!

We wanted to share some examples of tests that, while they may boost your code coverage, aren’t testing much.  Below are some examples we have actually seen (code has been changed to protect the guilty).

Trust, but verify

This is a code snippet using Mockito.

public void testSomething() {
   classInTest.doStuff();
   verify(classInTest).doStuff();
}

For those of you unfamiliar with what verify does in Mockito…it will pass if the method in question was called.  That’s right…this test is verifying that we called the “doStuff” method…which we did in the previous line.

Unfortunately for us, our code coverage tools will look at this and see that we executed lines inside the doStuff method and count them as covered, even though we didn’t actually test anything!

Preparation is key

public void testStuff() {
  when(thing.doStuff()).thenReturn("hi");
  when(otherThing.doStuff()).thenReturn("there");
  .... lots more setup ....

  classInTest.methodInTest();

}

That’s right…they took the time to do a whole bunch of setup and even called the method in test…but forgot to actually test anything!  There are no asserts or verifies.  Again, our helpful code coverage will show that lines inside of methodInTest were covered since they were executed in a test, even though we didn’t test a single thing.

Green means Go

public void testTest() {
  ... improper setup ...
  ... call method ...
  ... assert the wrong thing ...
}

Sometimes we make mistakes in our setup code and when we run the test, it fails, even though we are sure that the code is working correctly.  That leaves us with two options.  1) Figure out why our test is failing…step through, debug, print stuff out, etc.  See if it is a bug in the production code that we hadn’t noticed or if there is an issue with our test.  OR 2) Change the expected values in our test.

Option 2 is obviously the easier answer.  Yes, we’ve seen this.  No, this is not correct.  Tests are supposed to give you confidence that your code is working correctly.  It can also act as a sort of documentation for future developers.  Do you want future coders to look at the test and think that what you are testing is expected behavior?

What have we learned?

Code coverage is not actually telling the whole story.  As we have seen above, if you have poorly written tests lines of code covered don’t mean anything.  Anyone can be clever and find ways to execute more code in their tests…the smart programmer figures out how to actually test the code.

Please be vigilant in writing your tests and actually ensuring that your code is doing what you expect it to.  When doing code reviews, if you see any blunders like we show above, or other not so great tests, please help the other person write better tests.  You, your team and your code will be better off for it.

photo credit: <a href=”http://www.flickr.com/photos/62975503@N04/8227882239″>Facepalm</a&gt; via <a href=”http://photopin.com”>photopin</a&gt; <a href=”https://creativecommons.org/licenses/by/2.0/”>(license)</a&gt;

 

Hiding utilities behind a facade

Recently, at work, I got into a passionate discussion about whether we should hide things like StringUtils behind a facade.

I was helping a coworker add some tests around a monstrous method and noted that it used StringUtils static methods quite a few times in the method.  Initially I thought that this was a utility class that we had written, so rather than changing the class to have non-static methods, I opted to try to hide it behind a facade (in this case it was a simple pass-through object, but the methods on this object were non-static, thus allowing me to mock it with Mockito).  Upon investigation, it turned out we were using commons.lang StringUtils.  Still, I thought it might be good to put it behind a real object so we could mock it out.

I explained to my colleague that the reason I wanted to do this was so that we didn’t have to set up our tests with realistic data.  We could pass in very fake data and we could make sure that we were only testing the method in question (which was VERY important since it was a behemoth).  By putting it behind the facade, we were able to make it return whatever we wanted.

We spent the morning getting this all set up and she was ready to start off on her own in the afternoon.  Later that afternoon, she was talking with the other developer (and our boss), who works in the code base in question regularly about the tests she was writing.  He did not agree at all about us mocking out StringUtils.

A very heated discussion ensued.  In the end, since I didn’t work in the code in question, I left it up to them how they wanted to test it.

After the discussion, I was reflecting on why I felt so strongly about mocking out StringUtils, and I think I have pinpointed some of the reasons.

  1. The method under test was doing a LOT of things, including calling out to StringUtils nearly a dozen times for different things.  By mocking out StringUtils I felt more comfortable that we were tackling the code that was in the method without having to worry about what StringUtils was going to do with the data we passed it.  I was trying to reduce the complexity of the method in question by ensuring I was using things I had complete control of.
  2. I saw StringUtils as a collaborator and wanted to make sure that I was testing only the method in question, and not the collaborating code.  I believe I saw it this way due to point 1.
  3. I knew that we had some other classes in the code base that had a bunch of static methods on them.  I would treat these the same way.  Rather than treating each different class with static methods differently based on some gut feeling, I wanted to be consistent and treat any class that offered static methods the same way.  This allows me to focus on the tests, rather than spending mental cycles on deciding whether this particular things should be mocked or not.
  4. Hiding a 3rd party tool behind a facade is a best practice I have heard many times.  It allows you to swap out the tool for a new one with little effort.  It can also make it much easier to upgrade to a new version, even if the new version has changed the API, since if everyone using it is actually using the facade, you only need to make the change in one place, and everyone else just continues using the facade in the same way as they always had.

 

 

Simply refactoring the monolithic method into smaller methods that could do each individual task would have left me testing it much differently.

Unit Testing – Part 1

I’m fairly new to writing automated unit tests. I’ve read about it and I’ve tried it out and we even have a handful of java classes (10 to 20) that have full coverage from unit tests. But now it looks like we might be changing our view of testing and diving in head first.
So here are some things that I’ve learned that I didn’t really pick-up on when reading the books about JUnit.

Step 1 – Write Testable Code
Something that I missed when reading books about unit testing is that you have to change the way you write the code in the first place. This is generally a good thing, but something that is difficult to do across an entire development team.
Long methods – are very difficult to test – We have some methods that are over 1000 lines, it’s not easy (maybe not even possible) to test a method that is this big.

Dependency Injection – methods that create their own database connections, or create any new objects directly are very difficult to test well. For legacy code, I’ve found that a good way to handle this is updating the method to take these objects as parameters, and then creating a new method with the old signature that simply creates the objects and passes them in. This allows you to create mock objects for database connections, etc. which lets you test all the code in the method without needing a database to connect to.
ex:
public User buildUser() {
Connection conn = getConnection();
return buildUser(conn);
}
public User buildUser(Connection conn) {
... existing/old code to read user object from databse ...
return user;
}

References to Static Methods – Static methods are easy to test. Code that calls static methods are VERY hard to test, because you can’t mock/stub out the static methods. So you end up having to test all the code in the static method along with the code in the method you actually want to test. I don’t think small methods that take few or no parameters from well tested libraries are much of an issue. For example, if your code calls Math.random(), it’s still fairly easy to test. But if you have a lot of “utility” methods that take large objects as parameters or that do complicated logic it’s difficult to mock-up the test data to get those static methods to return all the different scenarios that allow you to run through all the code in the method under test.

If developers are writing code that is difficult to test, they will hate writing unit tests. It is time consuming, the tests will break often from small changes in the code, and code coverage will be low. They need to change the way they write the code in the first place before you can be successful in implementing unit tests.

In general, testable code is better quality code anyway. Having short methods that “Do one thing”, and creating methods that use Object Oriented Design (ie: not static) are easier to maintain, easier to read, and much nicer to work with. This is a side benefit of implementing unit tests.

I’ll continue with my lessons learned in later posts. Something I’m still not sure how we are going to handle is all the existing legacy code that doesn’t fall into the category of “testable” or at least not “easily testable”.

Mocking – How should I start?

In a previous post, I talked about what Mocking is, and why you might want to use it.  Now that we think we want to mock our objects, where do we start?

There are plenty of libraries available to allow you to create mock versions of your objects, and if you don’t like any of them, you can always create your own.  For many years, my team created our own mock objects rather than using libraries simply because we felt the libraries didn’t offer some of the things we wanted.  (I will be focusing on Java).

Rolling your own

Creating your own mocks is very straightforward, however it does add some additional overhead that may make your code a little harder for newcomers to understand.

When creating your own mocks, every class that you create that is going to interact with other classes (which is pretty much all of them) needs an interface*.  This will allow us to create our production class, as well as the mock version of our class.  So, it will look something like this:

                             Class <<interface>>
   ProductionClass implements Class       MockClass implements Class

Then, inside our MockClass, we create fake versions of the methods that can capture arguments passed in, additional methods that aren’t part of the interface to set return values, set up methods to throw exceptions, and to get information back out about method calls (like whether it was called, the arguments passed in, etc).

The benefits to creating your own mocks are many.  Creating your own mocks is very simple, you just have to create an interface and implement that, everything else is straight up Java.  Because you have complete control over the mock objects, you have lots of flexibility.  Another great advantage I have seen is that you can set your objects up to have default return values (like empty lists instead of null), which is very beneficial.  (When using Mockito, for instance, if you don’t tell a method what to return, it will return null, so you must set it up every time you use it.)

Since you are writing your own mock classes, you can set breakpoints inside the mock object itself, and inspect the state of the mock object easily (this is not true of Mockito).  Lastly, I think creating your own mock objects at some point in your career is a a good exercise as it gives you a firm understanding of mocks and how they work.

The downside is that there is a fair amount of additional overhead to working this way.  You have a bunch of interfaces that are only useful because you need to create mock objects, and a whole bunch of mock classes that you need to write by hand.  While both of these things are easy to create, it is additional overhead that many teams are unwilling to take on.

* Creating interfaces for your objects is very easy, but it can cause annoyances with naming.  Since most of the objects we create from day to day don’t have variations, and wouldn’t strictly require an interface, you are left with the conundrum of what to name the interface and what to name the implementation class.  We went with the convention of naming our implementation classes with the postfix of Impl.   Not the prettiest solution, but it worked for us.

Mockito (as a surrogate for mocking libraries)

I will admit that the only mocking framework that I have tried is Mockito, so some of my issues with mocking things using a framework are likely to be biased.

The biggest drawback I can see to using a framework to do your mocking is that there is a learning curve (and for Mockito it is a little steep), whereas when you roll your own mocks it is really quite simple.  I think there is a slight sacrifice in readability when using a mocking framework (or maybe it’s just mockito) until you are used to it’s syntax.  For me, Mockito seems to have several different modules mashed together to make a framework, and each has it’s own way of thinking about the problem, making the syntax less cohesive than one would like.

There are several nice benefits to using a framework.  First of all, you don’t have to create a bunch of interfaces just to mock out your objects, so there is less code lying around and it is a little easier to navigate.

Secondly, there is a lot of pre-built functionality in the frameworks so you don’t have to have a bunch of near duplicated code.  When you look at hand-created mocks, you will see a lot of similarities between the mock objects which should really be generalized.  The pre-built functionality ends up meaning that you have a lot less code to maintain, which is huge.

 

Recommendation

 

My recommendation would be that you find a small personal project and write your own mocks with that project just to get a deeper understanding of mocks and how you would do some of the things that a framework will provide you.

But, overall I would recommend using a mocking framework in your production code because it will actually make your life easier in the long run (though the going may be a little rough at first).  You will have less code to maintain and you won’t have to worry about debugging your mock objects (though that is usually not an issue).

Mocking – What is it?

As many of you know, we have a couple of interns at our office this summer.  These guys (yes, they are all guys) have never done any testing before (they are in high school still) and I am rather passionate about testing my code.  So, being a good mentor (ok…so being a selfish person who knows that maintaining the software will fall to me and my team), I suggested that they learn about unit testing and begin applying it to the code they are writing.

As part of their learnings, the interns needed to learn about mock objects, what they are and why you need them.

What is a Mock?

Simply put, a mock object is a faked out version of an object.  You have complete control over what values are returned, and can interrogate it to see what values were passed into it.

Why would we want to do that?!

When writing unit tests, we should be constraining our tests to one method and one path through that method.  But, we know that when we are writing code, often our objects and methods need to access other objects and methods.  This leaves us in a conundrum.  How do we isolate our code?

Here’s a simple example.  Let’s say we have an object that can translate from several languages into English:

public class TranslatorFacade {
  private Translator klingonTranslator;
  private Translator spanishTranslator;

  public TranslatorFacade(Translator klingon, Translator spanish) {
    klingonTranslator = klingon;
    spanishTranslator = spanish;
  }

  public String translateFromKlingon(String phrase) {
    return klingonTranslator.translate(phrase);
  }
  public String translateFromSpanish(String phrase)...

In this example, we are creating a simple Facade that can call a bunch of different translators, but we are using the translator object directly.  We already have tests around each translator, so we don’t want to retest the translator here, but we want to make sure our facade is calling the right methods and is returning the right things.

How do we avoid retesting each translator then?  By mocking out the translators, of course!

Let’s give a short snippet of what that might look like if we wrote our own mock objects.  First, let’s assume we have an interface for translators which looks like this:

public interface Translator {
   public String translate(String phrase);
}

Then, in our test we could do something like this:

private class TestTranslator implements Translator {
  private String translatePhrase;
  private String phraseToReturn;
  public String translate(String phrase) {
    this.translatePhrase = phrase;
    return phraseToReturn;
  }
  public String translate_phrase() {
    return translatePhrase;
  }
  public void translate_return(String phraseToReturn) {
    this.phraseToReturn = phraseToReturn;
  }
}

// WARNING!  I DO NOT KNOW KLINGON
public testTranslateFromKlingon() {
  Translator klingon = new TestTranslator();
  klingon.translate_return("HORRAY");
  TranslatorFacade facade = new TranslatorFacade(klingon, new TestTranslator());
  String english = facade.translateFromKlingon("KA PLA");
  assertEquals("KA PLA", klingon.translate_phrase());
  assertEquals("HORRAY", english);
}

I know there is a lot of code up there.  But what I am trying to show is that we can easily make fake versions of our translators to capture the values that were sent into the object, as well as control the values that were returned from it.

This keeps us from re-testing our implementations when we are using the class somewhere else, which means that when we introduce an error, we are directed very precisely to where the issue is, rather than to all of the classes that happen to be using the failing method.

Seems like a lot of work

The above example is very simplistic.  Imagine that we needed to do something extra with the value that was returned from the translator in our method.  Perhaps we need to censor it or quote it.  We want to make sure that our censoring code is working as we expected.  By introducing mock objects, we can be sure that we are testing just that by setting our return values to strings that contain words that need censoring as well as ones that do not without having to know or understand the implementation of the translator class.

Let me state that again…because I think it is important.  By mocking the objects we are using, we don’t have to know or understand the implementation of the objects we are using!  Think about how many libraries you use in your day to day coding.  Being able to test your code without knowing the details of the code you are using is a big deal!

Do I have to write my own mocks?

No.  In fact, there are a lot of frameworks out there that will help you create mocks very easily.  Mockito and PowerMock are two that come to mind off the top of my head, and I know that there are many more out there.