Code Coverage

We all want to achieve 100% code coverage.  If we have 100% of our code executed in tests, we can feel confident that we are testing our code well. Right?

Not so fast!

We wanted to share some examples of tests that, while they may boost your code coverage, aren’t testing much.  Below are some examples we have actually seen (code has been changed to protect the guilty).

Trust, but verify

This is a code snippet using Mockito.

public void testSomething() {
   classInTest.doStuff();
   verify(classInTest).doStuff();
}

For those of you unfamiliar with what verify does in Mockito…it will pass if the method in question was called.  That’s right…this test is verifying that we called the “doStuff” method…which we did in the previous line.

Unfortunately for us, our code coverage tools will look at this and see that we executed lines inside the doStuff method and count them as covered, even though we didn’t actually test anything!

Preparation is key

public void testStuff() {
  when(thing.doStuff()).thenReturn("hi");
  when(otherThing.doStuff()).thenReturn("there");
  .... lots more setup ....

  classInTest.methodInTest();

}

That’s right…they took the time to do a whole bunch of setup and even called the method in test…but forgot to actually test anything!  There are no asserts or verifies.  Again, our helpful code coverage will show that lines inside of methodInTest were covered since they were executed in a test, even though we didn’t test a single thing.

Green means Go

public void testTest() {
  ... improper setup ...
  ... call method ...
  ... assert the wrong thing ...
}

Sometimes we make mistakes in our setup code and when we run the test, it fails, even though we are sure that the code is working correctly.  That leaves us with two options.  1) Figure out why our test is failing…step through, debug, print stuff out, etc.  See if it is a bug in the production code that we hadn’t noticed or if there is an issue with our test.  OR 2) Change the expected values in our test.

Option 2 is obviously the easier answer.  Yes, we’ve seen this.  No, this is not correct.  Tests are supposed to give you confidence that your code is working correctly.  It can also act as a sort of documentation for future developers.  Do you want future coders to look at the test and think that what you are testing is expected behavior?

What have we learned?

Code coverage is not actually telling the whole story.  As we have seen above, if you have poorly written tests lines of code covered don’t mean anything.  Anyone can be clever and find ways to execute more code in their tests…the smart programmer figures out how to actually test the code.

Please be vigilant in writing your tests and actually ensuring that your code is doing what you expect it to.  When doing code reviews, if you see any blunders like we show above, or other not so great tests, please help the other person write better tests.  You, your team and your code will be better off for it.

photo credit: <a href=”http://www.flickr.com/photos/62975503@N04/8227882239″>Facepalm</a&gt; via <a href=”http://photopin.com”>photopin</a&gt; <a href=”https://creativecommons.org/licenses/by/2.0/”>(license)</a&gt;

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s