November 22, 2012 Book Reviews, UnitTesting No comments
Book Review: “The Art of Unit Testing: With Examples in .Net”
I always wanted to read some comprehensive book on unit testing, therefore book “The Art of Unit Testing: With Examples in .Net” appeared in my list of books to read. It has been more than 1 year since I planned to read it, finally it took me only 2 days to read it – yesterday and the day before yesterday. Sometimes comprehensive things are not that comprehensive as you imagine them or… maybe with time topic is no longer that vast for you.
Nevertheless, I remember my first months of my first job as Software Developer. During one of my first team meetings question of failed Unit Tests was raised. Our clients were really concerned about quantity of tests and reasons behind tests failing during nightly builds. I clearly remember that passionate discussion and someone saying “… or are there guys who don’t know what Unit Test is?” and loud laugh after that. I made smile on my face. I maybe heard once or twice about such combination of words, resulting in abbreviation UT, but it didn’t ring a bell. I wasn’t brave enough to ask. Now I think I wasn’t one in that room who didn’t know, otherwise why would we had problems in the first place?
Because of coincidences and my eagerness, when I was about to leave many considered me as internal expert in Unit Testing. No surprises, I have many posts on UnitTesting.
Reading this book forced me to think twice about some of the testing techniques I use. Probably I changed my mind about some code I write, both in test and outside. I took some notes for myself.
Now bit of critics
Good indicator if book is for beginners is if it has something like “right click on…” – Yes, this book has few mouse clicks.
Roy introduces many definitions. Some of them are bit contradictional. For example, it might look like there are fakes, stubs and mocks while reading the book, but at some point of time you read this “A fake is a generic term that can be used to describe either a stub or a mock object (handwritten or otherwise), because they both look like the real object.” At least Martin Fowler thinks that fake is different from stub and mock. It is something I also don’t like about definitions in UnitTesting, they are blurred. Every faking mocking framework has its own interpretation and word “mock” is too much overloaded.
As per me, there are some things, which might look not completely professional for the book writer. I mean phrases like this one: “When all else fails and your code is hard to test, you have three choices: use a “super” framework like Typemock Isolator, change the design, or quit your job.” I hope you know why Typemock is highlighted as well. :)
Sometimes stuff is over explained. For example, “One test calling another breaks isolation and introduces a dependency” and then goes explanation why this is bad idea with lots of bullet points. Well, I wouldn’t even think about mentioning stupidity of doing such things.
Book “Working Effectively with Legacy Code” is referenced really often. It worth reading it. I even have hardcopy of it, but only read few pages.
Book is really easy to read. It is straight, well structured and formatted. I enjoyed reading it.
My personal list of things I can improve about my unit testing
Caution: this list is my personal, for you it might look completely different.
– I tend to write extremely testable code, which often results in overengineered design with low percentage of “private” code. Roy recommends to not test methods, which are better left private to code under test.
– Stop overspecifying unit tests.
– One mock per test. I would like to violate this rule less frequently.
– Ideally one assert per test.
– Think more about applying some global testing techniques, like separation of integration tests from unit tests, or fast from slow, etc.
– Utilize more syntaxes and capabilities of testing/mocking frameworks.
Do I recommend the book?
In short, YES. If you are just starting with Unit Tests this book is excellent for your learning and I highly recommend it. If you have developers in your team who need to learn Unit Testing give them this book. Finally, read it yourself, at least scroll parts 1 and 2, read part 3 and slowly scroll part 4.
100% Code Coverage – real and good!
April 18, 2012 Opinion, UnitTesting No comments
I’m not going to write a long post discussing advantages and disadvantages of high code coverage. There are hundreds if not thousands of such posts out there and in the end almost all of them conclude that high code coverage in general is nice but not always justifiable, one of main reasons being redundant abstractions in favor of higher coverage. Here are my recent thoughts.
Achieve 100% Code Coverage by all means
It may sound crazy and not doable at all(*) or may have side effects if misused. I suggest very simple techniques to achieve high code coverage the right way:
Don’t be lazy. Recently I worked on a project and I already had 95% coverage. If I haven’t decided to increase coverage further I wouldn’t have found one missing mapping for a property. I maybe spent couple of hours to write more tests, instead of days of devs/testers/managers time to work around the bug. And in case of finding this in production it would cost real money for the company.
Work around external dependencies you really cannot test. Isolate them as much as you can and simply exclude from coverage report. I don’t think this is cheating. It is the best you can do, plus you do it explicitly. And, of course, you should have integration tests to test external dependencies.
Remember the Single Responsibility. Well… and few more things. You will be amazed how much code is simpler and easier to read if you just keep following SOLID. I think that developer should be able to clearly describe responsibility of a single class within one sentence.
Start with testing in mind, not with coverage number. It is vital to keep in mind that tests are intended to ensure you code works as designed and without defects, tests are NOT intended for high coverage numbers, which can be shown to boss. Thus always have tests to cover more important and sensitive code at first and only then move towards covering less important or easy to test code.
Refactor! Never write code you don’t like. It is fine to hate it the next day, but not at the moment when you are writing it. Usually crap code starts to appear when you try to add functionality which was not planned before. You must refactor constantly (same is applicable for your unit tests). Keep everything in synch.
Be a 100% good programmer. Don’t spoil yourself with 80% coverage or with just 60%. If someone says you are 75% good programmer, would you like it? Well, it is high number, isn’t it? I was worse developer few months ago than I’m today. Year ago I would disagree with today’s myself. High coverage, if used right, means that you know that your code works and that it is readable/refactorable/decoupled/structured/… and most of all – it is highly maintainable.
I hope my opinion sounds sensible!
Till next time…
Further reading:
-
I strongly recommend to read this paper: “How to Misuse Code Coverage”
-
There are research papers on this matter. In “Experiments of the effectiveness of dataflow and control flow-based test adequacy criteria” authors “evaluate all-edges and alluses coverage criteria using an experiment with 130 fault seeded versions of seven programs and observed that test sets achieving coverage levels over 90% usually showed significantly better fault detection than randomly chosen test sets of the same size. In addition, significant improvements in the effectiveness of coverage-based tests usually occurred as coverage increased from 90% to 100%.” – from MS research paper.
(*) I could agree with many exceptional situations you are thinking about. I would agree that with old systems it is difficult to do what I ask you to, I would also agree that if there are deadlines it is hard to stand against. There are bad programmers around, bad decisions taken and many other conditions. In the end it is your job to do the job right. And if you cannot, change the company or change the company.
WP7 Mango and NUnit from console
October 22, 2011 HowTo, NUnit, UnitTesting, WP7 3 comments
If you are building Windows Phone application you probably have faced… well… not rich support for unit testing. But of course there is one rescuing thing – WP7 is almost the same Silverlight application. Thus you have wider area to search for solution. Awesome… but I wouldn’t write this blog post if everything is that easy. Right?
Support for NUnit and command line
Microsoft for some reason doesn’t care about huge community of those who use NUnit for their projects, and believe me not for small projects. So there of course is mstest project template that allows you to run tests inside of the appropriate CLR. There is good Silverlight Unit Tests Framework and here is information on how you can cheat in order to get it working for the phone. Problem with these two frameworks is obvious – they are not supporting console – they run in native for Silverlight environment – either on phone or in web. See pictures (phone and web respectively):
I know that there are ways to make it happen from console under (msbuild for example this temporary wp7ci project on codeplex). Hold on… one second. Again something very specific to Microsoft – msbuild. But what if I’m using nAnt?
Of course there is port of the NUnit for the Silverlight (here is how), also you can change tests provider in the “Silverlight Unit Tests Framework” (further SUTF).
Nevertheless summary is simple – no support for running nunit tests for the WP7 from command line.
Support for command line – solution
I came up with odd solution to force nunit-console to run unit tests in command line. After I observed it crashing with error TargetFrameworkAttribute I reflected mscorlib and googled a bit to discover this attribute exists in mscorlib of 2.0.5.0 version, but nunit actually targets 2.0.0.0 one (.net 2.0). Thus I decided to download sources of NUnit and recompiled those against .net framework 4.0 (mscorlib 2.0.5.0). Reason for this error is that Silverlight also uses higher version of mscorlib.
Awaiting for NUnit 3.0 which is promising to have support for Silverlight.
Support for Mango – problem
Before upgrading to Mango our tests for WP7 were created by testDriven (to be honest – it is what they use inside of their SUTF). We didn’t have support for command line and tests were running only because they are so Silverlight compatible.
With updating to Mango everything just died. Tests projects simply didn’t get loaded into solution. With this message:
“Not a big deal” you say. Actually a big deal, because you can get rid of this message by editing project files to have target framework profile set to WP71 and to reference only WP71 assemblies. But in this case you lose all of you compatibility with Silverlight and when you run your tests you start to get weirdest exceptions in the world like this one below:
System.DivideByZeroException : Attempted to divide by zero.
at System.Collections.ObjectModel.ObservableCollection`1..ctor()
At least this brings some more sense:
System.InvalidProgramException : Common Language Runtime detected an invalid program.
Support for Mango – solution
Solution I came up with is not 100% right, but at least it works. I just had to pretend that I’m still completely compatible with Silverlight. So I created copy of my project. One is considered to be used from command line and other for usage from VS.
Project 1 is used under VS has correct references directly to WP71 assemblies, like below:
<Reference Include="System.Windows">
<HintPath>..LibrarySilverlightWP71ReferencesSystem.Windows.dll</HintPath>
</Reference>
This ensure that VS loads your projects without errors, also you make it think it is WP7 by these:
<TargetFrameworkProfile>WindowsPhone71</TargetFrameworkProfile>
and this:
<Import Project="$(MSBuildExtensionsPath)MicrosoftSilverlight for Phone$(TargetFrameworkVersion)Microsoft.Silverlight.$(TargetFrameworkProfile).Overrides.targets" />
<Import Project="$(MSBuildExtensionsPath)MicrosoftSilverlight for Phone$(TargetFrameworkVersion)Microsoft.Silverlight.CSharp.targets" />
Project 2 is used for console and is pretended to be pure Silverlight, so it has:
<Reference Include="System.Windows">
<Private>True</Private>
</Reference>
Which in reality copies (because of <Private>) wrong assemblies – from Silverlight, not from phone.
You would need to play a lot with which assemblies you want to get in folder where you run tests. I do have some confidence that Silverlight and WP7 are much compatible thanks to this brilliant explanation of what is WP7 from developer’s view.
Results
At #1186 I finally got build working and running tests. At #1193 I invented this Silverlight pretending trick. And finally till build number #1196 I ignored couple of incompatible tests and fixed some really failing tests.
Hope this helps someone. At least it is going to help my team.
Continuous Integration – Test Environment
September 4, 2010 CI, Environment, UnitTesting 6 comments
Couple of days ago I’ve been on CI seminar, where guy had been talking about concepts of the Continuous Integration and at some moment mentioned about testing environment and it suddenly dawned upon me and I would like to share my thoughts on this.
Two of many concepts of the CI are:
- Have repository with all needed to build product on the virgin machine using command line.
- Unit Tests, Integration Tests and Deployment should run on environment similar to production.
Feel the difference?
Ok, why do I ask about this?
Maybe a week ago we’ve moved to approach that builds sources and runs unit tests on absolutely virgin machine. Simply saying, nothing is installed into GAC, except of maybe .net framework. Everything needed we take from library folders under source control. And I agree that is really good thing. Compiler is also ok to build projects when they have reference only to root assembly in folder of other assemblies of some particular component. BUT, when you execute your code (run tests) CRL tries to find assemblies in execution folder and then in GAC. Since we have nothing in GAC, we should ensure that everything gets copied to execution folder, and here we dive into issues.
Fixes
In case of simple referencing this is not a big issue. You just should be as a spider and catch what is missed on enigma CI build machine. In some cases this is very trivial – you see System.IO.FileNotFoundException, in other cases it is not obvious.
Interesting trick
Here is one of tricts that we were needed to apply to make UT happy:
Since dll-s like sqlceme.dll are part of SQL CE, but are not CLR dlls we cannot reference them. So we add them as link in your project and then change Build property to “Copy Always” to have them in bin.
There were some other tricks we were needed to try. They are simple, but there are many.
Main Question is still remaining
All that is not main intent of this post. Main is this: Do I really need to do this. Why is this correct?
And if we are so dedicated to CI, then I would like to know also what are we testing? That our code is able to run even if none of 3rd party components are installed?
I still have some doubts, maybe I’m wrong. I would really appreciate any of your thoughts, comments that will help me figure it out and find the right way.
For fun (but maybe will get lucky) I also asked Martin Fowler in twitter:
Pair Programming – First Try
August 25, 2010 TDD, TeamWork, UnitTesting No comments
Few days ago I worked few hours on Sunday, so there were only me and one of my co-workers in the room. He is Delphi guy, but likes to learn .NET. He had to implement some custom resizing logic for the report and he asked me if I can help to think on the logic. We had spent maybe couple of minutes before we got some preliminary ideas.
Let’s do Pair Programming
We decided to implement some prototype/logic within .net. I suggested pair programming with TDD: “I will be writing tests – you will be implementing logic to make them work”. BTW, do you know how do we call this particular programming pattern? – It is called “Ping-Pong Pair Programming“.
So after I created basic test project and wrote a test that simply has something like “var report = new Report();” and then “report.Draw();“, which of course is failing UT, he proceeded with creating class for Report and putting in place stub method Draw. Then I added functionality that allows adding report columns, then sizing of them, desiredMinimalWidth and delimiting width. Then UT that verifies if we correctly resize columns, if they fit or do not fit into report. Misha was performing extremely well in implementing logic, we had been thinking on. At some point of time he got stuck, so we continued together on making latest UT work.
I enjoyed this try of doing Pair Programming along with TDD.
What are benefits of doing PP as per me?
- Knowledge sharing
- Less bugs
- Quicker problem resolving when working on complex tasks
- More control on what other in pair is doing (in other words you will be sure that he is not using his time to watch YouTube)
- Building great team spirit
Are there disadvantages?
It might happen that PP will not be feasible, especially in situations when problem they are trying to solve is trivial or when one of contributors is not interested in going great job. You can read lot of discussion over the internet about the time PP consumes and the quality of the work. Most commonly you will hear that when doing PP it takes +15% of time if two developers work independently and produces 15% fever bugs.
What are preconditions on doing Pair Programming as per me?
- Problem you going to solve are complex and require writing new code
- Good to have two competent developers, but one of developers is a must
- One of the developers is really dedicated to work
- A great plus would be if your system allows you to write Unit Tests
- You bet on quality, not on quantity (sometimes better decision for the business might be to deliver quick and lot)
In my further professional growth I will try to involve more people in this practice!
Using NHibernate Profiler with Unit Tests
July 5, 2010 NHibernate, QuickTip, UnitTesting No comments
Today, I’ve been writing few methods for my DAL classes, of course, I decided to have one or few integration unit tests that hits database and see actual SQLs with NHibernate Profiler.
So in couple of minetes I’ve got unit tests that had following line in the beggining of each:
HibernatingRhinos.Profiler.Appender.NHibernate.NHibernateProfiler.Initialize();
if you don’t know, this line attaches your execution code with NHibernate Profiler, so I’m able to see SQLs NHibernate generates.
When I run bunch of unit tests in my testing file, I’ve got strange picture with duplicating of queries for each test with arithmetic progression. And N+1 problem, but hold on, I’m sure that I did everything through joins.
Reason is that profiler appends to my code on each new test run and that is why it start thinking that I have multiple selects to get list of something.
Solution, which I would recommend as pattern for writing Unit Tests with NHibernate Profiler is following:
[SetUp]
public void SetUp()
{
HibernatingRhinos.Profiler.Appender.NHibernate.NHibernateProfiler.Initialize();
}
[TearDown]
public void TearDown()
{
HibernatingRhinos.Profiler.Appender.NHibernate.NHibernateProfiler.Stop();
}
Few simple mocking scenarios
July 3, 2010 RhinoMocks, TDD, UnitTesting No comments
Are you married? Yeah, quite not usual question for blog post about mocking, but if you are married you probably asked your girl to become your wife.
So you are playing role of PotentialHasband:
public interface IPotentialHusband
{
string
AskForHerHandInMarriage();
AskForHerHandInMarriage();
}
and she is PotentialWife, taking your ask into consideration and answering you true or false.
public interface IPotentialWife
{
bool
GiveHimAnswer(string hisSpeech);
GiveHimAnswer(string hisSpeech);
}
If she is Linda, you would need to have “I love you” in your speech to get her agreement:
public class Linda : IPotentialWife
{
public
bool GiveHimAnswer(string hisSpeech)
bool GiveHimAnswer(string hisSpeech)
{
return hisSpeech.Contains(“I love you”);
}
}
Btw, if your wife is indeed Linda, you just need put some comments below :)
What we are going to test is to see that after you asked, she will definitely give you at least some answer and not ignore you. Also if her answer is “YES” we would like to see MarriageService called Marry method.
public class MarriageService
{
public virtual void
Marry(IPotentialHusband
almostHusband, IPotentialWife
almostWife)
Marry(IPotentialHusband
almostHusband, IPotentialWife
almostWife)
{
Console.WriteLine(“You are now husband and wife!”);
}
}
Class which we are going to test is named LearningMocking and has constructor that accepts potential husband/wife and marriageService. Whole story takes place in RestaurantDinner method.
First let’s test that after you asked she gives at least some answer. Since we are not interested in concrete instances we just generate mocks for our interfaces, and passing null instead of marriage service. I’m using RhinoMocks in this example.
[Test]
public void EnsureThatAfterHeAskedSheAnswers()
{
var potentialHusband = MockRepository.GenerateMock<IPotentialHusband>();
var
potentialWife = MockRepository.GenerateMock<IPotentialWife>();
potentialWife = MockRepository.GenerateMock<IPotentialWife>();
var
learningRhino = new LearningMocking(potentialHusband,
potentialWife, null);
learningRhino = new LearningMocking(potentialHusband,
potentialWife, null);
learningRhino.RestaurantDinner();
potentialHusband.AssertWasCalled(x
=> x.AskForHerHandInMarriage());
=> x.AskForHerHandInMarriage());
potentialWife.AssertWasCalled(y => y.GiveHimAnswer(Arg<string>.Is.Anything));
}
This ensured that at least she will give you some answer. In next test we will have some mix and will use concrete class Linda. We have generated Stub for return value of your speech, so instead of
writing concrete class for husband we’ve used stub. This test shows that
she will not marry you:
[Test]
public void EnsureLindaWillRejectHim()
{
var potentialHusband = MockRepository.GenerateMock<IPotentialHusband>();
var
linda = new Linda();
linda = new Linda();
var marriageService = MockRepository.GenerateStub<MarriageService>();
var
learningRhino = new LearningMocking(potentialHusband,
linda, marriageService);
learningRhino = new LearningMocking(potentialHusband,
linda, marriageService);
potentialHusband.Stub(x => x.AskForHerHandInMarriage())
.Return(“Hey lady, would you marry me?”);
learningRhino.RestaurantDinner();
marriageService.AssertWasNotCalled(x
=>
=>
x.Marry(Arg<IPotentialHusband>.Is.Anything,
Arg<IPotentialWife>.Is.Equal(linda)));
Arg<IPotentialWife>.Is.Equal(linda)));
}
Let’s use another strategy to ask Linda:
[Test]
public void GiveHimAnotherTry()
{
var potentialHusband = MockRepository.GenerateMock<IPotentialHusband>();
var
linda = new Linda();
linda = new Linda();
var marriageService = MockRepository.GenerateStub<MarriageService>();
var
learningRhino = new LearningMocking(potentialHusband,
linda, marriageService);
learningRhino = new LearningMocking(potentialHusband,
linda, marriageService);
potentialHusband.Stub(x => x.AskForHerHandInMarriage())
.Return(“Linda, I love you, would you marry
me?”);
me?”);
learningRhino.RestaurantDinner();
marriageService.AssertWasCalled(x
=>
=>
x.Marry(Arg<IPotentialHusband>.Is.Anything,
Arg<IPotentialWife>.Is.Equal(linda)));
Arg<IPotentialWife>.Is.Equal(linda)));
}
Except of verifying that Marry method was called we also verify that it was called with instance of linda and none else with line Arg<IPotentialWife>.Is.Equal(linda). As well you could setup constructor arguments for MarriageService, since it is not an interface. Also with Rhino.Mocks you will need to have method Marry to be virtual. If it is not virtual and you don’t have interface for it you may need to extend that class with your own.
In the end we’ve got this method:
public
void RestaurantDinner()
void RestaurantDinner()
{
string wouldYou =
PotentialHusband.AskForHerHandInMarriage();
PotentialHusband.AskForHerHandInMarriage();
if(PotentialWife.GiveHimAnswer(wouldYou))
{
MarriageService.Marry(PotentialHusband, PotentialWife);
}
}
Nice picture:
Please write down if you would like to see more complex mocking examples.
Ensure that your Unit Test is true inspector of bad code
December 29, 2009 UnitTesting No comments
Few days ago I did refactoring and as I’m quite sure that we have good coverage I checked-in my code after all tests passed.
But today QA from my team stated that he/she cannot save new Customer. I dived into log files and found that exception occured on SaveCustomer method.
Ok, I moved to the appropriate UT. (Note: it passes ok.) Which looked like:
[Test, Category(“Integration”)]
public void SaveCustomer_EnsureWeCanCreateNewCustomer()
{
Customer customer = new Customer();
//some another setup code here…
try
{
//we hit database to save newly created customer
savedCustomer = CustomerRepository.SaveCustomer(customer);
public void SaveCustomer_EnsureWeCanCreateNewCustomer()
{
Customer customer = new Customer();
//some another setup code here…
try
{
//we hit database to save newly created customer
savedCustomer = CustomerRepository.SaveCustomer(customer);
//now we fetch customer from database requesting with the ID of just saved customer
fetchedCustomer = CustomerRepository.FetchCustomer(savedCustomer.ID);
//if fetched customer is not null we are ok!
Assert.That( fetchedCustomer, Is.Not.Null );
}
catch
{
//do nothing here….
}
finally
{
if ( fetchedCustomer != null )
{
//we want to leave our database clear after test run
CustomerRepository.Delete( fetchedCustomer );
}
}
}
This Unit Test will be succeded anyway, because
Assert.That(fetchedCustomer, Is.Not.Null) and similar asserts generates AssertionException.
Honestly I’m not aware why there was that catch{} piece, but anyway you could put there two catch blocks if it is really needed – one specific for the AssertionException and another for all cases of our life.
When you write your Unit Test you need to see it in both states – first
in failure and then in success. After that you can allow yourself
commit that code.
Main idea is:
You always need to keep one’s eye on quality of you Unit Tests and be sure that your Unit Test will fail when that is needed.
AssertDatesAreEqual
December 21, 2009 UnitTesting No comments
Have you ever faced with UT that failed once and then you always see it succeeded?
One of such tests could be test where you verify that date properties of some newly created entity are equal if they take DateTime.Now for that.
Today Hudson showed me that some test failed in project I worked near half a year ago, so test looked like:
var item1 = new Item();
var item2 = new Item();
//…
Assert.AreEqual(item1.CreatedDate, item2.CreatedDate);
var item2 = new Item();
//…
Assert.AreEqual(item1.CreatedDate, item2.CreatedDate);
Test failed because time difference between dates was greater than few milliseconds. This could occur when processor is too occupied.
So I changed assert call to more realistic:
public void AssertDatesAreEqual(DateTime dateTimeLeft, DateTime dateTimeRight)
{
Assert.IsTrue(Math.Abs((dateTimeLeft – dateTimeRight).Milliseconds) < 100,
“TimeSpan difference between dates is greater than 100 milliseconds.”);
}
{
Assert.IsTrue(Math.Abs((dateTimeLeft – dateTimeRight).Milliseconds) < 100,
“TimeSpan difference between dates is greater than 100 milliseconds.”);
}
