September 4, 2010 CI, Environment, UnitTesting
Couple of days ago I’ve been on CI seminar, where guy had been talking about concepts of the Continuous Integration and at some moment mentioned about testing environment and it suddenly dawned upon me and I would like to share my thoughts on this.
Two of many concepts of the CI are:
- Have repository with all needed to build product on the virgin machine using command line.
- Unit Tests, Integration Tests and Deployment should run on environment similar to production.
Feel the difference?
Ok, why do I ask about this?
Maybe a week ago we’ve moved to approach that builds sources and runs unit tests on absolutely virgin machine. Simply saying, nothing is installed into GAC, except of maybe .net framework. Everything needed we take from library folders under source control. And I agree that is really good thing. Compiler is also ok to build projects when they have reference only to root assembly in folder of other assemblies of some particular component. BUT, when you execute your code (run tests) CRL tries to find assemblies in execution folder and then in GAC. Since we have nothing in GAC, we should ensure that everything gets copied to execution folder, and here we dive into issues.
Fixes
In case of simple referencing this is not a big issue. You just should be as a spider and catch what is missed on enigma CI build machine. In some cases this is very trivial – you see System.IO.FileNotFoundException, in other cases it is not obvious.
Interesting trick
Here is one of tricts that we were needed to apply to make UT happy:
Since dll-s like sqlceme.dll are part of SQL CE, but are not CLR dlls we cannot reference them. So we add them as link in your project and then change Build property to “Copy Always” to have them in bin.
There were some other tricks we were needed to try. They are simple, but there are many.
Main Question is still remaining
All that is not main intent of this post. Main is this: Do I really need to do this. Why is this correct?
And if we are so dedicated to CI, then I would like to know also what are we testing? That our code is able to run even if none of 3rd party components are installed?
I still have some doubts, maybe I’m wrong. I would really appreciate any of your thoughts, comments that will help me figure it out and find the right way.
For fun (but maybe will get lucky) I also asked Martin Fowler in twitter:
6 comments
| Markdown | Result |
|---|---|
| *text* | text |
| **text** | text |
| ***text*** | text |
| `code` | code |
| ~~~ more code ~~~~ |
more code |
| [Link](https://www.example.com) | Link |
| * Listitem |
|
| > Quote | Quote |
I've had issues with one project, where asp.net application was running fine on development machines (of many developers) and even on testing server. But once deployed in production, we discovered some bugs that were caused by configuration of the environment. They were hard to track until we succeeded to setup an environment, that was almost identical to the production.
So I believe it IS necessary to build and test on and environment, that is closest to production. I learned it the hard way. Maybe it isn't much useful for unit testing (because you test isolated code), but it is useful when running integration tests.
gecka, thank you very much for your comment. I will use them when arguing my thoughts to management.
I am agree with gecka in general, but I can't get an idea of what is wrong with your unit tests? The only problem is that you don't have required assemblies in GAC? How about your production sever? I mean how code runs in production and why it can't be run on dev server, what is the difference between them?
Restuta, so the situation is following: build guys by intent did not install anything on dev build server. As I understand this is because they want to be ok with CI statement "everything needed is in repository and can be built on virgin machine".
Then, if it was nightly build, it is deployed to machines where everything is installed.
And that is not only about GAC. There are some 3rd parties, that install keys into registry and at run-time verify them. So simple referencing to lib folder will compile ok, but will fail on run-time, because nothing is in registry.
Got it. So my opinion is that dev server must me *the same* as live one (including all non-possible to store in repository dependencies). So I don't see any problems with that. Blindly following any rule is not a good way of doing things in general.
Moreover you are not breaking described statement, everything really builds on virgin machine, but just not runs =)
"Blindly following any rule is not a good way of doing things in general." – this should be a motto. :)