Wil Shipley is the co-founder of Delicious Monster. Before that he founded the Omni Group, which originally developed software for NeXT and later moved on to Apple's OS X. I generally enjoy reading Wil's blog. Although he can be rather blunt at times, and probably has an approach that wouldn't fly in most corporate environments, he is strongly opinionated and he sticks to his guns. For the most part this is a quality I admire in people, regardless of whether I agree with them or not.
Well in this instance I disagree with Wil. He recently posted a blog entry about his views on unit testing: Unit Testing is teh Suck, Urr.. The title alone should be enough to set a wise developer off.
I'm not about to blindly promote unit testing and test-driven development just because it's "what's in" right now. I have used unit testing techniques, encouraged a team of developers to use these techniques on a commercial web application, and written developer tools to help integrate unit testing into a continuous build workflow. I have even purposely not unit tested certain projects just to see how much extra time I was spending on "manual" testing. I believe in unit testing because I have seen it work and I have seen projects fail because of lack of unit testing.
First of all, talking about unit testing separately from test-driven development is non-sense. Unit tests are a developer tool! Unit tests are written by the developer to isolate a specific section of functionality while the program is being worked on; be that initial coding, debugging, or bug fixing. If this is not the case, call your tests anything else but please don't call them unit tests. Obviously this is my personal view of things, although it is supported by other developers. In Wil's case, his exposure to "unit tests" was clearly one of the bad uses of the methodology. He mentions being hired to write unit test code by Lighthouse Design for one of his first jobs. Huh?
A company that hires devs/testers for the specific purpose of writing "unit tests" is not doing unit testing! Of the many benefits of unit testing I would say most of them are developer centric. The benefits of having someone solely writing unit test are completely countered; the company is essentially paying someone to spend a considerable amount of time working through code in hopes of recreating the developer's original state of mind and capturing the results in the form of executable code. This is something that would have taken the developer very little time to do in the first place.
Wil's first general guideline is:
When you modify your program, test it yourself. Your goal should be to break it, NOT to verify your code. That is, you should turn your huge intellect to "if I hated this code, how could I break it" as SOON as you get a working build, and you should document all the ways you break it. [Sure, maybe you don't want to bother fixing the bug where if you enter 20,000 lines of text into the "item description" your program gets slow. But you should test it, document that there is a problem, and then move on.] You KNOW that if you hated someone and it was your job to break their program, you could find some way to do it. Do it to every change you make
So let me think about this for a second. When I'm coding along and I make a change I should stop and take the time to test that. Now how different is taking the time to manually run the program and put it into a few different configurations compared to taking that same time and writing a few lines a test code. Think about this honestly. Switching from developer to tester is a cognitive task switch; you have to shift your world view from building to breaking. To make matters worse you have to stop thinking in code and start thinking in whatever interface your program provides, which may still be crude since you are just developing it. This is an expensive context switch.
Lets extend this line of thought. What about the time the you spent thinking about how you were going to create a routine? What were you doing? Surely your mind was active, but could the time have been spent doing something more productive. For example, you could have been creating a very basic set of unit tests to handle the most common functionality; a skeleton functional test. When you approach unit testing from this angle writing and maintaining unit tests takes only a little more time than doing you regular development.
But the benefits don't stop there. Let's say you are starting the development of an app or more realistically a component. You have some basic functionality already in place that you can manually run 10 tests against and be fairly sure the general behaviour is correct. So you go back to the code and change it to keep adding the functionality that's required. Now let's say you need to run 2 tests to check this new functionality. But what about the functionality that was already there? Did you break anything with the changes? I can think of many examples during the early stages of development where you make changes that will almost inevitably affect what little functionality was already there. So you really have to run 12 tests, the 10 original tests and the 2 new tests. If you consider the frequency at which you iterate through these develop/test micro-cycles you can quickly get a sense of just how much time you are wasting, even in the course of a few hours, manually re-running these tests.
Shipley adds:
Too often I see engineers make a change, run the program quickly, try a single test case, see it not crash, and decide they are done. THAT IS TEH SUCK! You've got to TRY to break that shit! What happens when you add 10 more points to that line?
I'm going to argue that developer's fall into this pattern because of the tedium of constantly repeating very similar trial runs of their program.
He then goes on to compare testing strategies to nature and evolution.
When you get the program working to the point where it does something and loads and saves data, find some people who love it and DO A BETA TEST. The beta test is often maligned, but the most stable programs I've ever written are stable because of beta testing. Essentially, beta testing is Nature's Way (TM) of making systems stable. You think nature creates unit tests and system tests every time it mutates a gene? Aw hell nah. It puts it out in the wild, and if it seems better it sticks around. If not, it's dead.
I really don't follow his argument on this one. Nature never just changes something and then see if that change is successful; that kind of pseudo random genome re-sequencing is actually quite un-natural. If you think about it nature is intrinsically very efficient (lazy?). According to the theory of evolution, changes occur because of stresses that an organism is put under; no stress to change means no changes. That's not unlike unit testing. The changes are made to the system in hopes of allowing it to cope better with the challenges that are already in place.
I am in no way saying that some form of beta testing is not required. On the contrary beta testing is a much needed phase in the development life cycle. Beta testing can catch bugs ranging from localization issues to inconsistencies in the user interaction model. Unit testing is especially hard to carry out on the graphical user interface of programs.
Bill Bumgarner, one of the developers that worked on Apple's Core Data framework posted "Unit Testing" on his blog shortly after Wil's comments. Bill points out that Core Data makes heavy use of unit tests and that it would not have made it to market in the time frame that it did had it not been for unit tests. Bill also points to Delicious Library's reliance on Core Data. In effect Wil was able to get by without unit testing mainly because of the richness of the Cocoa frameworks.
I wanted to post this blog entry a while back but got caught up in school work and job interviews. Now in all fairness I should mention that Wil briefly returned to the topic of unit testing (a few sentences compared to a few pages in his original article) directly responding to Bumgarner's post. He acknowledged that unit tests were great for frameworks and bad for UIs; it really felt more like he was brushing off the topic than honestly addressing it. The remainder of the blog had a very apologetic tone since he was addressing the large amount of criticism to his blog post that came right after the unit testing post, "Quit School and Set Things On Fire."
Alright, this post has gone on too long as it is. Considering one of my goals with this blog is to work on my ability to write concisely I don't think I'm doing all that great. Now that interviews are over I'll hopefully have a little more time to blog about how that went after the next 2 weeks of assignment and midterm hell.
