Why Wil, WHY!

Wil Shipley is the co-founder of Delicious Monster. Before that he founded the Omni Group, which originally developed software for NeXT and later moved on to Apple's OS X. I generally enjoy reading Wil's blog. Although he can be rather blunt at times, and probably has an approach that wouldn't fly in most corporate environments, he is strongly opinionated and he sticks to his guns. For the most part this is a quality I admire in people, regardless of whether I agree with them or not.

Well in this instance I disagree with Wil. He recently posted a blog entry about his views on unit testing: Unit Testing is teh Suck, Urr.. The title alone should be enough to set a wise developer off.

I'm not about to blindly promote unit testing and test-driven development just because it's "what's in" right now. I have used unit testing techniques, encouraged a team of developers to use these techniques on a commercial web application, and written developer tools to help integrate unit testing into a continuous build workflow. I have even purposely not unit tested certain projects just to see how much extra time I was spending on "manual" testing. I believe in unit testing because I have seen it work and I have seen projects fail because of lack of unit testing.

First of all, talking about unit testing separately from test-driven development is non-sense. Unit tests are a developer tool! Unit tests are written by the developer to isolate a specific section of functionality while the program is being worked on; be that initial coding, debugging, or bug fixing. If this is not the case, call your tests anything else but please don't call them unit tests. Obviously this is my personal view of things, although it is supported by other developers. In Wil's case, his exposure to "unit tests" was clearly one of the bad uses of the methodology. He mentions being hired to write unit test code by Lighthouse Design for one of his first jobs. Huh?

A company that hires devs/testers for the specific purpose of writing "unit tests" is not doing unit testing! Of the many benefits of unit testing I would say most of them are developer centric. The benefits of having someone solely writing unit test are completely countered; the company is essentially paying someone to spend a considerable amount of time working through code in hopes of recreating the developer's original state of mind and capturing the results in the form of executable code. This is something that would have taken the developer very little time to do in the first place.

Wil's first general guideline is:

When you modify your program, test it yourself. Your goal should be to break it, NOT to verify your code. That is, you should turn your huge intellect to "if I hated this code, how could I break it" as SOON as you get a working build, and you should document all the ways you break it. [Sure, maybe you don't want to bother fixing the bug where if you enter 20,000 lines of text into the "item description" your program gets slow. But you should test it, document that there is a problem, and then move on.] You KNOW that if you hated someone and it was your job to break their program, you could find some way to do it. Do it to every change you make

So let me think about this for a second. When I'm coding along and I make a change I should stop and take the time to test that. Now how different is taking the time to manually run the program and put it into a few different configurations compared to taking that same time and writing a few lines a test code. Think about this honestly. Switching from developer to tester is a cognitive task switch; you have to shift your world view from building to breaking. To make matters worse you have to stop thinking in code and start thinking in whatever interface your program provides, which may still be crude since you are just developing it. This is an expensive context switch.

Lets extend this line of thought. What about the time the you spent thinking about how you were going to create a routine? What were you doing? Surely your mind was active, but could the time have been spent doing something more productive. For example, you could have been creating a very basic set of unit tests to handle the most common functionality; a skeleton functional test. When you approach unit testing from this angle writing and maintaining unit tests takes only a little more time than doing you regular development.

But the benefits don't stop there. Let's say you are starting the development of an app or more realistically a component. You have some basic functionality already in place that you can manually run 10 tests against and be fairly sure the general behaviour is correct. So you go back to the code and change it to keep adding the functionality that's required. Now let's say you need to run 2 tests to check this new functionality. But what about the functionality that was already there? Did you break anything with the changes? I can think of many examples during the early stages of development where you make changes that will almost inevitably affect what little functionality was already there. So you really have to run 12 tests, the 10 original tests and the 2 new tests. If you consider the frequency at which you iterate through these develop/test micro-cycles you can quickly get a sense of just how much time you are wasting, even in the course of a few hours, manually re-running these tests.

Shipley adds:

Too often I see engineers make a change, run the program quickly, try a single test case, see it not crash, and decide they are done. THAT IS TEH SUCK! You've got to TRY to break that shit! What happens when you add 10 more points to that line?

I'm going to argue that developer's fall into this pattern because of the tedium of constantly repeating very similar trial runs of their program.

He then goes on to compare testing strategies to nature and evolution.

When you get the program working to the point where it does something and loads and saves data, find some people who love it and DO A BETA TEST. The beta test is often maligned, but the most stable programs I've ever written are stable because of beta testing. Essentially, beta testing is Nature's Way (TM) of making systems stable. You think nature creates unit tests and system tests every time it mutates a gene? Aw hell nah. It puts it out in the wild, and if it seems better it sticks around. If not, it's dead.

I really don't follow his argument on this one. Nature never just changes something and then see if that change is successful; that kind of pseudo random genome re-sequencing is actually quite un-natural. If you think about it nature is intrinsically very efficient (lazy?). According to the theory of evolution, changes occur because of stresses that an organism is put under; no stress to change means no changes. That's not unlike unit testing. The changes are made to the system in hopes of allowing it to cope better with the challenges that are already in place.

I am in no way saying that some form of beta testing is not required. On the contrary beta testing is a much needed phase in the development life cycle. Beta testing can catch bugs ranging from localization issues to inconsistencies in the user interaction model. Unit testing is especially hard to carry out on the graphical user interface of programs.

Bill Bumgarner, one of the developers that worked on Apple's Core Data framework posted "Unit Testing" on his blog shortly after Wil's comments. Bill points out that Core Data makes heavy use of unit tests and that it would not have made it to market in the time frame that it did had it not been for unit tests. Bill also points to Delicious Library's reliance on Core Data. In effect Wil was able to get by without unit testing mainly because of the richness of the Cocoa frameworks.

I wanted to post this blog entry a while back but got caught up in school work and job interviews. Now in all fairness I should mention that Wil briefly returned to the topic of unit testing (a few sentences compared to a few pages in his original article) directly responding to Bumgarner's post. He acknowledged that unit tests were great for frameworks and bad for UIs; it really felt more like he was brushing off the topic than honestly addressing it. The remainder of the blog had a very apologetic tone since he was addressing the large amount of criticism to his blog post that came right after the unit testing post, "Quit School and Set Things On Fire."

Alright, this post has gone on too long as it is. Considering one of my goals with this blog is to work on my ability to write concisely I don't think I'm doing all that great. Now that interviews are over I'll hopefully have a little more time to blog about how that went after the next 2 weeks of assignment and midterm hell.

Thank you ADC

Every IDE has it's own quirks and ways of seeing the world. Unfortunately for developers that means spending time getting used to viewing the world in that particular way. Whether this is good interface design practice I will leave for another post but I'll leave a quick note while passing by. Since an IDE is really an environment rather than an application like Word and it's a tool that will be used by experts much like Photoshop or Illustrator, it may not be a very bad thing to make it just hard enough to use so that new comers have to spend some time learning how to do things in the environment. The benefit of this approach is that once users have gone through this learning process they are much more proficient; compare your average Word user who has discovered most of the features through self discovery (aka snooping around the interface) and uses on average 10 features to your average Photoshop or Illustrator user who has invested time and energy learning how to use the application. Are Graphic designers more intelligent than office knowledge workers? I don't think so. It's the interface itself that has made them more proficient.

Apple's IDE, Xcode, is actually quite powerful once you finally figure out what's what in their world. After spending some time using Xcode and not fully understanding the reasoning behind everything I think things finally clicked sometime this summer. After using Xcode for months I now finally understand the underlying principle behind Xcode's project structure. Like most things of this nature, once you finally understand it's not only simple but painfully obvious.

Xcode is not alone when it comes to applications that just make you feel stupid. I'm not going name any applications or peg them to the open source world or commercial software shops. What it does come down to however is complete user experience. It might not even be a question of the developers not caring. Rather, many times the developers forget what it was like to first start learning and using their own product. They become so immersed in their paradigm that it is obvious to them and very casually overlook the difficulties that new-comers might have with the product.

Although "Understanding Xcode Projects" published on the ADC is long overdue it is nonetheless a very elegant solution. No more than a few pages with accompanying annotated screen shots, "Understanding Xcode Projects" succinctly explains Xcode's philosophy. The document can prove useful to new-comers and experienced Xcode users alike. In about ten minutes you can look over the article and sit back and say "AHHH! So that's how things work around here." Rather than spend many frustrated attempts at tackling Xcode by the horns.

I think more developers should learn from this example; struggling with a user interface should not be viewed as a rite of passage. This is probably even more true in the open source world if only because technical writers are not as readily available. If you're a developer what's an hour spent preparing a help article like this one compared to the countless hours you've spent implementing all those neat features that you hope will get used?

Surf's up

Earlier this year there was a small uprising in the open source world over Apple's commitment to giving back to the community. At the core of the dispute was Apple's WebCore project. Open source advocates were upset that Apple wasn't giving more back to the KHTML project considering what had been gained for free. The argument really heated up shortly after Dave Hyatt posted to his original Surfin Safari blog that WebCore, and hence Apple's Safari browser, now passed the Acid 2 CSS test.

Because of the publicity Apple has given to the close relation between the WebCore project and the KHTML project the members of the KHTML project were being asked how soon it would be until KHTML would pass the Acid 2 test. This led members of the KHTML project to post a few open letters to the community expressing their discontent over how Apple failed to collaborate effectively with the open source project.

Hyatt soon posted a reply to his blog asking anyone with suggestions about how they thought the WebCore project could change to improve the situation. Since the situation was already heated many of the comments fell into what I would consider non-productive chatter. I wouldn't say that it evolved to a flame war but there was much blame being placed on either party and a definite lack of concrete solution proposals.

I care about the success of Safari and the WebCore project just as I care about seeing an open source project like KHTML succeed. So I decided that enough was enough and that I should write a detailed response proposing a handful of manageable changes that would allow both teams to benefit even more from their partnership.

The WebCore project is now being managed differently and there is a new Surfin Safari blog. Although I have no way of knowing for sure, some of the ideas I had suggested are very similar to the changes that have actually been put in place. Now it would be extremely vain of me to think that there are no other bright people at Apple that could have come up with similar ideas. However, it is somewhat comforting that I was partly responsible for getting things rolling in what seems to be a productive direction.

Here was my response:

First let me give some background about myself so you can assess my level of bias. I am a computer science student at a Canadian university and I have an Apple ADC Student membership. I am also an Apple Campus Rep and many of the students in my program are very pro open source (as am I) so responding to issues like this one is something I must deal with often. Our university has a co-op program and while I am only beginning my 3rd year I effectively have 1 full year experience working in a commercial setting, particularly using open source frameworks to produce a commercial web apps. I know this isn't a great amount of work experience but it's better than none. My minor is in business with a focus on management sciences.

With that said here are a few things I think could be done better based on the information I have, which I admit is incomplete much like most people who have posted.

1- From what I gather communication between Apple and the KDE group has been only through email or informally through blogs. One of my co-op jobs was with Research In Motion (makers of the blackberry) so if anyone I can appreciate the benefits of email. However, email and blog postings fall very far from being a rich enough communication medium to coordinate development efforts. In my opinion many developers fall prey to over-relying on email because of it's ease and speed. Both companies and open source communities use it because it's cheap and widespread and I think that companies and open source projects run into problems when they rely on email too much. Email can't capture facial expressions or vocal intonations and is often miss-interpreted and the miss-interpretation can't be corrected on the fly like you can when you are talking face to face.

Soln: Organize a monthly or by monthly conference call with the lead developers of both parties (I know Apple has the resources for this since Campus reps have regular calls). Invite some of the KHTML devs to WWDC and send some WebKit devs to the KDE conference. I think this alone would change the outlook on the situation and knowing that there are people on the other end.

2- Apple is very closed about the bugs it is working to fix. This is a company policy and it is completely up to Apple if it wants to leave it as such. However, I think it not only helps external relations (members of the community wanting to contribute to an open source project) but also Apple (internally) if bugs logged against all of it's open source projects are also open.

Soln: There is no doubt already a team responsible for the initial bug triage that comes into radar (be it crash reports, people clicking on the report a bug button in safari, or even people logging bugs on their own through bugreporter.apple.com). When these bugs are initially reviewed and dispatched if they are dispatched to an open source project (WebKit, Darwin, GCC, Bonjour etc.) they should be sent to a separate, open bug tracking system. For example if a safari bug it logged but it ends up being a bug in the client then don't put it in the open source tracking system but if the bug originates in WebKit then go ahead and open source the bug. I have no idea what Apple uses for Issue tracking software but I have experience with 3 of the most popular issue tracking products and they provided some way of synchronizing the status of bugs between 2 databases. Better yet the radar database schema could be changed to incorporate an Open source flag. Whatever happens Apple needs to let other people interested in one of the open source projects what is currently being worked on so that work is not duplicated and it is well coordinated. What is the point of open source otherwise.

3- There were both suggestions and criticisms towards adopting a common code base. A point to keep in mind, KHTML and WebKit in their current state are 2 separate albeit open source projects. I don't think Apple has done any false publicity here since they were fairly straight forward when they said that WebKit was a fork. Apple in my opinion has also done a good job of helping the KHTML project both in terms of added functionality and publicity. KHTML has come a long ways since Safari has been released (I'm happy about this since KDE is my platform of choice when I run Linux) and although Apple's contribution seems to have slowed down lately compared to the initial benefits that were seen I think this is mainly a communication problem. If it takes more time to integrate a fix from one project into the other than it does to actually write the patch because of differences in both projects (i.e.: KHTML using Apple patches and vice versa) then it only makes sense that a developer would rather spend the time fixing it and feeling some sense of accomplishment rather than spend the time sorting through some other developer's work. To address the issue of what Apple has to gain from going through the trouble of moving to a common code base, well lots. Better PR than what it has been getting lately, increased awareness and use of a common web engine, a more robust rendering engine since any abstractions introduced separate the engine functionality from native UI toolkit functionality that may have crept through undocumented, and in the end, less work since the community can be fixing some of the bugs. Notice that these are potential benefits for KDE as well in my opinion.

Soln: I think that fundamentally both teams still have many of the same core values and direction for the project. Both teams still care very much about making a light-weight, fast, and common rendering engine. In my opinion I think that the problems here stem more from what I would call a superficial project management issue than a core divergence in project direction. Simply put, I think it would be a shame for a combined effort with so much potential to cease existing. Most of the complaints I can gather from the KDE camp are frustrations about how hard it is to reap the benefits of the combined effort; how hard it is to actually do more of the work the love doing rather than code base administrivia. Mr. Hyatt's offer about simply using web core (which may seem drastic at this point in the game), his continued posting of patches, and the very fact that he posed the question "what can Apple do?" is very clear indication to me that at least he, if not more people on the WebKit team and higher up at Apple, are more than willing to try to improve the situation. If Apple truly didn't care then why would they even bother with this sort of thing. The proposition to just go ahead an use WebKit as a common code base may seem like an effort from Apple to simply supplant KHTML but, although I am just guessing here, I think it shows just how much Mr. Hyatt wants to improve the situation and maybe even how desperate he might be and the lengths he is will to go to help resolve this issue.

Just so that this post doesn't suffer from any recency effects and people finish reading this thinking that I am putting most of the pressure on KDE devs, I think that both teams have to work equally if they hope to gain.

Finally, my own blog

I've been wanting to start my own blog and web site for some time now. While the web site is still a work in progress I thought that the end of Spring term at university was as good a time as ever to get things going.

For the last six months or so I keep coming across various articles and ideas and I say to myself "oh I should really blog about this." Call me a hopeless dreamer but I truly believe that one person can make a difference in the world and I won't stop until I've made my dent. I may only be a 21 year old student but I am very opinionated. At the same time there are several people that I admire, a few of them being: John Gruber over at Daring Fireball, Joel Spolsky from Joel on Software, Bob Cringely who writes the I, Cringely column, and Bill Cowan a professor at the University of Waterloo, who I've been doing research with and doesn't keep as much of an online presence. The main reason I admire these people is because they all have strong opinions that are very well reasoned about yet expressed with a simplicity and elegance that few people are able to match. Talk is cheap and dreaming without action would truly prove to be hopeless. If I do really hope to make a difference in this world I have to learn to express myself with as much eloquence as the best of them.

As they say, practice makes perfect and what better place to practice and get feedback than on a blog.

So, why Blogger? I must admit that I am somewhat of a perfectionist; that combined with how much work they love to give us at university has led to me putting off creating my own blog. Blogger has everything I need for the time being and by the time I need more features I can either migrate my blog to some custom solution or who knows Blogger might have the features I need by then. At this point I think that getting into the rhythm of blogging regularly is the most important part of the whole process. This kind of thinking goes hand with an approach to technology development that I have been pondering about for a little time now; the most effortless solution often tends to be the best one. For a perfectionist it can be hard to let go of the notion that solutions must be rigorous and completely defined over the problem domain. Sometimes the quick and dirty solution can turn out to be even more enabling; it can turn out to be exquisite.