More money for testing

Last Friday, I followed the webinar of James Whittaker, Testing expert @ Google, called “More bang for your testing buck”. Now, I want to summarize the talk but also reflect on the interviews of the Testing study. In addition, I’m eager to hear your opinion about this subject.

The main question is: If you have additional money to spend on testing, where would you invest and why?”
In his talk, James reasons about two options: Spending it early and let developers test, or spending it in later phases and have test professional test the product. He argues that quality cannot be tested in, and that developers are actually creating the product. Thus, they might be more valuable than testers. In a second thought, he mentions reduced costs for testing early in the cycle. When a developer sees a bug, he often does not even have to write a bug report, he just fixes it. The third reason why money might be invested during development is because testers write too many, as he calls it, useless test plans. Test plans get written, get outdated and die.

“Developers grow trees.” he says, and nobody knows a tree better than the developer who grew it. On the other hand, who is responsible for the forest, i.e., the integration of trees? Exactly this task should be performed by testers. The second reason to invest in testers is because they bring in the user perspective. The third reason why somebody should invest in testers is because many applications nowadays are just built by putting together pieces of software, without ever writing a single line of code.

James gave exactly three reasons for spending money during early testing and another three reasons why somebody should spend money in later phases. Nevertheless, after presenting this “fair-play” summary he showed a couple of nasty bugs. These bugs can be identified by smart human beings, but there is no way to automate finding such bugs a priori.
A picture of one of the presented bugs shows the walking road from Cambridge to Hull, unfortunately the walker must swim for a bit.


All in all, he concludes that he would spend the money during later phases for manual testing of applications by “smart” testers, but that it is even more important to increase the productivity of testers by e.g., tooling. The time testers spend writing test plans has to be reduced. At Google, they use a tool for test planning called “Testify”. A second measurement to increase testers’ productivity is to reduce the time spend to file a bug. Also for this Google has a tool assisting testers.

After listening to this talk I am interested in how this measurements fit into the Eclipse world. The interviews of the Testing Study revealed that most of the commercial projects have dedicated testers on their team. Those indeed perform the tests on behalf of the customer, represent the user-perspective and mostly perform manual testing. On the other hand, nearly none of the open source projects have dedicated testers. Tests (automated and manual) are performed by developers themselves. In the interviews, some open source developers mention they are worried that they have a too strong technical perspective on testing, and might oversee the users' perspective. One might argue that many developers of open source projects are also their own clients and use their own software even on a daily basis.

I wonder: "Why do open source development teams not have testers on their teams but commercial projects do?" Do open source teams not need testers because “they eat their own dog food”? Is the community around a project to some degree responsible for manually testing? Or are commercial projects not up to date with their development strategies?

What do you think about this subject, and are you involved in an open source or in a commercial project?

Sneak Preview of the Study Results

For the last couple of months, more than 20 interviews with experienced Eclipse developers and testers have been conducted in order to get a deep understanding of the way testing is done within the Eclipse community. During the interviews, the participants have been asked about their general test process and strategy, but also about how they handle versioning, when and how they test the user interface, and which tools they use to facilitate testing. Participants revealed also the challenges they face during testing, how they handle them and which strategies they have identified as best practices in their projects.

The very good thing about such a study is that in contrast to the experience, preferences and dislikes of just one person or one project, more than 20 people could express their opinions and expertise, which gives the possibility to share best practices and pitfalls from different projects with different characteristics, needs and development strategies. Amongst others we interviewed people from following open source projects: EMF, SOA Platform, Mangrove, Usus, EclEmma, JaCoCo, IMP, CDO, GDA, Spoofax, RAP, Mylyn, GEF, and counting. But we also interviewed people involved in commercial products that we can not mention here, with the exception of GUIDancer and Tasktop.

To share all results with you, we submitted an EclipseCon proposal for 2011 called "Test Confessions: What Eclipsers Think and Do about Testing". If you are interested in this study, you can have a sneak preview (like a movie trailer) of a potential talk by looking at the slides. If we are one of the lucky ones that get accepted we will present the whole results next March 2011.

Some of the participants agreed to be named, some want to remain anonymous. At this point, I would like to thank ALL participants, including Sven Efftinge, Leif Frenzel, Markus Harringer, Marc Hoffmann, Alexandra Imrie, Lennart Kats, Ed Merks, Tracy Miranda, Adrian Mos, Benjamin Muskalla, Steffen Pingel, Aurelien Pupier, Holger Staudacher, Eike Stepper, and Jurgen Vinju for taking part in this study.