The main question is: If you have additional money to spend on testing, where would you invest and why?”
In his talk, James reasons about two options: Spending it early and let developers test, or spending it in later phases and have test professional test the product. He argues that quality cannot be tested in, and that developers are actually creating the product. Thus, they might be more valuable than testers. In a second thought, he mentions reduced costs for testing early in the cycle. When a developer sees a bug, he often does not even have to write a bug report, he just fixes it. The third reason why money might be invested during development is because testers write too many, as he calls it, useless test plans. Test plans get written, get outdated and die.
“Developers grow trees.” he says, and nobody knows a tree better than the developer who grew it. On the other hand, who is responsible for the forest, i.e., the integration of trees? Exactly this task should be performed by testers. The second reason to invest in testers is because they bring in the user perspective. The third reason why somebody should invest in testers is because many applications nowadays are just built by putting together pieces of software, without ever writing a single line of code.
James gave exactly three reasons for spending money during early testing and another three reasons why somebody should spend money in later phases. Nevertheless, after presenting this “fair-play” summary he showed a couple of nasty bugs. These bugs can be identified by smart human beings, but there is no way to automate finding such bugs a priori.
A picture of one of the presented bugs shows the walking road from Cambridge to Hull, unfortunately the walker must swim for a bit.
All in all, he concludes that he would spend the money during later phases for manual testing of applications by “smart” testers, but that it is even more important to increase the productivity of testers by e.g., tooling. The time testers spend writing test plans has to be reduced. At Google, they use a tool for test planning called “Testify”. A second measurement to increase testers’ productivity is to reduce the time spend to file a bug. Also for this Google has a tool assisting testers.
After listening to this talk I am interested in how this measurements fit into the Eclipse world. The interviews of the Testing Study revealed that most of the commercial projects have dedicated testers on their team. Those indeed perform the tests on behalf of the customer, represent the user-perspective and mostly perform manual testing. On the other hand, nearly none of the open source projects have dedicated testers. Tests (automated and manual) are performed by developers themselves. In the interviews, some open source developers mention they are worried that they have a too strong technical perspective on testing, and might oversee the users' perspective. One might argue that many developers of open source projects are also their own clients and use their own software even on a daily basis.
I wonder: "Why do open source development teams not have testers on their teams but commercial projects do?" Do open source teams not need testers because “they eat their own dog food”? Is the community around a project to some degree responsible for manually testing? Or are commercial projects not up to date with their development strategies?
What do you think about this subject, and are you involved in an open source or in a commercial project?
