Tuesday, December 14, 2010

More money for testing

Last Friday, I followed the webinar of James Whittaker, Testing expert @ Google, called “More bang for your testing buck”. Now, I want to summarize the talk but also reflect on the interviews of the Testing study. In addition, I’m eager to hear your opinion about this subject.

The main question is: If you have additional money to spend on testing, where would you invest and why?”
In his talk, James reasons about two options: Spending it early and let developers test, or spending it in later phases and have test professional test the product. He argues that quality cannot be tested in, and that developers are actually creating the product. Thus, they might be more valuable than testers. In a second thought, he mentions reduced costs for testing early in the cycle. When a developer sees a bug, he often does not even have to write a bug report, he just fixes it. The third reason why money might be invested during development is because testers write too many, as he calls it, useless test plans. Test plans get written, get outdated and die.

“Developers grow trees.” he says, and nobody knows a tree better than the developer who grew it. On the other hand, who is responsible for the forest, i.e., the integration of trees? Exactly this task should be performed by testers. The second reason to invest in testers is because they bring in the user perspective. The third reason why somebody should invest in testers is because many applications nowadays are just built by putting together pieces of software, without ever writing a single line of code.

James gave exactly three reasons for spending money during early testing and another three reasons why somebody should spend money in later phases. Nevertheless, after presenting this “fair-play” summary he showed a couple of nasty bugs. These bugs can be identified by smart human beings, but there is no way to automate finding such bugs a priori.
A picture of one of the presented bugs shows the walking road from Cambridge to Hull, unfortunately the walker must swim for a bit.
bugMapsSmall.PNG

All in all, he concludes that he would spend the money during later phases for manual testing of applications by “smart” testers, but that it is even more important to increase the productivity of testers by e.g., tooling. The time testers spend writing test plans has to be reduced. At Google, they use a tool for test planning called “Testify”. A second measurement to increase testers’ productivity is to reduce the time spend to file a bug. Also for this Google has a tool assisting testers.

After listening to this talk I am interested in how this measurements fit into the Eclipse world. The interviews of the Testing Study revealed that most of the commercial projects have dedicated testers on their team. Those indeed perform the tests on behalf of the customer, represent the user-perspective and mostly perform manual testing. On the other hand, nearly none of the open source projects have dedicated testers. Tests (automated and manual) are performed by developers themselves. In the interviews, some open source developers mention they are worried that they have a too strong technical perspective on testing, and might oversee the users' perspective. One might argue that many developers of open source projects are also their own clients and use their own software even on a daily basis.

I wonder: "Why do open source development teams not have testers on their teams but commercial projects do?" Do open source teams not need testers because “they eat their own dog food”? Is the community around a project to some degree responsible for manually testing? Or are commercial projects not up to date with their development strategies?

What do you think about this subject, and are you involved in an open source or in a commercial project?

3 comments:

  1. "Why do open source development teams not have testers on their teams but commercial projects do?"
    You are opposing open-source and commercial. IMHO, this is not a right way to view things. I am part of a team that develops a commercial open-source product, ie an open-source product which is intended to generate money to a company. We have testers for our open-source product, and I am aware of a lot of other open-source commercial products that also have teams dedicated to testing.

    ReplyDelete
  2. It is a valid point to separate open source, commercial and hybrid (commercial open source) projects. I simplified the categorization because out of all interviewed open source and commercial open source projects so far only two had dedicated testers. I am very interested in examples of commercial open source projects that do have testers. Could you mention some within the Eclipse community? I would like to interview them as well.

    In addition, why do you think you need/have a separated test team?

    ReplyDelete
  3. I would say here that in open source programs no one is responsible for the quality of the delivered product so it is something like common ownership, so if you exercise something and you get a bug there is no commitment or obligation to get this fixed as soon as you need!, in commercial products this is not the case you are committed to your customer to deliver a reliable product to the level which guarantee your customer satisfaction. another point I have here that if you teach the developer to not introduce a bug, I would say you can depend on their testing :)it is an old subject testing is not about writing test plan or define a long test process, testing is about tailor your practices to comply with your current requirement for a specific project and work cooperatively with developers to get the whole image and make them evolved as your development work changes , I would answer your talk about having tester writing a lot of meaningless test plan because in a lot of situation they do not have good visibility about how the tree has been grew up but the solution is not to give everything for the developers because definitely if they were aware that this step in growing up the tree may introduce another bug most properly they will not going to introduce them. so if I have more money and you are in critical situation you have to stop bleeding by invest more in testing, after that you can start from earlier phases, which always not mean to invest on developers.. it depends :) and really it depends on your case, if you have like a highly mature developers you may add more practices like static analysis for your code or traditional inspection and may be as you mentioned make the tester life more easier through tooling and get them more productive in the simple task so they will have more time to get closer to their customer usage profile, or just get your tester more earlier involved in the development phase...guys we left the waterfall models from years it is no more fitting the current high demands for quality with the fast time to market :)

    ReplyDelete