For the last couple of months, more than 20 interviews with experienced Eclipse developers and testers have been conducted in order to get a deep understanding of the way testing is done within the Eclipse community. During the interviews, the participants have been asked about their general test process and strategy, but also about how they handle versioning, when and how they test the user interface, and which tools they use to facilitate testing. Participants revealed also the challenges they face during testing, how they handle them and which strategies they have identified as best practices in their projects.
The very good thing about such a study is that in contrast to the experience, preferences and dislikes of just one person or one project, more than 20 people could express their opinions and expertise, which gives the possibility to share best practices and pitfalls from different projects with different characteristics, needs and development strategies. Amongst others we interviewed people from following open source projects: EMF, SOA Platform, Mangrove, Usus, EclEmma, JaCoCo, IMP, CDO, GDA, Spoofax, RAP, Mylyn, GEF, and counting. But we also interviewed people involved in commercial products that we can not mention here, with the exception of GUIDancer and Tasktop.
To share all results with you, we submitted an EclipseCon proposal for 2011 called "Test Confessions: What Eclipsers Think and Do about Testing". If you are interested in this study, you can have a sneak preview (like a movie trailer) of a potential talk by looking at the slides. If we are one of the lucky ones that get accepted we will present the whole results next March 2011.
Some of the participants agreed to be named, some want to remain anonymous. At this point, I would like to thank ALL participants, including Sven Efftinge, Leif Frenzel, Markus Harringer, Marc Hoffmann, Alexandra Imrie, Lennart Kats, Ed Merks, Tracy Miranda, Adrian Mos, Benjamin Muskalla, Steffen Pingel, Aurelien Pupier, Holger Staudacher, Eike Stepper, and Jurgen Vinju for taking part in this study.
Hi Michaela,
ReplyDeletenice slides. How do you feel about providing more details how teams actually build their software, i.e., which features of their build systems do they actually use?
It would be great to have a nice summary (and some instructions) how these teams perform (for instance) their integration tests, which degree of code coverage they reach, which tools they use (and how) etc.
The big picture of "How to do QA in Eclipse" may also cover such things. Thanks and keep up working :)
best,
Marcel
Hey Marcel,
ReplyDeleteThank you for the comment. Yes, we will definitely provide more details on how Eclipsers develop software, and especially how the test it. In the study, we tackle topics around the whole test process, including integration testing, unit testing, versioning, GUI testing, how the build and test system is set-up, which test runner they actually use and many more. I will continue posting on this blog to share more of the results.
\Michaela
Hi Michaela,
ReplyDeleteIt's a very nice and interesting study. Did you identify some actionable items from the survey results? How could we improve Eclipse testing?
Sung
Hello Sung,
ReplyDeleteThank you for raising this good point. We are, for example, analyzing the interviews according to “Test Challenges” Eclipsers face. Those challenges give potential for improvements and to facilitate testing and reach from tooling (like build and tests systems), to test processes and strategies. Another actionable item is spreading best practices. Many of the projects we interviewed are mature and have identified strategies other project could benefit from.
Michaela