Tuesday, June 19, 2012

Deadline Extension: 12 July - Call for Contributions Eclipse Day Delft



It's time to get your great submissions ready for the Eclipse Day Delft in the Netherlands. The submission deadline is approaching: Juni 27 update: 12 July, 2012. The topic is facilitating software engineering with Eclipse technology! Share your experience, tools and techniques and send your proposals to Eclipse Day Organizers. More contributions details here.

Tuesday, June 5, 2012

Test Confessions: A Study of Testing Practices for Plug-In Systems (ICSE 2012)

A study by M. Greiler, A. van Deursen and M. Storey.

Testing component-based systems in general, and plug-in based products in particular, is a daunting task; the myriad of plug-in combinations, versions, interactions, and configurations gives rise to a combinatorial explosion of possibilities. Yet in practice, the systems assembled from plug-ins are widely used, achieving levels of reliability that permit successful adoption.

So which test techniques are used to ensure plug-in based products have adequate quality levels? How is the combinatorial explosion tackled? Are plug-in specific integration testing techniques adopted?

Answering questions like these calls for an in-depth study of test practices in a community of people working on plug-in based applications. To that end, we conducted a qualitative study, in which we interviewed 25 senior practitioners about how they test plug-ins and applications built on top of the Eclipse plug-in framework, and challenged the results by a structured survey of more than 150 professionals. The outcome is an overview of the testing practices currently used, a set of identified barriers limiting the adoption of test practices, and an explanation of how limited testing is compensated, which we will present on Wednesday June 6 2:00pm at ICSE 2012.

In summary, the study reveals a strong focus on unit testing, while the plug-in specific testing challenges and practices are tackled in an ad-hoc and manual manner. This lack in plug-in specific integration testing is due to several barriers, such as unclear responsibilities for system integration, a lack in resources and knowledge about plug-in testing, long test execution times and many more. This lack of explicit testing beyond the unit scope is compensated for by self-hosting of projects, the process whereby the software developed in a project is used internally on a regular basis, and the involvement of the community.

Based on these findings, we believe that to leverage plug-in specific testing, and facilitate test automation, plug-in specific tool support is needed. For the research community, we see the need to revisit current test strategies and techniques with respect to plug-in specific testing needs. For the developer community, we recommend to make the role of the community more explicit, e.g. by rewarding community members or by dedicated test days.

We hope to see you all on Wednesday @ ICSE 2012. Slides are now online: