Wednesday, August 10, 2011

Understanding Eclipse Plug-in Test Suites @ The Eclipse Testing Day 2011

In a month, the 2nd Eclipse Testing Day in Neuss, Germany takes place. At this event, I will talk about the task of understanding plug-in test suite.

Testing plug-in based systems entails testing a complex net of multiple plug-ins which not only interact with, but also enhance the functionality of each other by extension mechanisms.
In the Eclipse Testing Study, testing practices of successful Eclipse projects have been studied. Among others, Eclipsers stress integration testing as an important testing activity for plug-in systems, and explain that understanding complex plug-in test suites can become a challenging task.

Understanding test suites is an essential precondition to be able to evaluate, enhance, and improve a given test suite. Thus, to remedy the problem of understanding plug-in test suites, we developed the Eclipse Plug-in Test Suite Exploration (ETSE) tool. ETSE combines static and dynamic information on plug-in dependencies, extension initialization, and extension or service usage during a test run. This information is used to create five architectural views of the system under test and its test suite, which developers can use to understand how the integration of multiple plug-ins is tested.

The architecture of ETSE is illustrated in the Figure below. First, the information gathering phase is responsible for collecting dynamic data, coming from the instrumentation of the test runs, as well as static data (e.g., manifest file, source code, plug-in.xml). This information forms the source views, which are further processed in the knowledge inference phase. Here the source views are transformed into five target views, representing the system under test and its test suite from an integration perspective. In the last phase, the target views are visualized for the developer or tester by the ETSE Eclipse plug-in.



At the Eclipse Testing Day, I will talk about plug-in test suites and present the architectural views, which help answering questions like: “which plug-ins are tested by this test suite?”, “where are test harness and test utilities located?”, “which extensions influence the state of the test system?”, and many more. During a short demonstration of ETSE the test suite of the well-known Eclipse project Mylyn will be analyzed and visualized.

More technical details on ETSE and the approach behind can be found here.

About the Eclipse Testing Day
The 1st Eclipse Testing Day took place last year and was a great success. Also in 2011, this event provides the opportunity to get in contact with Eclipse testers, developers and managers, and to exchange experiences about testing.
Registration is possible until 5th of September via Eventbrite

Monday, March 21, 2011

Participate @ the Eclipse Testing Survey

As an Eclipser testing matters to you! But when and why do you adopt a given testing approach? How do you test the integration of multiple plug-ins? Please take some minutes and share your experience in the Eclipse Testing Survey.
Click here to take survey!

What is the study’s aim?
This survey aims to shed light on the testing culture within the Eclipse community. The anonymized results of the study will be compiled and made available.

How to participate?
Please fill in our online survey at www.goo.gl/YHBC2

Who are we looking for?
We are looking for Eclipse plug-in developers and testers, as well as for developers and testers of Rich Client Platform applications.

Who conducts the research?
The research is promoted by the Software Engineering Group at the Delft University of Technology. The study is designed by Michaela Greiler and Professor Arie van Deursen.

Monday, February 28, 2011

Swinging Test Suites with WindowTester

Arie van Deursen

One of the topics that popped up a couple of times in our "test confession" interviews with Eclipse developers, is the tension between unit testing and GUI testing. Does an application with a thorough unit test suite require user interface testing? And what about the other way around? What does automated GUI testing add to standard unit testing? Is automated GUI testing a way to involve the end-users in the testing process?

In order to fully understand all arguments, I decided to play around a little with WindowTester, a capture-and-playback automated GUI testing tool recently open-sourced by Google, with support for Swing and SWT.

My case study is JPacman, a Java implementation of a game similar to Pacman I use for teaching software testing. The plan of attack is simple: take the existing use cases, turn each into a click trail, record the trail, and generate a (JUnit-based) test suite.


My first use case is simple: enter and exit the game. To that end, I open the recorder, launch pacman, and press the red "Record" button in Eclipse. Then I press the start button in the game, followed by the exit button, which quits the application. WindowTester then prompts for the place to save this interaction:




After that, WindowTester generates the following JUnit test case:


public class StartAndExitUseCase extends UITestCaseSwing {

import ...

public StartAndExitUseCase() {
super(jpacman.controller.Pacman.class);
}
public void testStartAndExitUseCase() throws Exception {
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.click(new JButtonLocator("Exit"));
ui.wait(new WindowDisposedCondition("JPacman"));
}
}

This test passes nicely.

The next use case requires me to make moves in several directions.
Unfortunately, WindowTester can't record arrow keys. To resolve this, I decide to modify Jpacman to use good old vi-navigation ('hjkl').
I then open JPacman, to make moves in all directions. Unfortunately, I'm a bit slow, and I bump into one of the randomly moving monsters, after which I die.
This is a deeper issue: Parts of the application, in particular the random monsters, cannot be controlled via the GUI. Without such control, it is impossible to have test cases with reproducible results.
My solution is to create a slightly different version of Pacman, in which the monsters don't move at all. In fact, I happened to have the code for this ready already in my test harness, as I used such a version for doing unit testing.

This works, and the result is a test case passing just fine:

public void testSimpleMove() throws Exception {
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.enterText("jlkhhh");
}

The test doesn't assert much, though. Luckily, WindowTester has a mechanism to insert "hooks" while recording, prompting me for a name of the method to be called.


This results in the following code:

public void testSimpleMoveWithAsserts() throws Exception {
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.enterText("l");
assertCorrectMoveToTheLeft();
ui.enterText("j");
assertCorrectMoveDown();
}

protected void assertCorrectMoveDown() throws Exception {
// TODO Auto-generated method stub
}

...

WindowTester generates empty bodies for the two assert methods, leaving it to the developer to insert appropriate code. This raises two issues.

The first is that the natural way (at least for me as a tester) to verify that a move down was conducted correctly is to ask the position of the player to the appropriate objects. But from the GUI, I don't have access to these. My work around is to adjust Pacman's "main" method, to make the underlying model available through a static reference. This results in the following code:

protected void assertCorrectMoveDown() {
Pacman pm = SimplePacman.instance();
assertEquals(1, pm.getEngine().getPlayer().getLastDy());
}

Writing such an assertion requires good knowledge of the underlying API, and a bit of luck that the API exposes a method to witness the desired effect.

My next use case involves a hungry Pacman consuming a dot, which earns the player 10 points. Doing this in a way similar to the previous use case iss simple enough. Would it also be possible to assert that the proper number of points are displayed correctly in the GUI? This requires getting hold of the appropriate JTextField, and checking its content before and after eating a dot.

To support this, WindowTester offers a number of widget locators. An example is the locator used above to find a JButton labeled with "Start". Other types of locators make use of patterns, the hierarchical position in the GUI, or a unique name that a developer can give to widgets. I use this latter option, allowing me to retrieve the points displayed as follows:

private int getPointsDisplayed() throws WidgetSearchException {
WidgetReference<JTextField> wrf =
(WidgetReference<JTextField>)
getUI().find(new NamedWidgetLocator("jpacman.points"));
JTextField pointsField = (JTextField) wrf.getWidget();
return Integer.parseInt(pointsField.getText());
}

Thus, I can test if the points actually displayed are correct.

The remaining use cases can be handled in a similar way. Some use cases require moving monsters, for which having access to the GUI alone is not enough. Another use case, winning the game, would require a lot of clever moves on the regular board: instead I create a custom small and simple board, in which winning is easy.

I less and less make use of the recording capabilities of WindowTester: instead I directly program against its API. This also helps me to make the test cases easier maintainable: I have a "setUp" for pushing the "Start" button, a "tearDown" for pushing "Exit", and I can make use of other JUnit best practices. Moreover, it allows me to create a small layer of methods permitting more abstract test cases, such as the method above to obtain the actual points displayed.

Are the resulting test cases useful? I recently changed some of the GUI logic, turning a complex if-then-else to handle keyboard events into a much cleaner switch statement (inspired by a warning PMD was giving me). All unit test cases passed fine. I almost committed, but then decided to run the GUI test suite as well. It failed. I first blamed WindowTester, but then I realized it was my own fault: I had forgotten some breaks in my switch and the events were not handled correctly. The GUI test suite found a fault my unit test suite had not found.

In summary, automated GUI testing is neither a replacement for unit nor for acceptance testing. It is a useful tool for covering the GUI logic of your application. The recording capabilities can be helpful to try out certain scenarios. In the end, however an explicitly programmed test suite, making use of the GUI framework's API, seems easier to maintain. In this way, JUnit best practices related to test suite organization as well as the application's observability and controllability can be directly applied to your GUI test suite as well.



I prefer steering GUI testing programmatically to working with a recorder.

Wednesday, February 16, 2011

GUI Testing in 23 Minds

During the Eclipse testing study, we ask participants about their experiences and their opinions on testing Eclipse Plug-ins and RCP applications. In this blog post, we want to share an excerpt of what some Eclipsers really think about automated GUI testing.

What the Adopters say:
Four participants out of 23 actively use GUI testing tools and have automated UI test suites.
P3 experiences that GUI testing is better suited to capture the customer's perspective, because “the tests can capture implicit assumptions and the workflow.” The company of P3 wrote their own tooling, since one fundamental problem of other tools is: “if you [as a tester] are not in the position that you can develop yourself, you always wait for something, or somebody.” Now, with their own testing tool, GUI testing is one of their strong points. P3 knows: “anything that will affect the user, we prefer to write as an acceptance test. [...] It’s a way of getting those discussions going.”

Also P11's team uses automated GUI tests. He reports: “One thing that is really good is the extension of GEF for the SWT-Bot. It is hard to test a graphical editor. We can test them very easily, and we are very happy about that.“

Still, not all are completely satisfied with the tooling. In P18's company they use SWT-Bot and also developed their own capture tool helping to develop test cases. P18 reports: “We haven't been 100% satisfied with the capture-replay, because too much is captured.” Their solution is: “after a capture, we always have a review to remove unnecessary code.”

Also P10’s company uses QF-Test to automate GUI tests. From his experience the maintenance of the test suites takes a lot of effort, he says: “it happens that during product evolution, suddenly something works differently and the tests do not work anymore. [There are] synchronization problems, sometimes the test has not been set-up in a clean way, or timing problems occur. To cope with that it takes a lot of time.”


What Non-adopters say:
All other participants do not invest in automated GUI testing. Some participants used it for some time, but they report that the investments are too high and the benefits too low. Maintenance problems and the disability to cope with evolution are two problems mentioned by many participants. P15 says: “In my experience, automated UI testing is very expensive with not a big benefit, especially [if you have a lot] of change, which makes it a high investment which might never lead to a benefit over running some manual tests. Although manual tests are manual, they tend to be more flexible and lead to better benefits.” And P14 says: “What we had was a QF-Test suite [...] but it became apparent that those [tests] are too rigid to use them further [if software evolves]. That's why we stopped using them.”

P21’s team developed in two projects GUI test suites, but now they discontinued.
His resume is: “The decision has been done mainly by us [developers]. We put an immense effort to write UI tests, [...] and in the end often there was more test code than code to test. I doubt that makes sense.” He also knows: “There has to be a strong commitment from the customer to invest in.”

Also recorders are not beloved by many developers. Participants report that “recorders are mindless” (P4), and “superficial” (P7). “When I use [those tools], I haven’t done the really hard work. [...] with all this investment, I still do not have what I want.” (P7).

The set-up and the configuration of the build environment to execute graphical user interface tests is experienced as a hard and time consuming task which requires a lot of knowledge and expertise.

What the Sceptics say:
Many participants see an alternative solution to automated GUI testing by keeping the GUI as small as possible and in decoupling the logic behind a GUI from the GUI code. P1 even thinks that “80% of the code are already tested with unit tests”, and also P16 says “we do not have UI tests, because we can cover that with unit tests.”

Conclusion
Benefits of automated UI testing reported in our study are the ability to bring in a customer perspective in the tests, foster discussions between developers and testers, and to cover code that is hard to test otherwise. The main risks of incorporating automated GUI testing recognized by the participants are the high effort needed to create and maintain the test suites, the unclear benefits compared to other types of tests, and the usage of immature tooling with inherent problems (e.g., timing, synchronization, set-up and configuration).


Do you have similar/different experiences? Which role has GUI testing in your daily practice? Please leave a comment!



Interested? More results? Come to our talk at EclipseCon. Come, see and share your experience.