One of the topics that popped up a couple of times in our "test confession" interviews with Eclipse developers, is the tension between unit testing and GUI testing. Does an application with a thorough unit test suite require user interface testing? And what about the other way around? What does automated GUI testing add to standard unit testing? Is automated GUI testing a way to involve the end-users in the testing process?
In order to fully understand all arguments, I decided to play around a little with WindowTester, a capture-and-playback automated GUI testing tool recently open-sourced by Google, with support for Swing and SWT.
My case study is JPacman, a Java implementation of a game similar to Pacman I use for teaching software testing. The plan of attack is simple: take the existing use cases, turn each into a click trail, record the trail, and generate a (JUnit-based) test suite.
My first use case is simple: enter and exit the game. To that end, I open the recorder, launch pacman, and press the red "Record" button in Eclipse. Then I press the start button in the game, followed by the exit button, which quits the application. WindowTester then prompts for the place to save this interaction:
After that, WindowTester generates the following JUnit test case:
public class StartAndExitUseCase extends UITestCaseSwing {
import ...
public StartAndExitUseCase() {
super(jpacman.controller.Pacman.class);
}
public void testStartAndExitUseCase() throws Exception {
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.click(new JButtonLocator("Exit"));
ui.wait(new WindowDisposedCondition("JPacman"));
}
}
This test passes nicely.
The next use case requires me to make moves in several directions.
Unfortunately, WindowTester can't record arrow keys. To resolve this, I decide to modify Jpacman to use good old vi-navigation ('hjkl').
I then open JPacman, to make moves in all directions. Unfortunately, I'm a bit slow, and I bump into one of the randomly moving monsters, after which I die.
This is a deeper issue: Parts of the application, in particular the random monsters, cannot be controlled via the GUI. Without such control, it is impossible to have test cases with reproducible results.
My solution is to create a slightly different version of Pacman, in which the monsters don't move at all. In fact, I happened to have the code for this ready already in my test harness, as I used such a version for doing unit testing.
This works, and the result is a test case passing just fine:
public void testSimpleMove() throws Exception {
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.enterText("jlkhhh");
}
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.enterText("jlkhhh");
}
The test doesn't assert much, though. Luckily, WindowTester has a mechanism to insert "hooks" while recording, prompting me for a name of the method to be called.
This results in the following code:
public void testSimpleMoveWithAsserts() throws Exception {
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.enterText("l");
assertCorrectMoveToTheLeft();
ui.enterText("j");
assertCorrectMoveDown();
}
protected void assertCorrectMoveDown() throws Exception {
// TODO Auto-generated method stub
}
...
IUIContext ui = getUI();
ui.click(new JButtonLocator("Start"));
ui.enterText("l");
assertCorrectMoveToTheLeft();
ui.enterText("j");
assertCorrectMoveDown();
}
protected void assertCorrectMoveDown() throws Exception {
// TODO Auto-generated method stub
}
...
WindowTester generates empty bodies for the two assert methods, leaving it to the developer to insert appropriate code. This raises two issues.
The first is that the natural way (at least for me as a tester) to verify that a move down was conducted correctly is to ask the position of the player to the appropriate objects. But from the GUI, I don't have access to these. My work around is to adjust Pacman's "main" method, to make the underlying model available through a static reference. This results in the following code:
protected void assertCorrectMoveDown() {
Pacman pm = SimplePacman.instance();
assertEquals(1, pm.getEngine().getPlayer().getLastDy());
}
Pacman pm = SimplePacman.instance();
assertEquals(1, pm.getEngine().getPlayer().getLastDy());
}
Writing such an assertion requires good knowledge of the underlying API, and a bit of luck that the API exposes a method to witness the desired effect.
My next use case involves a hungry Pacman consuming a dot, which earns the player 10 points. Doing this in a way similar to the previous use case iss simple enough. Would it also be possible to assert that the proper number of points are displayed correctly in the GUI? This requires getting hold of the appropriate JTextField, and checking its content before and after eating a dot.
To support this, WindowTester offers a number of widget locators. An example is the locator used above to find a JButton labeled with "Start". Other types of locators make use of patterns, the hierarchical position in the GUI, or a unique name that a developer can give to widgets. I use this latter option, allowing me to retrieve the points displayed as follows:
private int getPointsDisplayed() throws WidgetSearchException {
WidgetReference<JTextField> wrf =
(WidgetReference<JTextField>)
getUI().find(new NamedWidgetLocator("jpacman.points"));
JTextField pointsField = (JTextField) wrf.getWidget();
return Integer.parseInt(pointsField.getText());
}
WidgetReference<JTextField> wrf =
(WidgetReference<JTextField>)
getUI().find(new NamedWidgetLocator("jpacman.points"));
JTextField pointsField = (JTextField) wrf.getWidget();
return Integer.parseInt(pointsField.getText());
}
The remaining use cases can be handled in a similar way. Some use cases require moving monsters, for which having access to the GUI alone is not enough. Another use case, winning the game, would require a lot of clever moves on the regular board: instead I create a custom small and simple board, in which winning is easy.
I less and less make use of the recording capabilities of WindowTester: instead I directly program against its API. This also helps me to make the test cases easier maintainable: I have a "setUp" for pushing the "Start" button, a "tearDown" for pushing "Exit", and I can make use of other JUnit best practices. Moreover, it allows me to create a small layer of methods permitting more abstract test cases, such as the method above to obtain the actual points displayed.
Are the resulting test cases useful? I recently changed some of the GUI logic, turning a complex if-then-else to handle keyboard events into a much cleaner switch statement (inspired by a warning PMD was giving me). All unit test cases passed fine. I almost committed, but then decided to run the GUI test suite as well. It failed. I first blamed WindowTester, but then I realized it was my own fault: I had forgotten some breaks in my switch and the events were not handled correctly. The GUI test suite found a fault my unit test suite had not found.
In summary, automated GUI testing is neither a replacement for unit nor for acceptance testing. It is a useful tool for covering the GUI logic of your application. The recording capabilities can be helpful to try out certain scenarios. In the end, however an explicitly programmed test suite, making use of the GUI framework's API, seems easier to maintain. In this way, JUnit best practices related to test suite organization as well as the application's observability and controllability can be directly applied to your GUI test suite as well.
I prefer steering GUI testing programmatically to working with a recorder.
