GUI Testing in 23 Minds

During the Eclipse testing study, we ask participants about their experiences and their opinions on testing Eclipse Plug-ins and RCP applications. In this blog post, we want to share an excerpt of what some Eclipsers really think about automated GUI testing.

What the Adopters say:
Four participants out of 23 actively use GUI testing tools and have automated UI test suites.
P3 experiences that GUI testing is better suited to capture the customer's perspective, because “the tests can capture implicit assumptions and the workflow.” The company of P3 wrote their own tooling, since one fundamental problem of other tools is: “if you [as a tester] are not in the position that you can develop yourself, you always wait for something, or somebody.” Now, with their own testing tool, GUI testing is one of their strong points. P3 knows: “anything that will affect the user, we prefer to write as an acceptance test. [...] It’s a way of getting those discussions going.”

Also P11's team uses automated GUI tests. He reports: “One thing that is really good is the extension of GEF for the SWT-Bot. It is hard to test a graphical editor. We can test them very easily, and we are very happy about that.“

Still, not all are completely satisfied with the tooling. In P18's company they use SWT-Bot and also developed their own capture tool helping to develop test cases. P18 reports: “We haven't been 100% satisfied with the capture-replay, because too much is captured.” Their solution is: “after a capture, we always have a review to remove unnecessary code.”

Also P10’s company uses QF-Test to automate GUI tests. From his experience the maintenance of the test suites takes a lot of effort, he says: “it happens that during product evolution, suddenly something works differently and the tests do not work anymore. [There are] synchronization problems, sometimes the test has not been set-up in a clean way, or timing problems occur. To cope with that it takes a lot of time.”

What Non-adopters say:
All other participants do not invest in automated GUI testing. Some participants used it for some time, but they report that the investments are too high and the benefits too low. Maintenance problems and the disability to cope with evolution are two problems mentioned by many participants. P15 says: “In my experience, automated UI testing is very expensive with not a big benefit, especially [if you have a lot] of change, which makes it a high investment which might never lead to a benefit over running some manual tests. Although manual tests are manual, they tend to be more flexible and lead to better benefits.” And P14 says: “What we had was a QF-Test suite [...] but it became apparent that those [tests] are too rigid to use them further [if software evolves]. That's why we stopped using them.”

P21’s team developed in two projects GUI test suites, but now they discontinued.
His resume is: “The decision has been done mainly by us [developers]. We put an immense effort to write UI tests, [...] and in the end often there was more test code than code to test. I doubt that makes sense.” He also knows: “There has to be a strong commitment from the customer to invest in.”

Also recorders are not beloved by many developers. Participants report that “recorders are mindless” (P4), and “superficial” (P7). “When I use [those tools], I haven’t done the really hard work. [...] with all this investment, I still do not have what I want.” (P7).

The set-up and the configuration of the build environment to execute graphical user interface tests is experienced as a hard and time consuming task which requires a lot of knowledge and expertise.

What the Sceptics say:
Many participants see an alternative solution to automated GUI testing by keeping the GUI as small as possible and in decoupling the logic behind a GUI from the GUI code. P1 even thinks that “80% of the code are already tested with unit tests”, and also P16 says “we do not have UI tests, because we can cover that with unit tests.”

Conclusion
Benefits of automated UI testing reported in our study are the ability to bring in a customer perspective in the tests, foster discussions between developers and testers, and to cover code that is hard to test otherwise. The main risks of incorporating automated GUI testing recognized by the participants are the high effort needed to create and maintain the test suites, the unclear benefits compared to other types of tests, and the usage of immature tooling with inherent problems (e.g., timing, synchronization, set-up and configuration).


Do you have similar/different experiences? Which role has GUI testing in your daily practice? Please leave a comment!



Interested? More results? Come to our talk at EclipseCon. Come, see and share your experience.