4 Ways to Get Maximum Value from Exploratory Testing

Throughout my career in QA, I’ve come across many people who didn’t see the point of exploratory testing. In old-school testing organizations, leaders often think that if they can’t see and measure exactly where testers are spending their time, they can’t be certain those efforts will turn up anything useful. These leaders prefer to focus QA on unit tests, smoke tests, regression tests and structured manual testing – areas where they’re confident their teams are being productive (or can at least measure and monitor their activity).

However, when development teams think outside the confines of structured testing, they can better understand error conditions, edge cases and other areas of friction that users experience. Identifying the different — and often unanticipated — ways customers interact with apps and websites is essential if you want to deliver the best possible experience.

You need exploratory testing to uncover problems that occur beyond the happy path. But that doesn't mean you simply hand your app off to testers and let them poke around until they uncover a problem. With a little structure and guidance, you can set your testers up for efficient exploration, pointing them to the areas where problems are most likely to occur. Follow these four recommendations to get the most impact from exploratory testing.

1. Provide context about what's different.

Without context, testers will test anything and everything – including parts of the app that have not undergone any significant updates. When organizations see unstructured tests as a waste of time, it’s often because they don’t focus on the portions of the app that are most likely to have defects. Exploratory testing shouldn’t be like looking for a needle in a haystack; testers should have some guidance on where to concentrate their efforts for the best chance of success.

Ask testers to perform timeboxed testing on specific processes and parts of an app that have changed. If you tell testers that you fixed a registration issue related to double-byte character sets, they’ll focus on that process. Share release notes and changelogs so testers can apply more scrutiny to the parts of an app developers have touched. If you’re organized up front, it’s easy to see where exploratory testing makes the most sense.

2. Get a real customer’s perspective.

When I led the QA team for a telecommunications provider, we pushed out major releases quarterly because the updates were so intrusive and complex for our customers. And every quarter, within an hour or so, my team would report major issues. I’d go back and look at the smoke tests and regression tests that we ran on the whole build and discover that the issue that came up was something we never envisioned. We weren’t using the software the same way our customers were.

In one instance, creative users figured out how to get around some of the checks and balances we put into our system. Customers wanted to keep the same dial-in and passcodes for quarterly earnings calls, which they couldn’t do if another customer had already reserved the conference bridge for the same time. Creative call center folks realized they could reserve the numbers they wanted if they scheduled the call for a weekend, then edited the date and time for the call. Our software wasn’t validating that there were no conflicts on a date/time edit. The workaround customers used meant we had a critical issue on our hands. Had we tested beyond the sunny day scenarios we had scripted, in a manner that mirrored the ways our customers used the system, we could have found that problem. These issues can be avoided by getting feedback on how your customers use your system and building those key use cases and functionalities into your testing strategies.

You never realize how users will use your software until you give them the opportunity to put it through its paces. They will almost always discover things you didn’t explicitly script.

3. Take advantage of testers’ experience and creativity.

Experienced testers have a sense of what can go wrong. They’re quick to try to poke holes in areas that commonly pose problems. They bring what they know about your industry, apps in general, and problems that lie beyond the happy path to your tests. An experienced tester will examine edge cases they’ve encountered in the past to flush out hidden issues that regression and smoke tests simply won’t reveal.

This is one of the places where the community model delivers an advantage – it’s easy to quickly find skilled testers who can uncover unexpected issues. The depth and breadth of Applause’s uTest community allows us to source testers with a keen understanding of different industries, types of apps, and customer experiences: testers who understand where to focus their exploration to find valuable defects.

4. Get a fresh perspective to prevent tunnel vision.

When you’re used to doing something, you tend to develop a consistent process. But if you test the same things in the same way all the time, you’re likely to miss edge cases and unexpected use patterns. This is another area where the community model delivers an advantage. It’s easy to rotate in fresh testers to get a different perspective, as well as expertise on environments you may want to test that are outside your core capabilities. Regularly invigorating your testing team to put an app through its paces ensures maximum coverage and the greatest potential to find truly valuable issues. Introducing a fresh set of eyes can also help other testers on the team learn which edge cases to examine in the future.

Carefully planned and thoughtfully executed exploratory testing can help companies dramatically improve the user experience. With the right context and perspectives, unstructured testing can deliver valuable insight.