Comprehensive Testing: The Attainable Dream

Comprehensive testing is the goal, but how do you get there? It’s all about variety: going beyond feature, smoke and functional regression testing, using a variety of automated and manual test types. And comprehensive testing is often not possible with only an internal QA team, as most teams don’t have access to the variety of devices, platforms and geographical locations to perform the task.

In brief, it’s easy to fall short of what would be considered comprehensive testing by modern standards. The same might be said of security or accessibility testing, as both require specific experience or resources that most QA testing teams do not have.

This blog describes what comprehensive testing looks like, how to achieve it, and what challenges you will face along the way.

What does a comprehensive testing approach look like?

Comprehensive testing starts by developing a detailed and thorough test plan or strategy. The test plan doesn’t have to be a long, formal 50-page document that nobody ever reads. Instead, use checklists, outlines, mind maps or anything that ensures all elements will be tested.

A comprehensive test plan should include:

• realistic expectations based on current resources

• budget figures for supplemental testing

• a variety of test types and techniques

• unique tests that are not repetitive

• time to discover edge cases and other unplanned test scenarios

• details for collaboration amongst testers using a feedback loop

• business understanding of what the customer expects to do with the product

Many QA teams tend to repeatedly test the same suites over and over again. The same repetitive scripts eventually fail to find defects. While it might be inconvenient to abandon a suite that was helpful in the past, comprehensive testing’s goal is to test the application to the fullest, covering as many lines of code or logical decisions as possible.

Consider having developers create unit tests to serve as your smoke test suite. With quality unit tests, testing coverage increases dramatically to cover nearly all logic in the code. However, that doesn’t mean the code is free of bugs. Even well-written code contains defects and non-happy path scenarios that depend on other systems that require testing, such as database connections and APIs.

Next, the testing team must develop different types of tests. For example, consider spending time allowing testers to hunt for defects in unusual places. Exploratory testing helps find defects in application workflows and functionality that otherwise might escape notice. By exploring an application, testers can find and test edge scenarios and odd use cases that might hide a cache of significant defects.

Variety is the spice of comprehensive testing. Ensure variety in testing types, techniques and testing suites. How much testing a QA team can cover depends on the team’s experience and openness to active collaboration or communicating through feedback loops. Teams that fail to communicate will enter duplicate defects and repeat testing. Ask yourself how you can get the most from a QA team while still testing comprehensively.

How do you supplement QA teams and keep quality high?

QA teams must strive to realize a shared goal: comprehensively testing products to find hidden and critical defects. QA teams must be comfortable sharing knowledge collaboratively with external teams to achieve this overarching goal.

Supplemental testing resources include:

• crowdtesting to cover device and platform testing, as well as variances in connectivity, culture, experiences and payments in different geographical locations

• security testing that validates the cybersecurity fitness of the product and its integrated parts

• functional validation done by a team of new users or crowdtesters that offer fresh eyes on the product

• performance and load testing if not covered by unit or functional tests

There are three primary challenges with hiring supplemental testers or crowdtesters. First, you must ensure teams get the information and support they need. Two, make sure the internal team does not feel threatened with job loss — these resources should supplement, not replace, internal testing efforts. Third, commit a testing resource to help train external teams and serve as a communication point for questions. This testing resource is critical to help external teams get the information they need and analyze the results of their testing. Assign a QA manager or project lead if necessary — just be sure to enable their success by giving them ample time to perform the work.

Podcasts

Now that you’ve expanded your testing efforts and added a wide variety of testing types, techniques and suites, let’s make sure the entire application is being tested.

How can you measure comprehensive testing?

Many QA teams spend a significant amount of time trying to measure test coverage through metrics and reporting. Test coverage is a means of measuring the testing breadth and depth across an application. Test coverage metrics include:

• statement coverage — the percentage of statements in code that are executed during testing

• branch coverage — the different decision points to be tested

• path coverage — the line-by-line coverage all potential execution paths

Unit tests are the most direct way to measure code coverage because it’s easy to map what part of the code each unit test covers. The greater challenge with unit tests is measuring manual and automated test scripts to determine what lines of code they cover. Many QA teams use a test matrix design to ensure they cover all logical decision points within the functionality of an application. Creating a test matrix requires time and application experience, and it is still difficult to link directly to the code.

Let’s be honest, most QA teams neither have access to code nor know which tests exercise what part of the code. Instead, they go by experience, mapping out all the application functions, then grouping them in use case-type scenarios. Testers include positive or happy path tests as well as tests meant to break the logic.

Testing is often as comprehensive as the knowledge and experience of the tester. Most teams don’t have the luxury of tools or time that measure code coverage. Developers are lucky if they create unit tests, let alone help QA testers track down code coverage. In the real world, comprehensive testing is built by a tester’s experience and application knowledge using a variety of tests and techniques.

Some additional options for measuring your comprehensive testing efforts include:

• escape rates

• user acceptance test (UAT) results

• customer-reported defects

• customer support calls

What are the challenges with achieving comprehensive testing ?

The primary challenges for performing comprehensive testing are time and money, in that order. Executing a variety of tests takes time, especially when you add different types of testing techniques as well as the devices and geolocations of your end users.

Crowdtesting goes a long way toward achieving device coverage. If you can’t source crowdtesters or deploy a crowdtesting provider, consider prioritizing a list of device types and platforms to test. However, most cloud-based device farms are not free, so there’s still a cost associated with the approach. Crowdtesting works best for this purpose because it’s a rapid, independent and varied assessment. Test coverage expands exponentially when validating a wide variety of customer environments, devices and performance based on location.

Additionally, many QA teams are stretched thin by new feature testing. Add in more elements of comprehensive testing and you may find yourself light on resources. Review the skills of your testers. See if assigning a tester with the appropriate skills to do performance or security testing can work. For each QA team, the challenge is doing the best you can with the resources and budget allotted. So, discovering each tester’s skills can help hone the focus.

The next challenge is release schedules. Depending on the timing of release schedules, the QA team might only be able to allot a certain amount of time to any individual comprehensive testing task. Designing comprehensive testing suites that cover the high-priority tests for each area helps maximize testing. Consider creating small groups of tests and rotating their execution based on the available testing time and resources. Even if you are not able to execute all tests, more testing is always better.

Remember to leverage the skills of your internal or supplemental crowdtesting teams so testers focus on areas of expertise when possible. Applause, as a leading digital quality services provider, can help establish a successful crowdtesting program that delivers value back to the business while covering gaps in testing.

Ebooks