How Much Testing Is Enough?

How much testing is enough? There is no universal answer, but the concept of risk provides a reliable way to guide the decision.

Software testing as always is best when it’s known, intentional, planned and defensible. This blog introduces risk-based testing as defined by risk-based frameworks to determine how much testing is enough. Factors like the impact on customers, the business, development teams and application complexity go a long way toward achieving that definition. The end goal is to release an application where testing is strategic rather than reactive.

What is risk?

In a software application development context, risk involves exposure to danger, harm or loss. Some elements that introduce risk include code complexity, integrations with data and/or third-party software, APIs and security. When you consider the many devices, platforms and networks that customers use, it’s a nearly endless list of potential risks.

Applications have multiple points of failure. Some failure points are out of an organization’s control. After all, a business can’t control if one of their vendors changes platforms and experiences significant downtime. Or if a database provider might decide to make a significant security change without notice, and the changes cause applications to stop functioning.

Product and application risks can vary because each code component and function carries different levels of risk. A display bug on the main screen might be an annoying, minimal threat. But a critical login failure that prevents users from accessing the application poses a significant business and customer satisfaction threat. Severe failures in processing, security or data breaches can put a company out of business. Ongoing evaluations and testing are the keys to minimizing risks.

Benefits of risk-based testing 

Risk-based testing (RBT) is an effective way to shift QA from being reactive to a more strategic, proactive testing approach to reduce risk by prioritizing critical path tests. RBT can work for any existing development methodology and team. It’s a valid quality engineering approach that works to improve customer satisfaction and application quality.

RBT is flexible and strategic, fitting within Agile or DevOps teams or alongside crowdtesting teams for added business value and real-world test coverage. Quality applications with high customer satisfaction are how application providers meet business goals. 

Evaluating risk and test coverage

To assess the highest priority areas for test coverage, start by creating a list of risks based on application functionality and customer usage. When testing focuses on high-risk areas, teams can help to reduce the chances of critical failures. Plan testing timelines around the most critical functions to reduce customer downtime. For software that must satisfy regulatory requirements, best practice is to add a risk profile that covers each compliance regulation. For example, applications in certain industries are required to meet regulatory requirements for security and data privacy. Prioritize assessments in these critical risk areas. 

Get the full team together. Invite the customer support and operations group to assist if they aren’t already part of the team. Each member of the team has a knowledge of the application based on their job role and experience. Customer support and operations typically have a stronger understanding of how customers use an application and what issues they may encounter. 

The customer perspective helps determine what application functionality carries the highest risk. A display issue on an infrequently used administration dashboard potentially carries far less risk than a failed payment process, an incorrect calculation or a security or data access issue. Keep in mind though that customers can only see what they can see — something like a security vulnerability will be invisible to them.

To effectively validate the customer perspective and mitigate high-impact risks, Applause® partners with organizations to move beyond the happy path of lab-based simulators and dummy data. A global community of real customers using their own real devices lets QA teams uncover the unpredictable, edge-case defects that only occur in the wild. This approach helps teams assess critical user flows, such as executing real transactions with live, localized payment instruments.

Crowdtesting teams can also support validation tied to regulatory or standards-based requirements, such as accessibility criteria, localization requirements, geofencing, payment flows or other market-specific test scenarios. With a well-curated crowdtesting team and the right program expertise, organizations can expand practical coverage across priority OSes, devices, testing categories and customer profiles that are difficult to validate through internal lab-based testing alone.

Risk-based frameworks

Development teams that prefer to use standard frameworks to organize and plan testing can choose from several risk-based frameworks. Frameworks essentially help teams perform risk assessment on applications by working through potential consequences of failures. The framework determines test coverage or confirms that the existing testing plan provides reliable risk coverage.

Rather than attempting to execute all tests possible, teams can identify areas of the application or code with the greatest risk of system failure. Then they can focus testing on the areas with the highest probability of severe failures.

Consider the following standard risk-based frameworks: 

• Product Risk Management (PRisMA) identifies risks throughout the SDLC to increase the chances of customer acceptance. 
• Rapid Risk Assessment (RRA) translates abstract risk discussions into testing priorities using a template to ensure the team is in agreement. 
• Quality Functional Development (QFD) incorporates customer requirements directly into design and testing objectives to help avoid critical requirements that are missed in coding or testing. 
• Cost of exposure quantifies risk by calculating the cost of defects or critical failures. The intent is to ensure tests are created to cover all critical risks and avoid negative business financial impacts. 

Options for implementing risk-based testing

Releasing defective software poses extreme risk — customers can abandon a cart, switch to a competitor or leave a negative review far more easily than was possible in the past. Poor quality software systems introduce potential financial constraints or business effects, ranging up to business failure and job loss.

However, implementing risk-based testing can be straightforward without extra costs or time delays. Here are the main steps to start practicing risk-based testing: 

1. Pull key stakeholders together, including representatives from development, design, testing, customer support and operations. Have this core group review the application functionality to identify the critical failure functions and assess risks related to security, data integration, third-party software and APIs. 
2. Decide whether to leave low-risk areas untested, test them via automation or test them periodically; if product teams decide to execute test cases identified as low-risk, then any test failures should not stop a release.
3. Consider using AI tools that might be able to identify the highest risk areas in the application. But remember that an assessment of risk must be accurate for the bulk of customers and how they use the application. Ensure that human-in-the-loop processes are followed to ensure the accuracy of any AI tools.

Consider crowdtesting for expanding test coverage throughout development stages. Applause is a managed software testing service that addresses this need by combining a global, on-demand testing community with AI and automation. For many organizations, the goal is to scale testing coverage and free up internal QA teams for additional quality tasks. Applause helps them do that.

Improve quality and customer experiences with risk-based testing

Reputations and brands are built on customer trust. Trust grows into loyalty when customers perceive the value and quality they pay for. And, if you fail customers, they often turn into public failures — social media shares, problematic headlines and poor app store ratings.

Applause can help you reduce critical failures and protect your customer experience. As a fully managed service, Applause combines a global, on-demand testing community with AI and automation to help brands achieve extensive scale and real-world test coverage. This empowers development teams to catch critical defects before they reach production and provide a better user experience. Learn more about how to get started with Applause today.