New Software Testing Methods in the AI Age
AI is changing how software testing is done. From planning a QA approach to defining test types and techniques, software testers must grapple with the nature of their work fundamentally evolving.
Software developers and testers share the burden of adapting as AI changes how development teams function. For software development teams, that means redefining and rethinking assumptions about software behavior. New responsibilities arise — along with different types of risk. Testers and developers must be ready to create new testing methods and techniques that support, challenge and validate AI.
Here are some of the ways that I anticipate the work of testing will change as AI evolves.
What are the new testing methods and techniques?
The exact testing methods and techniques that AI will influence are currently being defined. These new methods will come from collaboration between developers and testers. They will amplify the need to test how users behave with an application. Effective testing has always aimed to validate the full user experience, but AI introduces complexities that demand more focus on behavior with an application. The differentiation for many products will be in how easily a user can navigate the experience.
Webinar
AI Testing: The Path to Exceptional Apps
Explore crucial components of an AI testing program, discover current trends and learn ways to effectively reduce risk and improve user experience.
The area where this might be most evident is in agentic AI. Currently, we might expect that an AI agent can be programmed to to assess or validate:
- functionality
- API usage and access
- permissions and access
- accuracy and relevance of outcomes
- application design around AI agent results
Traditional software development creates logical and defined paths to execute application functions. Agentic AI relies on defined prompts that enable an AI agent to act — interpreting instructions like a user would, deciding how the function works and creating strategies to accomplish goals.
It’ll be up to developers and QA testers to debug agent behavior, just as they assess user behavior. Developers can set up an autonomous playground with guardrails for AI agents to explore and test. An AI agent uses natural language instructions so that developers will write instructional text alongside code. Think of it like doing a line-by-line code review for a new or inexperienced developer and trying to understand their thought process based on written code.
Comprehensive QA has always sought to verify that an application achieves its intended goals, even anticipating unexpected user interactions. With AI, this verification becomes non-deterministic, requiring testers to validate that the agent achieves goals and manages outcomes. For example, can the AI agent avoid undesired outcomes, such as unplanned error messages? Can the AI agent hit all the planned error messages that guide users? When a function fails, is the AI agent able to recover and complete the task?
This type of testing emphasizes the importance of flexible, adaptive test scenarios — living documents that change as development and testing teams complete tasks — long part of the testing arsenal. Testing validates that an AI agent can perform the intended tasks and that its behavior is accurate. Testing focuses on acceptable and unacceptable behaviors. It’ll still be up to human testers to use exploratory testing to uncover critical edge case defects.
Test scenarios will confirm that a logical AI agent can complete stated goals, use the direct or happy path successfully, and get the expected output or result. Testers will evaluate the AI agent’s behavior to verify that the code is appropriately understood and the results are consistent.
Testers must understand the rationale behind an agent’s decisions, discover defects in reasoning and request changes to improve AI agent behavior and decision results. This is an evolution of the tester role — evaluating the nuanced performance and actions of an agent just as they would a human. Testers will review the work of agents as they would review the text execution process of a green or new human tester.
Why will all of this change?
The dynamic nature of evolving AI technology means teams must re-evaluate the task structures of their work. Adapting how development and testing function helps fully leverage AI’s capabilities and transition toward a continuous application delivery loop.
Development and testing in the age of AI must harness the ability to test more pathways through an application while avoiding negative results like crashes or agents taking invalid paths or achieving inaccurate results.
Organizations just getting started with AI should take a proactive approach to navigating the new risks and responsibilities that arise.
Get prepared for AI outcomes
What’s the best way to start preparing for the impact of AI on development and testing? Focus testing on behavior, not just functionality. For some teams, this adjustment in testing and development will take some time to adopt and adapt. Combining work tasks and roles within a team is necessary to provide support for both development and testing. The point is to maintain high-quality application delivery in a continuous loop, a proactive approach that helps teams navigate the new risks and responsibilities that arise with AI. This ultimately drives better outcomes for customers.
As a collaborative team, developers and testers can create effective customer workflows or test scenarios. Testers automate these scenarios and add in manual exploratory testing to identify usability issues or edge cases that automation misses.
By focusing on software quality in a continuous loop, teams can address issues wherever they find them in the SDLC. Rapid issue detection and correction typically improve the customer experience.
Why customer experience is still key in an AI world
It’s essential to improve collaboration and work toward a continuous release model with thorough testing. The end goal is to improve application quality and keep the customer experience as priority number one.
Even in an AI development world, the customer experience remains key. Software development organizations must pay attention to brand reputation and always prioritize customer privacy and data security. AI technology is impressive and valuable — until it begins to provide inaccurate or harmful responses. Organizations must focus on digital quality when using AI, which includes supporting the human factor. AI might test, create and evolve, but customers are humans, and humans look for quality, security and usability in digital products.
Ebook
Testing Generative AI: Mitigating Risks and Maximizing Opportunities
Discover why human feedback and judgment are crucial for testing Gen AI, helping you address inaccuracies, bias and safety in AI applications.
Crowdtesting is a proven method that leverages customer-centric testing that’s fundamentally human. Applause tailors crowdtesting teams to the specific experiences, demographics and platforms you need. Diverse testing teams assess the application or product across device types, operating systems, versions, networks and browsers. Applause handles all of the scheduling and management, providing organizations with user-focused testing from skilled digital experts. Crowdtesting teams also work well when paired with an internal test team. Internal test teams coordinate to keep testing results and defect management organized and efficient.
AI technology will continue to evolve and create challenges for both testing and development. AI technology used for software development changes the work of development and testing. Testers and developers must prepare to make rapid work task changes. New risks will arise as AI evolves — be ready to tackle the AI age’s challenges and continue delivering high-quality applications that drive results for customers.
Ebook
The Essential Guide to Crowdtesting
Learn how crowdtesting delivers a powerful competitive advantage, enabling organizations to quickly test digital experiences and pinpoint critical bugs before they reach customers.
7 min