How Engineering Leaders Are Adapting Software Development and Testing for the Age of AI

AI is transforming software development, enabling engineering teams to generate code faster, automate workflows, and accelerate product delivery. But increased speed doesn’t automatically translate into better software, higher productivity, or improved digital experiences.

In a recent webinar, Get Ready for What’s Next in Software Quality, I sat down with a panel of experts including Aatish Salvi, CEO at Applause, Adam Smeal, VP Digital & Engineering at Cava, and Andrew Lau, Co-Founder and CEO at Jellyfish, to discuss how engineering leaders are adapting software development and functional testing strategies for the age of AI. Read the responses to some of the key questions they debated.

How is AI changing development and testing, and how can you integrate it effectively into the SDLC?

Over the last few years, organizations have become increasingly invested in the latest AI solutions, moving from AI-assisted coding tools to more sophisticated agentic workflows that can execute complicated development tasks. Though each of these solutions have introduced new features and capabilities, most companies have yet to see the expected yield from their growing AI stacks. Engineering leaders frequently find themselves juggling multiple platforms while struggling to understand why their teams are failing to achieve the expected ROI.

“Each moment in time introduces new processes, new practices,” said Lau. “And companies are thinking ‘I bought all the tools, I think I’m getting them right, but I’m not seeing the yield out of this stuff.’ And it’s creating cognitive dissonance,” because their reality is falling short of what they read online that others are achieving.

“AI doesn’t have skin in the game,” added Smiel. “It lacks context, it lacks guidelines. So how do we take this agentic-style coding that the team has adopted and put personalization on top of it?” He emphasized that at the end of the day, humans will still be held accountable for the incidents and defects that show up in production. So, building organizational context into AI workflows is crucial. When AI has access to coding standards and architectural guidelines, the resulting outputs become more consistent and reliable.

Smiel discussed his personal experience with this practice: when Cava was introducing AI into the SDLC, they saw an increase in defects. The AI was lacking the stylistic differences and the personality you get from engineers who are actually building the products. By adopting AGENTS.md files and more context-driven development, they started seeing fewer defects and better quality. The code actually felt and looked more like a human wrote it.

“One thing that hasn’t changed is no matter how much AI you put in the system, the defect rates don’t shift radically,” said Salvi. “No matter how far or fast AI has evolved or where you’ve interweaved it, there is still a requirement for humans in the loop.” Effective AI adoption demands more than an abundance of tools. Deliberate, human-led changes to development workflows and quality controls are required to maximize the value of AI in the SDLC.

How can you manage the competing pressures of AI adoption, cost and governance?

“The word AI is actually starting to mature in the sense that we are using that term too generally,” began Lau. “For most corporations, there’s actually three distinct use cases we need to separate.” He went on to describe each of these categories of AI use as follows:

• Production workloads: AI used within a company’s products, where customer needs and cost optimization are top priorities.
• General IT: AI applications used by employees across different functions, where access, authorization and spending are key focus areas.
• Software development: AI used by engineering teams, where concerns about cost must be balanced with compliance, innovation and development speed.

Smiel offered a similar viewpoint, breaking down Cava’s AI usage into three buckets: operationals/consumer-grade AI, productivity AI, and AI orchestration. He explained that Cava is actively working to understand how employees use these tools and whether they are using the tools available to them as efficiently as possible.

“Token usage is a real problem. And what we’re realizing is that when given the tools, the business will use it for quite literally everything,” said Smiel. “When you need Google to find the nearest pizza place, you are now using ChatGPT and burning through 2,000 tokens to find pizza. So there’s a bit of a governance problem: Why are we giving you a jackhammer when all you need is a hammer?”

Ensuring that people are using the appropriate tool for the task at hand remains a challenge for many organizations. AI adoption should be intentional. While teams might be able to generate code faster than ever, evaluating the tradeoff between the value generated and the costs incurred is critical for realizing the potential of increased AI adoption.

How can you tell whether your AI workflows are actually delivering value?

An organization may have a good idea of what it wants to accomplish with its AI workflows. However, understanding the gap between perceived productivity and measurable business outcomes remains a key challenge.

Salvi shared an anecdote where he asked a room full of people if their organizations had implemented agentic workflows to improve internal efficiency and whether those workflows had allowed them to reduce the amount of human labor involved. The response to both was an overwhelming ‘yes’. However, when he asked whether they had seen an evaluation or report card that proved the agents were performing the tasks better than humans used to, the room fell silent.

“This is why right now it’s incredibly important to have some intentionality around practice patterns and benchmarks on what you’re shooting for,” said Lau, “because the ball is rolling so fast.” Employees may report that AI helps them work faster, and organizations may automate tasks previously performed by humans. And yet, many lack objective evidence that these workflows consistently produce the same or improved results.

An AI model may be working fine, but at the same time costing a ton of money. Salvi posed another question for organizations, asking “Have you looked at other models and plotted yourself on the cost-benefit curve and really understood the tradeoff you’re making in using the premier models that you’re using?”

To accurately gauge how a model is performing, it’s crucial to define what success looks like before deploying AI at scale. “You have to look at it holistically and actually isolate what you’re gunning for,” said Lau. Once success is defined, an effective evaluation should consider:

• Quality: Does the AI generated output meet requirements and perform as intended?
• Cost: How much does each workflow cost to operate, including token usage and infrastructure?
• Productivity: Does AI reduce total effort or meaningfully improve delivery speed?
• Customer experience: Does the output improve CX?
• Reliability: Does performance remain consistent as models, prompts and workflows change?

Salvi concluded that “the number of different things that could be the root cause for why you’re not getting your expected result is very large.” Because of this, evaluation needs to be a continual exercise. Organizations need repeatable ways to monitor performance and reassess whether their workflows are meeting expectations.

What should engineering leaders focus on in the next six months?

The panel wrapped up the discussion by looking toward the future of software development and testing. Lau stated his belief that the next six months “will bring better practice and innovation to verification,” but said he believes change will be the defining feature of the AI development landscape in the coming months.

Smiel expects that “context will be the biggest focus point for what we see over the next six months.” He argued that current approaches to providing AI with context are not scalable. For that reason, he believes there will be an increased focus on how information will be connected across the SDLC, without sacrificing code quality or having to consume more tokens.

Finally, Salvi predicted that “model evaluation that relies on specific use case-based benchmarks for whether your agents are doing the job right is going to become an increasingly important aspect of how you evolve your relationship with AI in the SDLC or the consumer experience.” He hypothesized that organizations will need to think beyond token-maxxing and instead understand whether their AI workflows justify their costs.

Ultimately, the challenge for engineering leaders is not simply adopting AI. Rather, it is understanding how to use it effectively, verifying that it does what it is supposed to do, and ensuring that the results justify the investment. To learn more about how you can evolve software development and functional testing strategies for the age of AI, watch the webinar, Get Ready for What’s Next in Software Quality.