A Year of Advancement in AI, Inclusivity and Quality
Brands win at the intersection of innovation and quality. When software development organizations prioritize the customer early and often in their software roadmaps, they earn customer loyalty and higher revenues.
The award-winning Ready, Test, Go. podcast, brought to you by Applause seeks perspectives across different industries and disciplines to explore the many ways quality and innovation shape digital products today. Whether that means implementing AI and LLMs into digital products, encouraging digital transformation across the business or seeking collaborative solutions for daily challenges, we spend each episode sharing the strategies and stories driving change. The goal is to deliver actionable insights and encourage our audience to explore the human ingenuity behind modern digital quality initiatives.
Across 11 episodes in 2024, we explored many topics, but four primary themes emerged.
The customer’s journey and experience are paramount
Technology shapes how we interact, work and live. But it ultimately serves the customer. An app might have a bunch of bells and whistles but fail to resonate with users more than a very simple app that serves its purpose well. Brands can expect prompt abandonment if they fail to meet user expectations.
How quickly will they churn? Try 10 seconds, says Todd Unger, Chief Experience Officer and SVP of Marketing and Member Experience at the American Medical Association. He argues today’s customer journeys travel from awareness to transaction nearly instantaneously — a shift from traditional marketing funnels to a “tornado funnel.”
Take this lesson from a modern pop superstar. “I think there’s a theme song for customer experience,” said Unger, referencing Ariana Grande’s “7 Rings” song, “’I see it. I like it. I want it. I got it.’ Those are really the steps in the digital customer journey. And friction is what lives all the way through that process.”
Organizations must eliminate this friction wherever it manifests. If necessary, place extra emphasis on organizational ownership and orchestration to create awareness around CX issues and fix systemic challenges.
Friction indices and customer-centric metrics can help measure change, but it’s less about the exact measurements and more about progress. “The best data is the one that you act on and that leads to changes for the better, for your customer,” Unger said.
This ideal applies in every market where you serve your customers. Business leaders know the challenges around localization. Yet they often struggle to adapt digital products to new markets due to cultural, monetary, language, regulatory or other challenges. However, as much as we can stand to learn more about the customers we serve, we can learn from the customers we serve. That knowledge can lead to positive change throughout the organization and the SDLC, and it can even lead to positive change around the world.
“Pessimism is a privilege for those who can afford to despair,” said Payal Arora, Professor of Inclusive AI Cultures at Utrecht University. “It’s much like evolution. Evolution has not finished; it’s in process. And so is technology.”
Arora argues that rational optimism fuels innovation and helps Western technology companies better serve the global marketplace. Some examples include surveillance systems of care, where digital tracking technologies enable safety and fairness, especially for marginalized groups. Likewise, global mapping tools underserve some regions.
Webinar
Localization Testing in the Real World
Join our webinar to hear from our panel of localization testing experts who shed light on common successes and struggles with localization testing.
She also claimed big tech companies are at a disadvantage in the Global South, where the next billion internet users will come from, as startups can act more nimbly in hyperlocalizing their digital products. Serving these customers isn’t about altruism; it’s about understanding and relating. And technology companies must reflect the general optimism of these vast and diverse users to innovate in new, effective ways.
“We have no option but to have hope,” Arora said. “Hope gets us out of the bed and gets us to think about ideas for the future.”
AI defines new possibilities
Generative AI can help you cook your dinner, fix your car or even talk to your teenager — as if. But for all the disruptive possibilities of AI in a broad sense, there’s a risk of taking too big a bite of the hype pie. Implementing AI at scale and realizing return on investment is far more challenging than generating that image of a Victorian unicorn riding an outer space rollercoaster.
“The hype that I see, is the gap between expected return on investment relative to where these systems are actually architected for big companies,” said Ben Van Roo, CEO of Yurts, an enterprise AI integration platform.
Start with realistic use cases that have simple expectations. From there, experiment with open source tools and smaller-point solutions, but understand that not every AI project is ready for an enterprise-grade rollout. Above all, Van Roo argues, avoid the bright, shiny, empty promises in favor of proven approaches and offerings.
“[AI is] not touching the things that actually matter that help us run our day-to-day business,” he said. “As soon as companies can really make these tools a part of how they actually get work done at much deeper levels, that’s going to change the game for lots of companies.”
How are early adopters of generative AI solutions architecting and gauging their products? Take a different lens to timeless UX and software principles, like conversational feedback for chat-based apps, to guide these modern AI-infused applications.
“You’re going to want to create a life cycle of improvement, care and feeding [for the conversational assistant],” said Richard H. Miller, technology advisor and author. “This concept of fine tuning, of including additional data…all these kinds of things help you create an enterprise solution that sort of is fed with the right data, provides the right results, and you know that because you’ve monitored it.”
These conversational systems need constant improvement. Miller discusses the need for robust testing matrices and metrics that can inform development — he writes about those concepts in greater depth in his book, UX for Enterprise ChatGPT Solutions. But, as many technology leaders will tell you, the specific metrics are less important than the broader goal: in this case, consistent improvement.
“The importance is how to understand how to get [your scores] to be better,” he said. “And so these metrics are what you can use to see that you’re approaching your goals.”
Josh Poduska, former AI Strategist and Client Partner at Applause, sees workplace adoption increasing in the coming years, placing extra emphasis on governance and monitoring. High-stakes industries like healthcare and finance must adapt quickly and comprehensively to ensure models remain accurate over time.
“If over time trends change, cultures change, opinions change of users of your model, that model is going to degrade over time,” Poduska said. “You want to have some visibility into that. So you set up certain tests, some automated, some human-based, and you track the results of those tests over time.”
Webinar
AI Testing: The Path to Exceptional Apps
In this webinar, we’ll walk through the AI testing framework that outlines the key practices and capabilities needed to evaluate AI tools and improve digital quality.
One of the best ways to keep that data fresh is to source it from real users who are representative of your customer base. As organizations increasingly personalize their AI systems, including implementing multimodal capabilities, crowdsourced testers can assist with data collection at scale and model tuning, but also red teaming and real-world testing to ensure helpful outputs.
“As AI continues to advance, more and more you’re going to need new data collected fresh from humans, usually in a crowdsourcing manner, so that you get a broad swipe of people,” he said.
In the pursuit of effective AI, we must also strive for ethical AI that does not introduce additional harms. Garbage in, garbage out — and that garbage can come in the form of biases that carry harmful consequences for customers and risk for the business. As much as AI can handle in the blink of an eye, digital products still need human eyes to read, refine, inform and influence results over time.
“We could never not have humans monitor AI systems,” said Rishu Gandhi, Senior Data Engineer, Machine Learning & AI at Wells Fargo. “That would be a very scary world.”
Gandhi is a proponent of ethical AI, which aims, first and foremost, to do no harm. That means rolling inclusive thinking into each stage of the development cycle, including involving people of diverse backgrounds and communities into the process. By leveraging domain expertise to refine models iteratively and continuously monitoring those systems to assess societal impact, organizations can align their products with their compliance and ethical goals. Additionally, audits and reviews help introduce unbiased feedback, another necessary
“I think it becomes very important to have that accountability, and also transparency,” Gandhi said.
Collaboration and crowdtesting remain critical
Human perspective remains a vital element in digital quality. Sure, you can automate a lot, and AI can help with the task, but meeting human expectations requires human testing and ingenuity. And when testers apply their expertise in a collaborative fashion alongside designers and developers, the organization as a whole benefits from that proactive approach to quality, one that cannot be replaced by AI or automation.
“Automation is great; it just cannot work alone,” said Mark Kalet, a QA leader. “You need the human element. People are more versatile. You have to be able to move and turn on a dime, and it’s something that automation can’t do.”
Technical competency goes a long way, but Kalet values versatility and personality just as much or more in hiring and evaluating team members. Each team member brings unique problem-solving capabilities and skills, but they must ultimately work within the context of the team to achieve business goals. Kalet argues that a fundamental shift is needed in the tech world, one that values employees as valuable assets rather than disposable resources.
“If you’re seeing people as resources, it’s like cells on a spreadsheet; there’s no distinction,” he said. “They’re numbers. They’re nothing.”
Crowdsourced data and testers play a major role in achieving business goals with digital products. Crowdsourced data helps refine AI models and applications, particularly for nuanced, industry-specific use cases. And while automation lends a hand in daily tasks, human influence matters a lot, whether its testers validating products pre- and post-launch or auditing those systems.
“Third-party testing and auditing is emerging as a really good area to make sure that you have an outside view of how your model or your platform is performing, as it relates to artificial intelligence and machine learning,” said Jason Mills, VP of Solution Engineering at Snowflake. “Applause happens to be one of the leaders in this globally in providing those services…and I think it offers a tremendous value to organizations.”
It’s all about finding balance — between automation and human oversight, innovation and quality. Crowdtesting helps gather perspectives from all over the world that would otherwise be shut out of the conversation. A customer-first mindset and an emphasis on ethical innovation will chart a successful path.
“In this race to automation and AI, I think we will see some governance come in from both government organizations, as well as internal organizations that are trying to figure this out,” Mills said. “Our hope is that we can have a fairer world…and we can provide more insight into the decisions that are made, whether it’s fair lending or whether it’s access to health care or other things. We want to see the world a better place with these solutions and the data set that powers them.”
Industries prioritize digital innovation
Digital transformation and innovation push fascinating possibilities across all industries. Increasingly, the battleground is digital, as brands can address challenges and provide access to services previously considered impossible.
Take healthcare as an example, where providers can engage with patients in deeper, more ongoing ways than ever before. Dr. Liz Kwo, Chief Commercial Officer at Everly Health, refers to this as the engagement continuum, where digital interaction might begin with a search or a passive reminder and evolve into digital diagnostics and treatment. AI and voice, in particular, offer quite a bit of promise for brands that invest in implementing and refining those technologies.
“It’s about augmenting medical professionals’ ability to make informed decisions,” Kwo said. “With AI, vast amounts of data can be analyzed and aid in identification of disease at their nascent stages.”
Ebook
Testing Generative AI: Mitigating Risks and Maximizing Opportunities
Learn how a human-in-the-loop approach helps developers refine AI responses to better meet human expectations and address complex ethical issues.
Health and fitness brands, however, must prioritize trust and accessibility within their products. As consumers wear digital devices throughout their day and place more trust in digital health services, there’s a great regulatory and ethical need to protect consumers from data breaches or poor health outcomes. While there’s potential for harmful outcomes, there’s enormous positive potential as well — consider how health tech companies are uniquely positioned to tailor solutions to specific populations, such as Medicaid users or non-English speakers, while also leveraging local providers and advocates.
“There is sometimes a lack of trust among those that don’t always engage in the healthcare system, but need to or should because early diagnosis and prevention can actually lead to better outcomes,” Kwo said. “When I think about how to engage patients to support them, you really have to build that trust.”
Building trust applies across all industries, even if patient bloodwork and x-rays aren’t involved. Take streaming media as an example. With more services than ever and an increasing demand from subscribers to deliver high-quality platforms and content, media brands have to compete for hearts, minds and wallets.
Start with data to make informed decisions. Well-governed data and informed decision-making helps organizations create personalized recommendations, maintain subscriber engagement and prevent churn.
“Without the data, you’re just taking guesses on what people are looking for,” said Stewart Frey, a media technology and solutions expert.
The same intelligence and investment should go into AI and programmatic advertising, which can deliver targeted, personalized ad experiences. Of course, this creates a balancing act for streaming brands still trying to crack the code to consistent revenue streams. The path to profitability is not in cutting investment in innovation and quality, but rather iterating on it to deliver better experiences.
“Nobody has patience anymore,” Frey said. “They go to [social media to complain], and that devalues your brand. So I think media companies need to keep investing in the technology. The technology needs to get better. They need to invest in the content. And they need to invest in the video quality as well.”
Subscribe for more insights
The award-winning Ready, Test, Go. podcast, brought to you by Applause brings technology leaders, authors, speakers and experts to the table to discuss a wide range of digital quality topics. Whether you’re intrigued by the transformative power of AI, inspired by the collaborative potential of dev and QA organizations, or curious about industry-specific innovations, subscribe today to keep up with the trends:
If you’re ready to elevate your digital quality strategy, explore how Applause’s solutions can empower your business. Whether it’s world-class manual functional testing, improving inclusivity through accessibility or localization testing, or validating AI outputs, Applause is ready to assist with a global community of more than one million digital experts. Let’s talk today about how we can help you achieve your digital quality goals.
Ebook
The Essential Guide to Crowdtesting
Learn how crowdtesting really works, as well as how it complements your in-house teams to deliver proven value.
22 min