Applause, a world leader in testing and digital quality, released the results of its first ever Generative AI Survey. Deployed in August 2023, the survey gathered the insights of more than 3,100 digital quality testing professionals from around the globe about their experiences with generative AI services. The survey revealed that a majority of users are concerned about bias, copyright issues, and hallucinations in content generated by AI.
The August survey was conducted as a follow up to an AI survey Applause issued in March of this year which centered on user sentiment around voice applications, chatbots, and ChatGPT.
As noted in numerous recent examples in the media, bias continues to be a problem with AI-fueled technologies. When survey participants were asked about their level of concern regarding bias in AI-generated content:
-
90% expressed concern, with 25% “very concerned” that bias may affect the accuracy, tone or relevance of the content produced, a 24% increase from the March survey.
-
47% said they have experienced responses or content they considered biased.
-
18% had experienced responses they considered offensive.
Other key findings:
Generative AI in the Workplace
-
Despite the numerous concerns, more than half (59%) of the respondents said their workplaces support the use of generative AI services.
-
23% of that group said there are restrictions in place for using the technology.
-
Of the remainder, 19% said their workplaces do not support use of the technology, and 21% were unsure.
Data Privacy
-
69% believe it is “very” (35%) or “extremely” (34%) important to consider data privacy when developing new AI technologies.
-
67% feel that most generative AI services currently infringe on data privacy.
Copyright Issues
-
91% expressed concern that content produced using generative AI could be in breach of copyright or intellectual property protections.
-
One-fourth of that group were “extremely concerned” about potential breach of copyright.
Satisfaction with chatbots
-
In a sign that generative AI is potentially having a positive impact on the interactions users are experiencing with chatbots, this survey saw a 30% reduction in dissatisfaction with chatbots versus the survey in March.
-
Only 21% said they were either “somewhat” or “extremely dissatisfied” with their use of chatbots, down from 30% in March.
“As the survey demonstrates, generative AI services are becoming more widely used, including in the workplace. While the technology can be extremely helpful and is being used to complete a variety of tasks – from evaluating job applications to writing business contracts, or, even as one of our survey respondents replied, to write electronic dance music, it is important to recognize that the technology is still evolving. Biased, incorrect, and harmful content is a big hurdle to surmount. To improve AI, we need to invest in strategies – in regard to data collection, training, and testing – to identify bad content and take steps toward rectifying the issues. Testing outputs with real people provides the feedback needed to understand these subtleties,” said Rob Mason, CTO for Applause.
The Generative AI Survey is one of a series of surveys Applause has conducted so far this year. In addition to the March Voice and Chatbots Survey, the company released its second annual State of Digital Quality Report in May 2023, which analyzes a representative sample of its testing data and reports on the most common flaws in digital experiences in several industries including retail, finance, media and telecommunications, and travel and hospitality. Later this year, they will conduct the fifth annual global survey on holiday shopping trends.
About Applause Generative AI Solutions
Applause offers comprehensive services designed to evaluate the effectiveness of large language models (LLMs) and generative AI models. By subjecting these algorithms to rigorous testing, Applause helps clients identify areas for improvement, enhance model performance, and ensure reliable and unbiased outputs. The testing process encompasses real-world scenarios and diverse user interactions, providing valuable insights into the algorithm’s capabilities and limitations.
Based on improvement guidelines, Applause uses generative AI to improve the quality of written test cases using the new Smart Suggestion feature within Applause Test Case Management (TCM). Smart Suggestion uses ChatGPT to optimize test cases and return suggestions to improve original test case content. Clear and well-written test cases provide faster time to testing, less time spent responding to questions, clearer reported bugs, lower costs and time overhead, and more efficient test cycle execution.