Survey Examines User Perceptions About Generative AI

When Applause surveyed its global community about generative AI last year, use was on the rise despite concerns about bias. We surveyed the community again this year to learn how consumers are using different Gen AI tools, including chatbots, and how usage and user experiences have evolved as the technology has gained traction. This year, 6,361 consumers, software developers and QA professionals participated in the survey, revealing that while user satisfaction is increasing, opportunities for improvement still exist.

Responses are increasingly helpful.

Last year, 7% of 2,110 respondents said that the responses they received from Gen AI chatbots were always relevant and appropriate, while 50% said that was true most of the time. This year, more than twice as many people participated in the survey and they were more likely to find chatbot responses helpful. Of 4,229 responses, 19% reported that the Gen AI tools they use understand their questions and provide helpful answers every time, and 58.5% said that was the case most of the time.

With more than 2,900 respondents indicating they use at least one Gen AI chatbot daily, there’s plenty of data to assess. In addition, 37.5% of respondents reported that they use different chatbots for different tasks, while 26.5% stated that they have swapped one service for another, typically due to performance issues.

The most common Gen AI UX errors according to 4,174 respondents:

• gave a general answer that did not provide enough detail: 17.5%
• misunderstood my prompt: 16.8%
• gave a convincing but slightly incorrect answer: 10.7%
• generated obviously wrong answers: 10.3%

Use cases are diversifying.

For most users, chatbots replace existing search engines and research tools — 91% of respondents have used chatbots to conduct research, and 33% of those respondents use Gen AI for research daily. For 81% of survey participants, chatbots have replaced search engines for queries; 32% of those use chatbots for search daily. Other popular use cases include language translation, creative writing and drafting emails, proposals or similar business communications.

Not surprisingly, Gen AI has myriad use cases in the software development and testing process. When respondents who indicated they are using chatbots to write or debug code, build test cases, or for test reporting were asked to expand on how they are using chatbots for testing, 1,532 QA professionals indicated that most frequently task bots with test case generation (19%), text generation for test data (17%) and test reporting (16%). GitHub Copilot and OpenAI’s Codex are the most popular tools.

Despite bias, hallucinations and other flaws, users are optimistic about the technology.

This year, 38% of respondents reported that they had seen hallucinations, 50% had seen content they considered biased, and 19% had seen content they found offensive; all 1-2% percentage points higher than last year’s findings. While a greater proportion of users reported they had encountered problematic responses this year, 75% of 4,245 respondents stated they believe that chatbots are getting better at managing toxic or inaccurate responses.

How satisfied are users with Gen AI experiences? More than a third (36%) stated that they’re extremely satisfied, and 53.6% reported that they’re somewhat satisfied. When asked about potential features they’d like to see in generative AI chatbots, survey responses mentioned better source attribution, more localized responses, support for more languages and deeper personalization.

Despite improvements in the technology, some users still aren’t sold on Gen AI’s value: 1,001 survey participants reported they had never used the technology or had only tried it once or twice. When asked why they haven’t adopted Gen AI, 28% of respondents said there’s nothing they want to use it for. While that’s good news for existing search engines and translation software for the moment, almost daily AI advancements may win over more users in the coming year.

The Generative AI Survey is part of the State of Digital Quality content series. In May 2023, Applause released the second annual State of Digital Quality Report, which analyzes a representative sample of testing data and examines the most common flaws in digital experiences in several industries. This year’s report is scheduled for Q3.