AI Assisted Shopping: What Users Expect

To set the stage, Applause surveyed members of the uTest community to see how different people were using AI in their holiday shopping experiences. The survey found: 

Respondents indicated a variety of different ways they might shop using AI. The most popular use cases for AI in shopping included:  

(n=1,163)

In addition, many organizations are developing agenting shopping and purchasing agents, which streamline the end-to-end buying experience. 

Garcia explained the two primary approaches to agentic shopping experiences: those built by the frontier models themselves and those where retailers partner with those models to create their own experiences. 

“In the current state, there have been a number of experiences designed directly by some of the AI frontier model products. When you think about products like ChatGPT, Gemini, Perplexity, and the likes of those frontier models, they’ve designed end-to-end experiences for shopping that really haven’t involved the retailers. So that’s the part where retailers really have no visibility on what’s happening,“ Garcia explained. In these instances, instead of starting a buying journey with a Google search, many consumers are starting at a different point, using GenAI models.

Garcia predicted that in the coming year, “we’ll see a lot more retailers re-owning that experience. So they’ll start developing experiences directly within the Gen AI models. For example, ChatGPT has announced partnerships with major retailers like Walmart and Target.” This plays out in two ways: first, consumers will have access to different retailers apps through AI platforms. “Now they have a lot more control of what that experience looks like,” Garcia explained. 

Second, retailers will keep integrating features popularized through the different Gen AI models into their own products. “Whether that’s an advanced form of search which recognizes the intent of the user in a more precise way or it’s a dedicated chatbot experience or some type of visual search, that is going to yield better results for the user,” Garcia said.

To better understand AI shopping from a user experience perspective, Applause conducted an unmoderated study where participants received tasks that they could complete at home on their own time, recording their screens, faces and voices. “We encouraged them to narrate what they were doing as they completed the tasks, and we also asked various follow-up questions to dig deeper into what they’re thinking and feeling,” Balen said. The study consisted of two parts: 

1. 1. Understanding participants’ mental models of AI-assisted shopping
   2. Learning what people think about AI shopping agents through engaging with them.

“For the first part, we wanted to understand the participant’s mental model of AI-assisted shopping because that would inevitably influence the way that they interacted with an AI shopping agent. We asked what comes to mind when they hear AI-assisted shopping and asked about their experiences when they used AI to shop,” Balen said.

In the initial part of the study, Balen said the team found that most participants had used AI to find the best deals because price is a major deciding factor. Many had also used image search such as Google Lens or Pinterest Lens, and it makes sense to them. “Those are analogous to doing a search with words, which they have done many, many times before. So that’s a familiar and easy way to get into using AI,” Balen said. 

“It was also very common to use AI to aggregate and analyze reviews, which is a huge time saver,” Balen said. While participants didn’t want to take time to review multiple sources, analyzing reviews from multiple sites was seen to hedge any bias that may stem from paid or hand-selected reviews that may appear on the product manufacturer’s website.

Though none of the study participants had used an AI agent to shop for and purchase an item, Balen said, most were curious about it. The technology is still emerging and not widely available, which may have created some hesitance, Balen said. “The concept is still mostly theoretical, and they’re not sure if they’re quite ready to give up that agency… Some mentioned that they might not be fully comfortable with an AI platform being the middleman between them and purchasing a product.”

For the second part of the study, Applause UX researchers asked participants to use a particular AI tool to shop for a gift. They were instructed to ask the AI tool for recommendations on what to buy for a friend or a loved one, then provide feedback on that experience. 

“Among the ten participants, there was a wide variety of experiences with the AI shopping agent,” Balen said. “However, it’s still gift shopping, so their basic journeys are similar.” Balen walked through the typical process: Shoppers started off with an initial prompt, which varied greatly depending on the participant’s mental model of AI shopping agents. “Most of them included details like the gift recipient’s age and how they were related to the participant. After that, there was a series of back and forth, evaluating the recommendations and writing more prompts to refine them.”

Once the participants selected an item from the options the AI provided, they moved on to comparing different versions or brands. Balen said that participants expected the agent to “scour the internet far and wide, in all the nooks and crannies, to find the best price for that item.” Once they found the best price, shoppers wanted to be able to click a link to go to a site where they could purchase the item or be able to save the link for later. 

“While those wants or expectations were similar among all of the participants, what each step along the journey looked like varied greatly between individuals, in part due to their mental model of an AI-assisted shopping agent,” Balen said. 

Two main mental models for AI-assisted shopping emerged. 
1. AI shopping as an advanced version of search: These shoppers were typically less familiar with AI tools. Their interactions were characterized by:

2. AI shopping as a conversation with a robot assistant: More familiar with AI tools, these shoppers use it like an advanced assistant that can engage in extended chats. Their positive experiences using AI in a wide variety of functions led to interactions with:

Participants who viewed the AI as an advanced search engine used simple prompts like “top toys for girls ages 10-11” or “gift idea for a 60 year-old man that loves sports.” Those with more experience using AI, however, provided details such as the intended recipient’s occupation, interests, and family structure, then invited the AI to ask further questions for clarification. 

“Participants were really happy that they received recommendations for products or brands that they would have never thought of,” Balen said, especially when shopping for teenagers or friends that have a niche hobby. They also appreciated that the AI agent explained why it made each recommendation, because that meant it acknowledged and reiterated the language the participant used in the prompt. This helped participants feel as though the experience was being personalized to them.

In some cases, Balen said, the AI agent offered a selection of follow-up prompts or suggestions for how to further refine the recommendations. Every participant who received these suggestions appreciated them, regardless of their previous level of experience with AI. Those suggestions didn’t always appear however, and Balen couldn’t determine what signalled the AI to provide them or not. 

Balen said participants also appreciated when the agent collected and displayed various points of comparison between similar items, as well as the option to click into a particular item to see a product detail page with almost everything you would need to know about the item, including a summary of reviews where and where to purchase it.

Most participants wanted a greater variety of recommendations: the initial results included too few options and didn’t provide an obvious way for novice AI users to ask for more. While experienced users know how to request more options, they didn’t want to have to ask.

The AI provided some inappropriate recommendations, such as a ski tuning service for someone who was about to start his own ski tuning business. In other instances, inconsistent results stemmed from lack of conversational memory. For example, the agent repeated suggestions that the participant indicated he didn’t like, or it forgot a previous criteria after a new one was introduced. 

Balen described the experience for a father initially looking for closet organizers for his daughter, who then asked for suggestions for his son. “He said, ‘I’m done looking at that stuff’ and pivoted to ask for recommendations for his son who likes Xbox. And then the agent recommended more closet organizers. He wasn’t angry, but he was pretty flabbergasted.” 

Even when everything went well and AI helped a participant find an ideal gift, most users couldn’t find a way to save the results or bookmark their search to reference it in the future. The interface on the AI agent was confusing as well. Balen said, “This particular AI agent had one tab for chat and then another tab for shopping and then another tab for more detailed shopping results. Participants were put off by that, because sometimes they would start in the shopping experience, and then they would be taken to the chat without any notification. So it was confusing, and they expected the entire experience to be in one unified place.”

Balen summarized, “This all goes to show that UX research and testing is essential throughout the development of AI tools, not only to ensure that the design is intuitive and user friendly, but more importantly, to ensure that the use case makes sense, that AI is being implemented in ways that customers or users will actually find useful, and to ensure that the model is behaving as intended by the developers.”

Garcia outlined some ways that organizations developing AI tools can build in functionality that will help all users get great results – not just those that understand how to construct rich, detailed prompts. “It’s a lot to expect users to really be able to come up with advanced prompting techniques,” he said. 

For organizations working on apps based on existing frontier models, Garcia recommended the following ways to improve the AI-assisted shopping experience: 

“All of those are things that you have to do before even conducting regular UI testing and functional testing,” Garcia explained. In addition, he emphasized that it’s important to make sure that apps are developed using the right training datasets. “Having the right text prompts, having the right audio, videos, images if you’re doing a type of visual search, all of those are really important. So having diverse, high-quality human sourced datasets is critical for that part.” 

Going back to model evaluation, Garcia reiterated the importance of having a diverse set of users providing different prompts and follow-ups and grading the responses that the model gives or the experience it is providing. He added that security is also extremely important. “Conducting red teaming or adversarial testing to make sure there’s no toxicity, bias, inaccuracy, or other type of harms, it’s also critical.”

Finally, Garcia pointed out that testing needs to be an ongoing effort as models – and user expectations – adapt and evolve. “At the end of the day, testing the end-to-end experience in production environments on a regular basis is really important.”