A Brief Anatomy of an Applause User Experience Study

There are many types and modes of user experience (UX) research. In a previous blog – Exploring the Types of UX Research Methods – I took a broader look at the variety of UX research methods.

In this blog, I get a bit more focused and look at some examples of the main components and interactions in a typical engagement. For Applause, these tend to be the remote, unmoderated qualitative UX studies, though we do a wide variety of study types depending on our clients’ ever-changing spectrum of needs. Whether a customer needs comprehensive support, from UX strategy and research to insights and actionable solutions, or has extensive in-house expertise and only requires testers in niche markets with specific devices and OSes, we are here to be your trusted partner.

Kicking off the study

It’s all about reaching the overall goals of our clients.

Depending on the engagement, we can get into the thick of it, often helping our clients sort out goals. This may be valuable for organizations that are new to user experience testing. For example, in some accounts, we’ll ask clients what burning questions they must understand. This can be a very interesting starting point, as it can lead to heated discussions, even arguments, over what is most lacking in terms of understanding users. We spend quite a bit of time scoping our studies to identify the best methodology, how many studies are required, how many participants and more.

We also work with a much different type of customer in certain larger organizations. They often have very strong UX researchers with extensive experiences. Their teams may approach us having a fairly developed plan of what they’d like to do, but since they don’t know all the nuances of how we’ll do the study, there are some open switches. We will review their plan and bounce ideas off of each other, and then agree on details of how to construct their specific studies.

In the end, regardless of the client or needs, the kickoff phase of studies arrives at the same place, it’s just that we’re filling different needs for our varied customers in obvious and nuanced ways.

For example, one of our large global social media customers knew what it wanted to do, but it lacked the ability to find the right testers in smaller markets such as Thailand and Kenya. They needed a very specific type of small business owner that regularly uses one of its apps on certain devices with set operating systems.

Another larger client, a global e-commerce provider, needed us to find tax accountants that use artificial intelligence (AI) as study participants. Obviously, this is very niche, but Applause has a lot of experience in finding very specific types of participants and testers.

Flow and type of our remote, unmoderated study

In the remote unmoderated study, participants complete tasks and answer questions on their own time, devices and in their own environment. And, as the name indicates, no one is proctoring this experience. Typically, though this can vary based on customization needs of clients, we have our participants go through a task-based survey, which can take up to an hour.

From the Applause side, a solution delivery manager (SDM) works closely with the account. The SDM gathers the overall requirements and then turns internally to put all the pieces together within Applause. Most of the time, we select a lead UX researcher to work with the client on strategy, involving the SDM in the process. The researcher listens to fully understand the client’s overall goals, goes through any designs they want to test, or establishes any questions the client would like answered within the study. There is a lot of Q&A in the initial meetings, often around the client’s interest in understanding what patterns and trends we’re seeing in the market. The client often asks what other clients are doing in their industry.

Moving from planning to execution, results and insights

After this initial meeting, the researcher creates a study plan. The plan contains the objectives and the reasoning behind why the client is doing the study in the first place, what is covered in the study, overall goals as well as the participant demographic from which Applause will recruit. In this phase, we establish deliverables to the client and the estimated delivery timeline. After official approval by the client, our Research Operations team takes the established participant demographics and creates a screener – a specific survey we send out to our community – to source participants that match the needed requirements. Before we move forward with the participants, we always allow the client to weigh in on who we have found. Typically our researchers help the operations team select participants. Often, we use 15 participants – 5 per condition needed. So, for example, perhaps 5 on Android mobile, 5 on iOS and 5 on the web. It can be many more depending on the scenario and markets required.

While recruiting is happening, our researchers develop the questions for the study. Once established, these typically open-ended questions may take participants approximately an hour to complete. Once questions are ready, the client reviews them and we edit accordingly and launch the study. During the study, participants will often record themselves going through the tasks or discussing their feelings about the experience. Generally, there are written responses around their overall experience at the core.

Once we get the responses back, our lead researcher analyzes the data and writes a report, generally in slide form. We present the findings on a read out call where we give the client actionable recommendations. We provide next steps, which might be a recommendation for a more targeted study or a little more market research. The main objective here is to establish whether there is an immediate need for more UX work now to move ahead, or if the client is good to go for a given period of time, at which point, it may be a good idea to circle back for some further exploration.

Current trends

Markets are complex, customers change and we see an unsurprising trend flowing from all of this: One study often leads to more studies. It’s as if some truth is discovered but then there’s a whole new set of questions to be answered that weren’t in the general mindset before. Often, we do a qualitative study and clients tell us that they love the information, as it’s either new to them, validates what they already know, or falls somewhere in the middle. Now, however, they want to understand how widespread the issues are, and so they may request a quantitative study.

We’ll also get requests for iterative qualitative studies. The client understands that we did the study in the prototype phase, for example, and wants to go back now that the app is live and test it again. These types of study cycles are common.

Recently, in working with a global gaming provider, we were testing in three distinct markets. We found a major issue. Testers attempted to purchase something that requires an alternative form of payment, but when attempting to add the payment option, it would bounce the user back to the start of the purchasing process. The client team had no idea about this severe issue. While discovering this issue, our participants found other functional defects as well. Because this issue was found in two of the three markets in which we were testing, the client wondered what other markets this might be happening in. This led us to recommend several more in-market studies in search of this bug. In this study, we were testing on the mobile app, but it wasn’t known whether the issue would occur on another device as well.

In a separate engagement with the global gaming company previously mentioned we had teams of study participants in nine different markets over a year’s time. As part of the study, the users had to purchase a subscription on a console. Our teams found a very high severity payment issue that impacted the Argentinian market. Argentina has unique legal requirements regarding online purchases. Users would make a purchase and that transaction was shown in US dollars rather than in the local currency, forcing users to find a way to understand what the exchange rate might be. To make matters worse, there was a tax added because users were purchasing from a company outside of the country. All of this led to a major markup that users had no visibility into when starting the process.

Conclusion

Though we typically do remote UX studies, we can do a wide spectrum of studies.

One of the things I love about my work at Applause is that we’re constantly adjusting to the needs of our clients as dictated by the needs of the market. If there are methodologies our clients want to try, our team has the knowledge and skills to use them.

We can consult with clients beyond the study, or if they want to throw designs at us after to evaluate, we’re happy to do that. After studies, we’ll often meet to discuss strategies for the future. We can do heuristic evaluations, recruit participants, do analysis or augment any aspect of your team when you need it. In addition, we regularly work with persons with disabilities for studies that want to dive deeper into the experiences of users leveraging assistive technologies. And we have significant AI experience that can be infused into UX studies in several ways.

No matter what market or what vertical you’re in, we have the experience and global reach to help you plan for and execute UX studies small and large.

For more information, visit our User Experience Testing page.