How an Online Ticket Marketplace Got Started With Crowdtesting

One of the most common reasons companies turn to crowdtesting is a need to increase or maintain test coverage without bringing on additional full-time employees. When one event ticketing company went through a reorganization, the VP of Engineering realized that working with a crowdtesting partner would give him greater device coverage, more output, and greater flexibility than hiring additional QA staff.

Testing the hypothesis with a manual functional testing pilot

The ticketing company signed on for a crowdtesting pilot with Applause to assess its effectiveness. The initial scope of the pilot entitled the organization to a specific number of test case execution hours as well as test engineering hours over a two-month period. The test engineering hours were a crucial component that delivered test plan support, weekly communication with the ticketing company’s scrum teams, and vetting and prioritizing issues the testers discovered.

Part of the pilot process is determining what’s helpful for the development team, without putting more (or unnecessary) work back on them. Regular check-ins helped the Applause test engineers get a sense of what types of issues the development team cared about most. Without that vetting process, some teams feel overwhelmed by the volume of bugs Applause’s testers discover.

Though the pilot was initially focused on certain test cases that the engineering team provided, once testing started, it became clear that there was less documentation for test cases than both parties expected and more value in exploratory testing. The teams worked together to refine the focus once the Applause testers began executing.

Measuring success

A critical part of the pilot process is the mid-point check-in. While pilots start with a certain scope of work, that isn’t always a fixed solution over the one or two month pilot period. To get the most value out of the engagement, it’s crucial that whoever owns the business relationship on the customer side is strongly aligned with their counterpart on the Applause side. Thanks to tight communication with the ticketing company’s business leaders at the mid-point check in, we were able to further adjust to meet their long-term goals.

The mid-pilot success criteria conversation revealed that as a consumer-facing technology business, the ticketing company wanted to be able to build and release all of their products every single week, at the same time, without issue. While they could handle the build process, they had difficulty matching what they needed to do from a testing standpoint to the speed of development.

As staff from the ticketing company and Applause got to know each other better and strengthened communications, they recognized that the initial scope of the pilot only partially aligned with the development team’s top priorities. When the ticket marketplace’s team shared that they ultimately wanted to achieve high-quality, simultaneous weekly releases, the pilot was restructured, dedicating one testing team to the Android application and another to the iOS. That shift ensured testing could be done in parallel every week from Friday to Sunday, with the build ready by Monday for release.

A few of the most valuable bugs discovered during the pilot included:

• Customers unable to log in or create an account.

• No events displayed when a customer changed locations.

• The inability for customers to enter credit card details after canceling PayPal checkout.

• Filters not working properly.

Key takeaways

Clearly state the goals and how you’re measuring success.
Some of the most important conversations during the pilot set-up phase focus on these questions:

• Is this testing valuable? Is the pilot focused on an issue that directly impacts the business? What kind of dollar value is linked to those defects?

• How will the teams work together? What is the plan for overall communication? Will you integrate via Slack, Jira, or another bug tracking system during the pilot? Are the right people involved in the conversations?

• How are both parties measuring success? Every single week there should be discussion of KPIs for long-term, short-term and mid-term goals for the pilot to allow an opening to shift resources mid-pilot.

Think beyond the original scope of work: what will serve the organization best?
The pilot process isn’t just about completing the scope of work that was agreed upon at the beginning of the project — it’s an opportunity for conversations about your organization's overall testing needs and where crowdtesting may be able to provide value.

As a company’s digital teams (whether it’s QA, product and/or engineering) start working with Applause’s testing services and solution delivery teams, it’s important that both sides enter the pilot with a flexible mindset. At the midpoint check in (if not before) it’s vital to assess the value of the testing performed to date, the effectiveness of communications and integrations between teams, and if there are opportunities to improve. Applause is focused on delivering the most value we can during the pilot, and to help establish the foundation for a long-term partnership.

In our work with the ticketing marketplace, both teams entered the pilot with an initial set of expectations on how we’d work together that differed from the final outcome. Based on feedback, we adjusted to provide services that better aligned with what they really needed, without increasing costs. The result: Applause did more than uncover bugs that impacted customers’ ability to complete transactions — we helped the company reach the goal of releasing both iOS and Android native applications weekly, with better quality, without adding full-time QA staff.