Israel & New York
Web & Mobile
Questioning Outsourced Software Testing
What’s the easiest and most cost-effective way to test a social trivia game, to be released as a website, iPhone app and Facebook app, in less than three weeks?
For the answer to that question, like all others, you should reference Answers.com. With branches in Israel and New York, the publicly-traded, 90-person company has provided curious web users with answers to millions of questions since 2005. The leading answer engine provides extensive information by combining over 250 trusted reference resources, a video library covering thousands of topics, and its user-generated Q&A contributed by its millions-strong global community.
In 2010 – as three versions of the company’s "blufr" app moved to production – Answers.com’s Senior Product Manager Lea Aharonowich decided that Applause could provide the most efficient method for testing under real-world conditions.
The "blufr" app is described as a “funky, fast-paced trivia game” that challenges you to decide whether certain statements are true or false. The objective for users is to gain points, awards and bragging rights. The objective for testers was to find critical bugs prior to launch.
This case study will examine how QA professionals from around the world helped Answers.com test their application across three major platforms: the Web, iPhone and Facebook.
Test Cycle #1: www.blufr.com
To start, Answers.com began by running a standard, functional test cycle of the "blufr" web application. The objective here was to verify the product’s core features and functionality against various operating systems (Windows, Mac), browsers (IE, Chrome, Safari) and other categories. With the help of their Applause project manager, Answers.com would create and upload detailed test cases to reflect these criteria.
From there, Applause assembled a 10-person team comprised of their top-rated testers, who would perform these tasks as part the functional and exploratory test cycle. Aharonowich and her team would spend the next week monitoring submitted bugs via the Applause online platform. By the time the release was closed, Aharonowich had familiarized herself with the Applause process – and began to realize the full benefit of crowdsourced testing.
"This was a side project developed with an outsourcing firm, and so we wanted to test it with an outsourcing firm as well," said Aharonowich. "That way it wouldn't strain our in-house teams."
Test Cycle #2: blufr iPhone App
With their web-based testing complete, Answers.com would quickly transition to the world of mobile, as they prepared to launch the blufr iPhone application – a project that would require location-based testing. As such, Answers.com selected testers from the US, UK, Belgium, China, France, Israel, Italy and the Netherlands.
Due to the relatively small number of participants, Aharonowich said that a few defects were able to slip through the cracks. But overall, her team was able to gain invaluable insight into their app, as well as the mobile testing process in general.
"As I learned more about the process and technology, it became easier to focus on more specific parts of the applications," said Aharonowich. "I also learned the importance of inviting some of the same testers back for another release. Optimizing our time was really the key."
Test Cycle #3: blufr Facebook App
For the third and final test cycle, Answers.com would turn their attention to the "blufr" Facebook app.
Over an 11-day span, the Applause community would complete a series of detailed test scripts meant to simulate the likely actions of Facebook users. This included installation, sign up, game play and other common tasks. Additionally, these actions would also be performed against a number of browsers (IE, Firefox, Chrome, etc.) and operating systems (Mac, PC and Linux). As a result, her testing team discovered a wide variety of critical bugs, consisting of functional, technical and usability issues.
Overall, more than 85% of the bugs submitted by the testing team would be approved by Aharonowich, who said she expects to make Applause a big part of their testing activities going forward.
"I can easily see us outsourcing this type of testing to uTest (now known as Applause) again in the future," said Aharonowich.
"We started out using uTest (Applause) as a sanity check, but we eventually discovered so many other use cases and user behavior, which made it an easy decision to expand our efforts."
-- Lea Aharonowich, Senior Product Manager, Answers.com