Tel Aviv, Israel
How Lean Startups Test Software
Did you know that the average consumer loses over $300 a year from unwanted charges? Wouldn’t it be great if there were a service that alerted you to hidden fees, billing errors, scams and other similar charges? Thanks to the founders of BillGuard – a free service powered by the collective knowledge of millions of people – consumers now have such an option.
Launched in 2011 at TechCrunch Disrupt, co-founders Yaron Samid and Raphael Ouzan attributed much of their early success to the Lean Startup methodology – a process designed to get companies up and running with minimal costs and delays.
As part of this strategy, Raphael (who is also the company’s CTO) knew this would have to include their software testing activities. With that in mind, he began searching for a cost-effective testing solution that could keep up with their rapid pace of innovation.
"Not many companies ship as fast or as often as we do," explained Raphael. "uTest proved to be very valuable to our business, since it didn’t take long to get started, and once we were up and running, we could test our code on extremely short notice."
This case study will show how uTest enabled BillGuard to run continuous functional testing of its unique web application as part of its Lean Startup launch schedule.
BillGuard: Testing Goals and Philosophy
With several iterations each week, Raphael and his development team needed a more flexible method of testing their product within a highly agile framework. A bigger challenge, however, was that testing had to be completed with real credit and debit cards. An impossible task for some, but not for uTest and its global community of testers.
"Every bank connection is different, so it was critical that we could test with live data. This why our use of uTest was so amazing and unique, because the testers had to use their own credit cards to test the system."
-- Raphael Ouzan, Founder and CTO, BillGuard
Two weeks after their first code had been deployed in mid-2011, Raphael and his team were ready to begin their functional test cycle. Starting off with two test cycles a month, their team was able to achieve comprehensive regression testing against new features – a process that created what Raphael called an "early feedback loop."
"Getting feedback at such an early stage was an enormous benefit for us," said Raphael. "Establishing a feedback loop like this the number one reason why I recommend uTest to others. As a startup, you don’t want to spend time on things that you can’t measure. You want to make sure the core features of your application are working before moving on to the next sprint, and uTest gives you that assurance."
Test Cycle Results
Shortly after the conclusion of their first two test cycles – where they had received bug reports, completed test cases and user feedback in real-time –Raphael and his team decided to scale their efforts and switch to a continuous testing model.
"We want to get to a point where we are receiving feedback on a daily basis," said Raphael. "Our platform is huge and it’s important that we continue to iterate very quickly. Using uTest, we can be sure that our latest versions are always fully tested. That’s the biggest reason why we’ll run continuous testing. We understand the value."
Summary and Next Steps
Throughout the testing process – from defining their testing goals to approving bugs – Raphael and his team would benefit from the guidance of their dedicated uTest project manager.
"We really developed a close relationship with our project manager," said Raphael. "We had a lot of conversations where we would discuss ways to make things more efficient. Beyond that, she helped us filter out the noise, identify the showstopper bugs and made sure that each project stayed on point. She essentially became part of our team."
With consumers already eager to try their product – and with VC investors putting up $3 million in financing – the future looks bright for the BillGuard team. As the company expands into new markets, they expect uTest to remain their partner in testing.
"uTest is great for testing, but their real value for us was that we could embed QA into every single line of code," said Raphael. "uTest is a truly unique solution. We’re very happy to have discovered them."