The Pros and Cons of a Bug Bounty Program

Severe defects that reach customers have a very real effect on the bottom line. If a product doesn’t function as intended or in the way the customer expects, it’s almost a certainty that they will discontinue use, unsubscribe or otherwise bid adieu to your brand — potentially sharing their negative experiences far and wide as well.

This is why we have software testing, to catch as many defects as possible. One way to augment software testing is with a bug bounty program in which individuals, typically people outside the company who have testing experience, hunt down potentially costly defects in exchange for cash. Bug bounty programs are a widespread practice today; many companies rely on the perspectives that these bug bounty hunters bring to products.

While bug bounties are an effective way to catch some defects, the ROI might not always match expectations. Let’s learn more about what a bug bounty program is, pros and cons, and how it contrasts with a crowdtesting model.

Why bug bounties

A bug bounty program can provide a cost-effective means of finding defects the internal organization would otherwise miss. A bug bounty offers a sanity check of sorts for a digital product, enabling testers with an outside perspective to find vulnerabilities and defects for areas within a defined scope.

Money is a huge motivator for testers that participate in bug bounty programs, but they can also provide notoriety within the community. Some high-profile bug bounties have leaderboards and provide a tester with an opportunity to showcase their skills to potential employers or new clients.

Bug bounty programs are especially useful from a security perspective. Internal security teams can barely keep up with all the software vulnerabilities that pop up on a daily basis. Put simply, a hacker will nearly always be one step ahead of enterprise security in recognizing exploits. Bug bounty programs give organizations a chance to solicit help from white-hat hackers, many of whom make a living putting their domain expertise to work in reporting vulnerabilities.

Pros of bug bounties

For years, bug bounty programs have been a trusted source of information for organizations all around the world. These programs offer numerous advantages, including:

• filling blind spots

• continuous evaluation

• preserving resources

Filling blind spots.

No matter how thoroughly the organization documents software requirements and specifications or how well the product is designed and coded, there will always be defects. Whether minor nuisances or major concerns, bugs will find a way to infiltrate a product.

One way bugs creep in is through institutional blindness. While internal testers offer a fresh perspective compared to developers, they too have patterns of behavior and preconceived notions of a product. This can lead them to only test certain paths or areas, especially if the organization under-invests in automation or reduces the available time for testing.

Bug bounty testers are not tainted by institutional preferences — even if they might still have some individual tendencies of their own. Furthermore, they are motivated to find defects, unafraid of rocking the proverbial boat in their assessment of a product.

Continuous evaluation.

As long as a product is in a testable state, the organization can run a bug bounty to support it. Whether the product is already in use by customers, is ready to test as a minimum viable product, or even if just a prototype, a bug bounty program can reveal key vulnerabilities throughout the product’s lifetime.

Bug bounties are evergreen and flexible. Unlike internal testing teams, there will always be diverse, credentialed testers available to find defects in a product. It’s a dependable strategy that yields helpful results, which means organizations can factor it into their product planning and documentation. The flexibility of bug bounties also enables organizations to run them continuously, which is how many organizations run bug bounties.

Preserving resources.

Hackers or testers in a bug bounty program find the weak spots or defects that elude internal teams. Most businesses cannot scale up internal resources on a whim — and, even if they could, that might not help reduce defects and vulnerabilities, as institutional blindness and truncated release schedules will still limit how productive QA teams can be. Bug bounty programs enable organizations to save money by spending it more efficiently on outsourced help.

If the business or the organization is in a financial pinch, it is easier to scale down resources dedicated to bug bounties, either by reducing payouts or limiting them to high-value defects — or scrapping bug bounty programs altogether. This is yet another example of the flexibility of bug bounty programs.

Cons of bug bounties

While there are advantages to bug bounty programs, there are challenges too. Keep in mind these bug bounty cons before you design your program:

• tester’s perspective

• money diverts focus

• control and communication

Tester’s perspective.

You can find credentialed, skilled professionals to participate in bug bounty programs, especially if you’re willing to pay for top-tier talent. But just as internal teams can be limited by their own blind spots and perspective, so can bug bounty hunters. Ideally, a diverse group of testers helps cover a wide spectrum of potential defects and vulnerabilities, but there’s no guarantee they will think to test products in the same manner as an actual customer will experience them, let alone on their devices of choice and in high-value locations.

Furthermore, testing to find a defect and exploratory testing a product can be quite different. Bug bounty hunters care about finding high-value problems, where an internal tester might focus on a specific product or feature, and might probe it just to see what happens. These are different perspectives; and both can vary from the customer’s perspective, which is to simply use a product in the manner they need to.

Money diverts focus.

If you have the chance to earn $100 or $500, which one would you choose? There’s no shame in it — you’d choose the higher dollar amount, and so do bounty hunters.

High-severity defects typically result in higher payouts. On the surface, this seems like a win-win. However, if a large percentage of bug bounty hunters are only searching for critical vulnerabilities, they will miss smaller-value defects. These issues can add up, negatively affecting the user experience.

Organizations can get creative with payout structures to combat this issue. For example, it could award a bonus to the bounty hunter that finds the highest quantity of approved defects. But be aware that stating dollar values and bonuses for issues will always have an effect on the bounty hunter, even if it’s an unconscious one.

Control and communication.

When defining the parameters for a bug bounty program, it’s important to be as clear as possible with the scope and qualifications. It’s easy for gray areas to intrude. For example, what if the organization can’t sufficiently reproduce the reported bug or vulnerability? A bounty hunter would likely expect payment, but the organization could disagree.

Bug bounty hunters aren’t doing their work one cubicle over. And while many of us have grown accustomed to remote work and collaboration, there is room for miscommunication or different interpretations. Clear, effective communication is key, from the beginning of the project all the way through to disputes with bug hunters.

Additionally, most bug bounty programs take place on products in production — unlike crowdtesting, which can run throughout the SDLC. Thus, the organization relinquishes some level of control with the product when it grants special access or permission for testers. In these instances. Non-disclosure agreements help protect intellectual property, but what if a security tester actually has a black hat agenda? Many organizations don’t have measures to properly vet these kinds of testers. Managing bug bounty hunters creates additional overhead that makes these programs difficult to maintain and secure.

Conversely, the tester is operating in good faith that the company will pay according to their posted bounties. Trust is a two-way street, and both parties need to honor their agreements.

Bug bounty program vs. crowdtesting

Many people see bug bounties and crowdtesting as one and the same. Both involve digital experts probing an app for defects or vulnerabilities. While both approaches can be helpful ways to improve digital quality, even in tandem, there are some notable differences between bug bounties and crowdtesting.

Scope. While you can set parameters for a bug bounty program, the scope is ultimately limited by what defects the organization will accept. If a bug hunter finds a defect outside the payment scope, they might not report it, and might not get paid for it even if they do. Yet, on a micro level, bug bounties can also miss the mark, as some customer flows require a very specific set of actions to occur, such as a new account creation.

Crowdtesting allows for both a broader and narrower scope, depending on the testing need. Organizations can get very specific with crowdtesting instructions, including UX considerations such as whether an app feels clunky or intuitive. Broader exploratory testing assignments are also an option, where testers are paid for bugs according to severity level, not necessarily a specific type of exploit.

Volume. Both crowdtesting and bug bounties rely on skilled professionals who know how to root out defects. Crowdtesting, however, allows for as many or as few people as you want, with as much technical skill as needed. So, rather than collecting a moderate amount of defects from a highly technical audience — with a limit to how many bug bounty hunters you can find and pay — crowdtesting enables any amount of defects from as diverse and vast a pool of testers as you choose.

For this reason, sorting through crowdtesting defects can make for some challenges. More bug reports — not to mention on more devices, OSes and in more locations — means more triaging, and it’s easy for internal staff to become overwhelmed if they don’t properly designate severity levels or the scope. A crowdtesting partner like Applause is vital in managing communication, sourcing more bug documentation when needed and ensuring an appropriate volume of testing coverage.

Flexibility. Digital quality is a multi-faceted effort. Not only do you need functional and secure applications to ensure high quality — both crowdtesting and bug bounties can be useful here — but sometimes you need usability research or participants, high-quality training data, accessibility assessments, or testing done on specific devices. Bug bounties fall short in these areas, but crowdtesting excels.

What if you want to maintain a consistent team of testing experts? Or, the opposite, what if you want to have a brand-new team for each testing program? What if you want to test as soon as features are developed? What if you want to scale up the number of testers beyond your normal requirements? It’s difficult to gain this flexibility with a bug bounty program, but crowdtesting enables this kind of adaptive approach, especially when working with a partner who sources and manages the global pool of testers on your behalf.

Through the power of a skilled, specialized pool of crowdsourced digital experts, an organization can augment nearly any internal testing initiative, helping to improve aspects ranging from quality and usability, to accessibility and device coverage.

Targeted. Most bug bounty programs do not target a specific demographic. Often, bug bounties are application- or region-specific, which makes for a wide open pool of testers, but not necessarily testers who are reflective of a target customer.

Crowdtesting enables an organization to dial in on a specific type of person. For example, to support a new product launch in APAC, an organization could specify testers who are located in the relevant region, don’t have an account with the company, own a specific device on which to conduct testing and speak a specific language or two. Crowdtesting isn’t just about technical ability — though that quality certainly helps with uncovering defects — it’s about specifying the group of people needed for the task at hand.

Here is where Applause shines. Applause supplies digital experts all around the world as your partner in digital quality greatness. We work with customers to understand their goals, pain points and wishes, and turn those into an action plan. Within hours, our crowdtesters can get to work, even mid-sprint, to help identify problem areas in your digital products.

If you want to harness the power of real people using real devices, carriers and OSes in real locations all around the world, let us know how we can help you achieve your digital quality goals.