How a Canary Release Strategy Improves Quality

When it’s time to deploy application code, you want to be certain — as much as you can be — that you’ve eliminated severe defects. In lieu of total certainty, many organizations protect themselves with different software deployment strategies.

Just as development strategies vary, software deployment strategies differ as well. For that reason, there is a wide variety of ways to deploy code to production servers.

Some popular software deployment strategies include:

  • standard deployment

  • blue-green deployment

  • canary release

  • A/B testing

Let’s discuss the standard, blue-green and canary release strategies, including what they are, how they are implemented, and what gains in quality businesses get by using them.

Standard deployment

Standard software deployment is a movement of a code package from a developer server to a shared development server. After the code package gets tested, the organization then releases it to the staging server. QA teams then re-test the code in the staging server. Once the QA tests pass, the organization moves the code to the production server, where it is released to all customers. The only tests that occur in the production server, in this method, are performance or unit tests that check that the server system is operational.

Many software development organizations, new and old, use the standard deployment method. Unfortunately, with this software deployment strategy, you don’t know how the code impacts the production server until it’s already released to customers. While thorough testing occurs in the development and staging servers, they can only simulate the production environment — not provide an exact copy. The data, database connections, APIs and other connected services likely do not exist in the same way, if at all, on the development or staging servers. So, in the standard software deployment strategy, organizations release all of that untested code to customers in one big push.

If you want to test code in production, also known as shift-right testing, consider using the standard method in combination with a canary release strategy, or switching entirely to canary or a blue-green deployment, depending on your business needs.

Blue-green deployment

Blue-green deployment implements the use of two production servers that are exactly alike. One server is a staging server, where the new code release is deployed and tested. Once testing is complete and the release is approved, that server becomes the live or true production server. The second server now becomes the staging environment, and is then updated with the next release candidate build.

Basically, with blue-green deployment, you have two production servers that you switch back and forth. Blue-green is a simple implementation model, if you can handle the cost and maintenance of two identical production servers. One risk with blue-green deployment is that the servers can get out of sync, leading to unexpected code and configuration setup. When the two servers are no longer exact copies, code might pass testing on one server, while defects or errors affect the second server. Another risk is simple human error. What if users are mistakenly connected to the staging server instead of the updated production server? While that is likely an easy fix, it is definitely an error that negatively affects users.

The primary benefit of using blue-green deployment is it enables you to release a full code package or a subset of code and test it out repeatedly as needed before releasing it to all users. You control which users access the production server.

A/B testing

Similar to blue-green deployment in theory, A/B testing basically takes a production server and deploys different versions of the service’s code, or back-end processing code, and routes it to specific users within the production server. The organization then routes user traffic to each server based on the version of the code installed. Both versions are tested, and the version deemed to have the best quality is fully deployed to all users.

Although it’s an inexpensive way to test in production, it’s highly complex. The organization risks taking the production server down, or significantly impacting performance while experimenting with different code versions.

Canary release

With the canary release strategy, the organization deploys software changes incrementally to select external or internal users. For example, half of the software updates might go to a group of customers who have agreed to receive early code releases and test for defects. Similarly, the organization could release a portion of the new code to a testing team, which tests the specific code released on production. Once the initial code drop passes testing, it then releases additional sets of code until the full release is deployed.

The canary release strategy is a software deployment option in which you release continually as each release passes testing. Using a canary release enables the organization to avoid sudden critical failures on the full release. If defects are detected, canary testing makes failure analysis faster and simpler, because the problem lies in the portion of the code that was last released — no need to troubleshoot the full release.

Advantages of a canary release strategy

A canary release strategy reduces software development cost by only using a single production server instance. A canary release saves time and expense by having users and testers validate the functionality during each incremental drop of new release code. It’s faster to deploy via canary release, and avoids issues with different servers not matching the production server. You also only have one production server to manage.

With a canary release strategy, you test in production, but when a problem is found, the feature can be immediately switched back until the defect is fixed. It’s like a light switch — you turn it on to test, and, if all goes well, it stays on. A canary release does add a level of accountability for releasing code. During releases the deployment staff must be available to switch code when needed, then troubleshoot and fix any defects the customer or the testing group reports.

The canary release strategy allows the software development team to better manage the effect of a release on the code base. The canary release strategy enables businesses to test in production with real users, on real data. There is only a single production server involved, which makes the cost lower than with blue-green deployment. It’s quicker to switch off features when defects are found in the release set; the entire customer base is not affected, making it safer than the A/B release method. Plus, the code stays together rather than separating out the back-end services as in the A/B method.

Implementing a canary release strategy

To execute a canary release strategy, first verify:

  • developers are able to manage the deployment

  • customers are willing to receive untested code

  • testers are flexible and report defects quickly.

The QA team must be on point for rapid testing, so any defects are reported, fixed and re-tested in a timely manner.

Consider a canary release to an external test team only, instead of a group of customers. Address any defects reported, re-test and then plan a customer release. This way, the code receives more testing in production prior to a wide customer release, and the organization can control what code is released for easier troubleshooting.

Plan to spend time setting up the required infrastructure and resources to implement a successful canary release strategy. Expect to spend time adding code to toggle your code sections as needed. When the code is toggled off, you’ll need to confirm it is actually off. You will also need a method of controlling who sees the new code by user, rather than by server. Additionally, decide who will cover the release, or which developer is able to turn off and on the code switches as needed.

Let Applause be your beta testers

No matter the industry, you want to deliver high-quality experiences to your customers. In other words, don’t force your customers to be your beta testers.

Applause is the market leader in testing services. The Applause global community of digital experts is ready to provide real-world testing, on real devices, anywhere and anytime you need it. Applause provides rapid feedback ahead of new software releases, via In-Sprint Testing that allows you to manually test features almost immediately after development. Or, if the product is ready for production, you can launch to a vetted group of Applause testers that meet your demographic and geographic requirements. Applause enables enterprises to identify points of friction and severe defects that would otherwise affect the bottom line and drive customers to competitors.

Contact Applause today to learn how we provide world-class functional, usability, voice and IoT testing for businesses all around the world.

Want to see more like this?
Amy Reichert
Freelance QA SME/Test Engineer
Reading time: 8 min

What Holiday Shoppers Want in 2022

Applause global retail survey provides insights into changing expectations and preferences for holiday shoppers in 2022

Mastering the Art of Omnichannel Experiences

A guide to developing exceptional omnichannel experiences

Why Merging DesignOps and DevOps Benefits the Whole Org

To follow through on shared business goals, align designers and developers end-to-end

Applause Launches Ready, Test, Go. Podcast

New digital quality podcast features expert guests talking software engineering and quality

Build Empathy. Build Better Products. Build Customer Loyalty.

Learn how Webex focused on developing empathy in their inclusive design efforts as they began to improve their world-renowned platform

What is the European Accessibility Act?

Read this article to learn what the European Accessibility Act means for your business and how you can prepare.