Steps to Maturing Test Automation within a Deployment Pipeline

Drew HornDrew Horn
minute read

Speed kills, but that holds true in a variety of ways when it comes to test automation. It is true that automation can significantly increase the speed at which you are testing and help you expand your code coverage.

It is also true that by trying to automate too much (or the wrong things) too fast, you can set yourself back a long way.

With that said, automation is a tool that all companies should prioritize when operating in an agile development environment. How you go about setting up your automation practice will determine just how many of the benefits you can realize.

Remember, you have to crawl before you can walk and walk before you can sprint. To help move you along, I have outlined some examples that detail the steps you should take along the SDLC to help your deployment pipeline evolve from beginner to intermediate to expert.

Beginner Stage

In this beginner stage, it is all about creating quick wins for your team. You will want your development team to begin by checking in their tests and immediately provide pass-fail feedback. Once unit tests are passing, you can get started with a very small set of automated smoke tests.

functional testing: development team, unit tests, automated test, manual test, additional test/release (top axis left to right) the process is: check-in, smoke test, regression, deploy (left to right)

Initially it may be easier to run these tests at night, but you will eventually want these running as part of the build process so they are automatically triggered upon a successful build. Once they are passing with consistency, you can incrementally add more smoke tests to your automation suite.

Adding one small test at a time will allow you to gain confidence quickly in your automation, as it establishes the building blocks for your automation practice. Not only does this build confidence, but it saves time in the long-run from having go back and fix a larger set of tests.

Intermediate Stage

Now that you have built some confidence in your initial set of smoke sets, you can begin to attack additional layers of the application. For example, if you have an application that allows for you to test both integrated and UI tests, this would be a great opportunity to fan out your tests and run them in parallel with each other. The more you can cover of the functional testing pyramid, the more confidence you can have going into your regression tests.

functional testing: development team, unit tests, automated test, manual test, additional test/release (top axis left to right) the process is: check-in, smoke test, regression, deploy (left to right)

Your regression tests are where you can really start running multiple layers in parallel to decrease the time to get results and feedback on the quality of the application. Like your automated smoke tests, your automated regression tests will grow bit by bit until you’ve reached a comfortable level of test coverage and can begin optimizing.

To complement your automated regression tests, your manual team should prioritize any manual tests as well as continuous manual exploratory tests. The greater percentage of regression tests you can automate, the more time you can spend on exploratory testing, increasing the coverage of the edge cases where interesting bugs usually hide.

The most important competency of this intermediate SDLC stage is that you establish a test case management system and reporting structure. It’s key to make sure that all of your results are going to a single place to be seen from a single view. By doing that, you have a consistent view of any failures and can comfortably decide whether or not to deploy.

Expert Stage

The biggest difference between the intermediate and expert stages is the implementation of quality gates. This aligns with your need for a test case management system, as quality gates will allow for you to reject any builds before deploying or moving on to further testing.

functional testing: development team, unit tests, functional teams, acceptance (top axis left to right) the process is: check-in, smoke test, regression, deploy (left to right)

Again, your goal here is to provide rapid, early feedback to your development team so they can perform timely root cause analysis. In doing so, you ensure critical bugs do not make it further along into development, eliminating more costly and timely fixes down the road.

The key here is to do just enough testing at each phase before you expand and fan out your testing. There is a lot that needs to be covered, so find a comfortable point before adding in new test environments.

Lastly, you want to build confidence at each quality gate before running more time-consuming, costly automation. Oftentimes when you run those time-consuming tests, it can end up being a bottleneck and slowing you down. Doing so further down the road gives you more confidence once you’ve gotten there.

Wrapping Up

Taking your deployment pipeline from beginner to expert does not happen overnight. It requires a commitment to the process and patience for moving ahead incrementally. That commitment includes an investment in technology, an established set of processes, and an ability to track and report on your progress.

Though this all may seem like a daunting task (and it is), many of the struggles companies face with automation can be mitigated with crowdtesting resources. With the resources you need, your deployment pipeline will look like an expert’s in no time.

Automation Manual Testing

Automation and Manual Testing: Striking the Right Balance


Learn how to balance automation and manual testing to accelerate your agile development team.

Read Now

You might also be interested in: