Why Now is the Time for Test Suite Maintenance

It is often embarrassing for a QA manager to admit when it is time for test suite maintenance.

We’ve all been there. During the lifetime of an application we’ve written test cases for version 1.0 features, creating our first regression test suite. But, as the application matures, more features are added, GUI layouts get changed, and new technologies are incorporated to build the latest and greatest application update: version 2.0. Later, testers write test cases for new features as part of project delivery, which they test thoroughly and eventually move into the regression suite as we plan for the exciting version 3.0.

Here is where questions around test suite maintenance come into play:

• What about those version 1.0 test cases? Are those older test cases still valid?

• While building the latest version, did we make any changes to the application that affect the steps or expected outcome of those test cases as written? Probably.

• Did we update those test cases to reflect the new design? Probably not.

• Will testers know that we changed that feature and not report those failed tests as bugs? Sure, if testers are a constant in the life cycle of the application, they might. But chances are that some testers won’t be around the next time the application must be regressed.

Any number of factors can put you in the position to hand the test suite over to a new group of testers that have never seen your application, including personnel changes, stakeholder strategy changes and corporate mergers. In these instances, an out-of-date test suite will cost time and money in the form of bug reports, which will ultimately end up being filed as Working as Designed or Deprecated during bug triage.

We get it. The shiny new features are exciting, and more pressing. There are deadlines to meet, and there is very little time to write test cases for the new version, let alone update the existing test suite. “We’ll have time to update after we finish this sprint,” some managers tell the team. However, the next sprint is already in the planning stage, with even more exciting new features. That time never materializes.

During my time with Applause, I’ve helped many companies rebuild, update and maintain test suites. I’ve seen it all, from outdated feature tests to test cases with only a title and a link to a ticket in Jira — which I didn’t have access to! I’ve seen regression suites contain multiple examples of the exact same test case, and I don’t how many times I’ve run into the unhelpful “TBD” in the steps. All of these test inefficiencies waste valuable testing time by causing a tester to come back for clarification; or, because the tester followed the steps to the letter, they logged a bug that gets ignored later on. Any of these scenarios should trigger test case maintenance, but the bug is often just closed with every intention to update the test case later. Later never arrives.

Test case maintenance is an area where Applause can make an impact on suites in a short amount of time. Our teams approach test cases with fresh eyes and can quickly identify all these issues, documenting the outdated and identifying missing or inaccurate information. As we learn the current state of an application, Applause works with you on test suite maintenance — and, in most cases, we can even make it more efficient.

One client of mine handed my team a test suite housed in TestRail that contained about 1,600 test cases. By the time we finished our review, we reduced the overall test case count to roughly 600 test cases, which ended up covering more functionality than the original 1,600 count. Applause achieved this by identifying and removing deprecated cases, updating the test steps of stale but still valid cases, and combining test cases. When it comes to test case combination, the only difference was the platform on which the test case was executed. From there, we applied a labeling structure to each, which enabled faster test plan creation.

Your time is valuable. The longer you put off test suite maintenance, the more time you will lose to the slow bleed caused by inaccurate test cases. Let Applause help you get your time back.