Why Innovation Requires an Adaptive QA Approach

The original computer programs ran in batch processing — set up the data, press a button and check the output. Next, we had the interactive terminal, then the text screen. Windows brought the event-driven paradigm, then databases on the server, then the web. Today, mobile is the predominant platform for end users.

Each innovation in computing brings ingenuity and value to users — and headaches to risk managers. At each juncture, testing must adapt to meet digital quality needs.

Let’s talk about industry maturity and what that means for QA professionals.

Maturity creates challenges

Michael Porter, a Harvard business professor, proposed that industries go through change. The personal computer went through a hobby phase, then intense growth, standardization and, eventually, decline. The same tactics that made Apple successful in the 1970s didn't work in the 1980s. IBM beat Apple, but the same standards that made it successful would be its downfall, as Dell was able to make more PC clones faster, cheaper and with lower profit margins. Once the smartphone was able to do 90% of what a computer could, the personal computer entered the decline stage, where the way to win is on price.

Modern-day software has a similar model. Whenever a new technology emerges, it can generally do a handful of interesting things. QA professionals test the software by exploring it.

Eventually this results in more complex applications, which leads to a marketplace for testing and testing tools. When enough people want something, someone will build it, so we get tools designed to help with testing that technology.

It's not just tools. New technology introduces new failure modes — ways that the software can break. For example, with the web browser, the server could go down or a third-party API could fail. Browser compatibility became a problem during the growth phase, as browsers interpreted JavaScript differently. One company I worked with used transparent .jpg image files at a time when Microsoft's browsers could not support transparency. Every time we got a new icon, we'd have to check it and file a ticket. Eventually we just put a rejection for those .jpg images in the version control software.

As tools mature, we develop a series of quick attacks and begin to develop tools for automation. Over time, as developers learn to code around the common failure modes of the platform, the same test strategies become less effective — at least we hope so. Boris Beizer called this the Pesticide Paradox — over time the techniques to find bugs become less effective.

Pushing maturity back

Every now and then, something new arrives that pushes the industry maturity back. For example, responsive design websites required testers to develop new ways to test screen resolution. Today, testing smart TVs presents a host of challenges because, while the devices are popular and profitable they are not standardized. Entire toolsets and approaches are emerging to test them, and they look very different than what we had before.

That seems to be the way of things. Consider the evolution of mobile devices. Mobile devices appear, become wildly popular, and create a need to test thousands of different device/OS/screen size combinations. Then, consolidation and standards emerge. Eventually, as custom forks of the Android OS end, we may drop the need for individual device coverage. Compiler makers will start to make software that is compatible with both Android and Apple phones, which would remove the need for separate development teams. Instead of finding bugs by simply breathing on the app and uncovering compatibility issues, testers would actually need to start testing.

Ironically, the environments where a tester can add the most value are likely the ones where innovations are in the early stages of introduction and growth. In maturity and decline, testing is "figured out," and lower-skill testers can follow the predefined steps to find enough problems to make the software good enough, but defects can still evade these techniques.

The place for innovation

The challenge continues with emerging devices and technologies. Virtual reality headsets, smart TVs and automotive dashboards all enable programmers to do new things. New versions of software running on them will cause issues like memory problems, overheating and excess battery drain. Mapping the software to features, and getting the possible combinations covered will lead to innovations on how to track testing, and the depth of test effort. We can study the defects that escape. When they are introduced, we can come up with techniques to find the important bugs quickly.

Or we can innovate on the maturity side — coming up with stable, predictable, repeatable processes that are measured and continually improve. This process can feel a bit like squeezing fruit for juice — the more we squeeze, the less value there is to wring. Yet it is valuable work and worth doing.

Cutting edge or legacy, tooling or human exploration, there is always a way to add value in software. The trick is to accurately assess where you are, where you are going, and how to get there.