Recognising and Avoiding the Automation Bias

In 2011, three Japanese tourists in Australia drove into the ocean. Even though something didn’t seem right, they kept driving until the car started floating away. Only at that point did they start to question whether they may have made a mistake. Their navigation system had told them to follow the road into the water, so they assumed it must be the right way.

I like to call this kind of thinking automation bias. Humans now trust machines more than their own judgement. The way I see it, this gradual change has taken place over three main stages.

The first is overreliance on technology. Last year on vacation, I rented a car for myself and the family. For the first time in a while, I had to try to reverse the car into a parking space without a reverse camera. Despite having spent 20 years of my life reversing a car without one, my most recent car came with parking sensors. In the space of a few years, I had forgotten how to reverse. The moment you get used to using a technology, the less confident you feel about doing the action without it.

There are so many examples of this. If you asked me what seven times fifty four is, I would use my phone’s calculator, not do the sum in my head. Skills like mental arithmetic that we learned as kids are unlikely to serve us anymore if we do not regularly put them into practice. 

This brings us to the second stage. As we have started to become more reliant on technology,  it has started to influence our behaviour. Take smartwatches as an example. Whereas they used to be simple gadgets that measure your steps, now they are marketed as medical devices that we listen to as though they have the expertise of a doctor. They tell you when to walk, how much water to drink, etc. 

The third stage in this transformation is addiction. Whereas at the start it was interesting to track your heart rate variability on your smartwatch, now it is fun. Everytime we hit a milestone, we get a hit of dopamine that makes us want to continue optimising our health. The feeling of achievement drives us to keep improving so we can experience the dopamine hit again and again. It’s the same with reverse cameras: reversing perfectly is now not only helpful, it gives us a buzz.

You might be asking what any of this has to do with automation. This presentation was developed for AutomationSTAR in Vienna. The point is that we see the same pattern of behaviour driving decision-making in automation engineers. We have stopped thinking for ourselves and are letting machines do all the work without questioning it. And I think it’s time we took a step back and looked at the impact this is having on digital quality.

The way I test now is completely different to how I tested ten years ago thanks to Agile, DevOps, automation and so on. This is not a bad thing. However, advancements in technology and processes also means we are losing critical testing skills.

The key lost skill is critical thinking. We don’t spend time thinking and strategising any more because we think technology is doing this for us. Before we came reliant on technology and automation, testers used to act like Sherlock Holmes. When we received a requirement, we would read it again and again to see where we could break it. We also took time to do a proper risk analysis, considering the risk to the business and customer. The same goes for root cause analysis — we didn’t just want to fix the bug when we found it, we worked to uncover why the bug occurred and how we can prevent similar issues in the future. 

Like other technology users, testers are also very driven by dopamine. We have become addicted to tracking vanity metrics that give us a sense of achievement, but actually tell us very little about the quality of the digital products we are producing. We have a general rule that nothing should go to production without 90% test coverage. Yet, in reality, you could have 100% test coverage and it wouldn’t necessarily mean the quality was good. Another vanity metric is the number of automated tests. Some teams are really proud of the fact that they have automated thousands of test cases. We also like to track the number of bugs we found. Again, neither of these metrics really tell you much about quality. 

Automation testing itself doesn’t really mean anything. It just means that you automate the test that you write.  Before DevOps, the business used to be really involved in testing, giving the go or no-go decision. With automation, they no longer have as much insight into what has been tested, or what is going right or wrong. During sprint demos, they just see the data on the automation dashboard. If everything is lit up green, they give the go ahead to ship the product. We have come up with this unwritten rule that a green screen means quality is high.

As a result, I like to think of DevOps today as more like DevOops. If you look at the app store reviews for any app, I bet you will see the same pattern I always notice. Customers will complain that the app was working brilliantly… until all of a sudden it became full of bugs. This happens because new releases are rolled out that have not been sufficiently tested. In the spirit of releasing quickly, seemingly small updates are overlooked. We see the automated tests go green and we push the product.

Gen AI is going to lead testers to lose their critical thinking skills even more. With Gen AI, we can fully automate test cases, generate risk analyses, and generate and execute code  — the tasks that used to still require human input. If Gen AI can now help us with these more strategic tasks, we have even less reason to use these skills. Which begs the question: five years from now, what skills will testers have?

The point is that losing skills itself is not necessarily a bad thing. In fact, losing skills goes hand-in-hand with evolution. Billions of years ago, early mammals used to be amphibious, or later swing from trees — characteristics we no longer have. So the real question is which skills do we actually want to lose to Gen AI. One criticism of Gen AI so far is that humans don’t want machines to create art, they want machines to do our chores so we have more time for art.

In the context of automation, critical thinking is art. We need to retain the skills that make us unique as testers, like exploratory testing, accessibility, usability and localisation. Gen AI should be left to do the repeatable chores that don’t require human intelligence, such as regression, error handling, API testing, data-driven testing and cross-browser validation. Gen AI is a valuable tool but we must be intentional about what we use it for. Otherwise, we end up just exacerbating the automation bias.

As the testing industry progresses, I think we need to take a moment to recalibrate. We have got so caught up in trying to optimise releases for speed that we have forgotten what it really takes to ensure high-quality products. Instead of focusing on vanity metrics and having a green automation dashboard, we need to focus on customer-centric KPIs like reviews and CSAT scores. We also need to rethink our relationship with machines to make sure we are including human perspectives where they matter and only automating repeatable tasks. That’s how you reverse the automation bias to increase speed without sacrificing quality and avoid falling into the trap of DevOops.