Why Automated Accessibility Testing Tools Miss So Much

Like so many things in life, we humans often wish we could push a button and a machine would do exactly as we ask. Of course, more and more, that’s becoming reality, but in some areas, there’s a long way to go.

Currently, as businesses around the world ready themselves for the launch of the long-anticipated European Accessibility Act (EAA), firms focus on meeting the requirements of EAA and harmonizing national laws in EU States to the Act. Working to the fast-approaching date requires broad efficiencies in testing for accessibility. Automated accessibility testing tools are often part of the web accessibility testing, and certainly have their place, but there are limits to what these automated tools can do, and it’s key to understand where the handoff to human testers should occur.

Find out more about our accessibility solutions.

Machine discoverable vs. non-machine discoverable accessibility bugs

When I have conversations about the can and can’t dos of automated accessibility testing, I like to start by establishing a baseline: there are certain issues that can be found by a machine and there are certain issues that cannot. Often, when we consider the word “automation,” we tend to think that something is fully hands-off. Not the case with automated accessibility testing software. When I raise this point, it usually leads to a discussion around the percentages of what automated accessibility testing can find vs. what requires human intervention. The main point I try to make is that we must accept that a majority of accessibility issues cannot be found with automated accessibility testing tools, as illustrated below.

At Applause, we have a robust program and we test a wide variety of products at any time, so we have a strong dataset to inform assessments. We are very familiar with automated accessibility testing tools, as we routinely use them to help us find accessibility issues at the initial stage of assessment. We intimately know the software’s benefits and limitations, and as a result, have found:

Benefits of automated accessibility testing tools:

• Can potentially find between 20% and 40% of accessibility issues
• Can cover approximately 25% of WCAG Level A success criteria, and 17% of Level AA
• Can be used in combination with manual testing to provide full coverage
• Saves time if used in concert with manual testing

Weaknesses of automated accessibility tools:

• Incomplete tests that miss a large number of key blocking issues
• Create a large number of false reports that require an expert to sift through for relevant issues
• Are mostly based on axe-core ruleset with enhancements

Humans excel at determining what makes sense for humans

Imagine a blog containing an image of three students studying in a library. The blog topic is about hard work being the key to college success. The author has entered the alt text for the image as “Students.” An automated accessibility testing tool will find no issue with that alt text, though it adds little value. However, a human tester – understanding the context of the image within a blog about hard work and college success – would quickly see that the description in the alt text field would be better written as something like “Three students studying hard in a library reviewing test results with high grades.” This is just one simple example of the context of content. Automated tools are made to scan code and find gaps, but they don’t yet understand human contextual cues, and would miss this issue easily found by a human tester.

Example of what automated testing tools miss

In this short video, an Applause accessibility expert examines a webpage form to illustrate how common automated accessibility tools miss issues.

Code issues typically missed with automated accessibility testing

Furthering the points illustrated by the video above, I list some of the issues Applause accessibility experts and testers routinely find that automated accessibility testing tools miss:

• Incorrect screen-reader-only text present (such as a note to a designer etc.)
• Visible text incorrectly hidden from assistive technology
• Inappropriate image alt text
• Uninformative or incorrect accessible names
• Controls that are removed from the tab order
• Illogical tab order and reading order
• Keyboard focus traps
• Focus outlines missing and/or not being visible enough
• Visible heading not set as programmatic heading

Visual issues typically missed with automated accessibility testing

• Zoom not fully supported
• Text spacing causing content to overlap
• Text over images not meeting the required color contrast ratio
• Unexpected changes of context on focus or input
• Essential graphical elements not meeting the required color contrast ratio

Content issues typically missed with automated accessibility testing

• Inaccurate captions
• Missing audio descriptions
• Instructions referencing shape, color, size
• Relying on color alone for communication
• Automatic timeouts
• Animated content that cannot be stopped
• Uninformative/inaccurate page titles
• Visible labels missing
• Uninformative/inaccurate labels or headings
• Error text missing or uninformative
• Authentication requiring a cognitive test
• Content relying on gestures or motion
• Content available to auto-populate through a multi-step process

Is my website accessible?

While there are tools that can shed some light on this common question, they are just a starting point. Knowing how testing tools help, along with their limitations, is critical when it comes to any software testing, but particularly when it comes to automated accessibility testing.

Applause has more than a decade’s experience helping global firms along the entire accessibility and inclusive design journey: delivering inclusive experiences, adhering to global accessibility guidelines, embracing digital accessibility at all levels of the organization, optimizing and managing resources, gaining the end-user perspective or avoiding litigation/navigating remediation.

We’re here to help. Find out more about our accessibility solutions.