Anatomy of an Accessibility Pilot
Investment in accessibility is key to compliance and product usability. Even organizations devoted to addressing conformance gaps and whose sites are inclusive to varying degrees can face testing and remediation challenges. This may be due to a variety of factors. Many of these challenges, however, can be addressed by working with a knowledgeable accessibility testing partner.
In this blog, I share a pilot that Applause did with a small tech startup that was acquired by a very large global tech company. The acquiring company has deep accessibility and inclusive design experience, something that the newly acquired company lacked. Applause undertook a two-week pilot to help the newly acquired company — which we’ll call “Newco” in this blog — identify compliance gaps and high-priority bugs that contributed to these gaps along with key product usability issues.
Background
Newco’s product involves visualizations of large amounts of data depicted in creative and compelling ways, but not in accessible ways. In addition, Newco has very limited engineering resources to fix any accessibility issues found. Given the limited development hours available, we opted to focus on usability testing to surface which issues are of the highest concern to users, at our customer’s direction. While we knew that we weren’t going to make Newco fully compliant to the letter of WCAG by the end of the pilot, we knew we could make a meaningful impact for users.
Applause structured the pilot to:
- Conduct small accessibility audits to identify high-value accessibility gaps
- Guide complex fixes through expert review & consultation
- Prioritize the roadmap for new features needed to support accessibility through user interview and research
Three stages of the accessibility pilot
Audit Sample
As we initiated the pilot, and while keeping a focus on usability throughout the engagement, we started with an accessibility audit across a limited set of scoped web pages, working to review against WCAG A and AA success criteria. The goal was to:
- determine the types and frequency of issues occurring on pages
- identify the most critical a11y defects
- create a list of high-value accessibility issues that the dev team could address immediately.
In this initial phase, the goal was to stop bleeding. We worked to get ahead of things and stop Newco from pushing out issues. We started to build a relationship with the engineering team and collect small samples of issues we found.
This is reminiscent of in-sprint testing where the team has just created a new feature and we test the feature while it’s fresh in the developer’s mind. Though we potentially could find thousands of accessibility issues in an initial audit, it doesn’t serve our client. This large group of issues can psychologically overwhelm teams and, of course, create practical concern based on limited capacity to address the issues, even if it was wise to address them immediately. The key here is that the majority of these issues are likely not high priority. We try to settle in between the perspective that all WCAG issues are important and the fact that the primary goal is to make sure that users can do their jobs efficiently, even if there are some bugs.
As counterintuitive as this may seem, with Newco, we mentioned that there were issues we were not going to file because we didn’t want them working on them at this point. Of course, we are working closely with the client and making decisions with their full awareness and agreement after hearing what we have found and what we suggest. It may seem strange to suggest that an organization put a quantity of issues aside for later because that’s all but saying that the company isn’t compliant, but trying to fix all the issues right away just doesn’t work. It’s simply not realistic. Once accessibility work is initiated, it’s often wise to fill out a Voluntary Product Accessibility Template (VPAT). Once this template is completed, it is called an Accessibility Conformance Report (ACR). The ACR can be presented in different ways. Done correctly, the ACR gives the reader an idea of how accessible your product/service is to assistive technology users. This is particularly helpful when a product is not yet compliant, but shows “good faith” and that you are working on it.
Key issues and prioritization found in audit sample
| Classification | Issue Types | Action |
|---|---|---|
| Easy to Fix P1 and P2 decoupled issues on atomic components | 7 Issues logged | Move to top of backlog for quick fixes |
| Moderate Fix P1 and P2 issues that require page context or occur in complex components | 10 issues logged | Plan for fixes, review issues with design/development team for agreement |
| Harder to fix Components that will require additional engineering | 2 issues logged + visualizations | Schedule workshop/training to develop approach to solutions |
| Low Impact P3 and P4 issues | Not logged at this time | Backlog issues as available based on user impact prioritization on key flows |
Expert review
Next we did a user-focused review of the key scenarios by an accessibility expert. The goal here was to:
- establish a rough performance baseline for key workflows using assistive technologies (AT)
- provide analysis of complex controls and visualizations and begin proposing solutions
- determine an application’s suitability for further usability testing
Newco did not have capacity set aside for their development team to work on accessibility. Considering this, we knew that any fixes we asked them to do would need to have the highest impact and ROI based on their guidance on priorities. At this point, we also wanted to make sure that additional work didn’t pile up while they were fixing issues we found. So, as they continued working on features improvement and daily work, we wanted to ensure that we didn’t create technical debt while finding accessibility issues.
This is where our expert review is key. Instead of conducting a full audit, we had our accessibility experts go through and examine the interface and come up with high-level findings. For example, our expert might find some fundamental issues with the design of the toolbar that appears on a page. We know that it’s going to require some iteration with the design team to remediate it, so our expert works to set up a roadmap for the next few months to fix this issue and any other high-ROI issues.
Example of expert review results
The goal here was to discover if an AT user could use the application fully.
One of the characteristics of Newco’s app is that the filter field uses a nested combobox. The review revealed many individual accessibility issues with this component, but the best course of action was to review the component hierarchy and validate that keyboard and screen reader users were able to properly flow between the filter hierarchy. We found that all users would benefit from a component redesign.
User interview
In our last phase of the pilot, we had Newco do a one-on-one interview with a blind AT user in an IT administration role. The goal was to:
- gain insights around preferred screen reader workflows
- inform prioritization and investment in different visualization types
- ideate on emerging usage of AI to solve difficult visualization problems for screen reader users.
The interview helps to validate that we’re fixing the right things.
A common trap for companies that embark on accessibility journeys is to get hung up on not being fully compliant. Often, the focus becomes How do we fix everything to ensure we’re compliant? It’s easy to fall into this camp.
A smarter approach is to consider that among the potential thousands of issues on your site, there is a much smaller subset of key issues – the ones that really matter. This is the way to move forward. As I mentioned above, the key is to start. Find and fix the low-hanging fruit, then fill out a VPAT that shares where you are and shows that your intentions are true when it comes to making the website or product accessible.
Yes, technically you need to fix all of the accessibility issues in your product at some point to avoid legal risk, but this process is a journey. We can help determine what is truly impactful for users, based on their direction, and ensure the roadmap is focused on empowering users first, and preventing technical debt as part of the roadmap. We make these determinations through the research we do, either by running a full usability study or through interviews.
Newco had some unique product visualizations that didn’t have well-known accessibility alternatives. For example, a typical alternative for a pie chart is a table. That’s because pie charts are typically generated from tables. But if you have a novel graphic, say traffic on a network or similar, there isn’t really a standard accessible alternative for that. This is where exploratory interviews help. AT users can determine if this feature is necessary or just a nice-to-have. If it’s a needed basic feature, then this goes into the backlog. If it’s considered non essential, then it’s deprioritized. The power of pulling in actual users of AT here is key to back up the decisions we make.
User interview insights
To help understand the types of feedback we received, here are two insights our blind user found:
Data should be available as tables and for exporting: Our user relies on and prefers table views for most data presentations.
- With up to approx 20 data points, he can form a mental model
- Filtering and sorting options expand the number of data points he can work with
- He typically exports data for his own analysis into Excel
- Bar chart and some maps can be easily accessed with tabular alternatives
Larger data sets pose challenges for UX: Our user said that time series and spatial data pose challenges. He emphasized focusing on the purpose of visualizations.
- Statistics and summaries are most useful (and what sighted people are looking for as well)
- He cautioned against over-engineering solutions that don’t provide value in many circumstances (like sonification)
- Raw data, filtering, and user control, and alerts remain key
He called out an opportunity for AI to fill a gap here: he regularly sends screenshots to AI tools because they can provide a helpful summary of the visualization. He can then ask targeted follow up questions.
Automated testing in accessibility
A pure automated scan solution for accessibility issues can be very disruptive to your actual goals. An automated scan can yield thousands of issues. This can really create challenges in terms of what to work on, since, as I’ve mentioned, the vast majority of those issues have no user impact. At the same time, automation misses many problems that impact real users. By some estimates automated tools only find between 20% and 40% of accessibility issues, and cover approximately 25% of WCAG Level A success criteria, and 17% of Level AA.
This dual problem of not finding the issues that impact users most — while missing those, generating a flood of low-priority issues — can really grind organizations to a halt. Automated tools are helpful when used thoughtfully and in the context of a larger program, but manual testing with real AT users is the best way to sort out what issues impact the user experience.
Putting all the accessibility pieces together
Of course,our customer’s accessibility compliance is a worthy goal, but it’s key to remember that the first order of business is to ensure that users can do their jobs efficiently, even if there are some bugs. After the major blocking issues are resolved, organizations can hone in on lesser issues and work their way through the longer list of backlogged issues.
For the pilot with Newco, Applause performed baseline audits and worked with them to address the high-priority issues that would yield the most ROI. In addition, we recommended office hours and training suggestions, codesign sessions and many other best practices that were tailored to Newco’s specific circumstance and needs.
11 min