Automated Accessibility Testing: Updated for 2024

What is accessibility testing?

Accessibility testing evaluates whether mobile and web apps are usable by people with disabilities. These can include visual, hearing, mobile, and cognitive impairments.

In the QA world, it's pretty common for testers to focus on the common accessibility issues like visual or hearing disabilities while overlooking a variety of other disabilities that also make using the web harder.

Some examples of overlooked disabilities include:

• People with arthritis or other mobility issues who have difficulty moving a mouse and prefer keystrokes to navigate through tabs.
• Users with photosensitivity who may get seizures from flickering or flashing components.
• Those with cognitive disabilities such as dysgraphia or dyslexia who have difficulty processing information in busy or inconsistent designs.

What does accessibility testing actually test for?

What accessibility testing looks for can be divided into several subcategories:

• Visual accessibility testing. It checks for adjustable text size, customizable color schemes, proper contrast ratios, and compatibility with screen readers.
• Auditory accessibility testing. This checks for accurate closed captions, transcripts for audio content, and visual alternatives for audio cues.
• Motor accessibility testing. Motor tests sure full keyboard navigation, easily clickable buttons and links, and support for alternative input devices.
• Cognitive accessibility testing. This test looks for clear, simple language, a consistent layout, and options to disable animations or auto-play content.
• Speech accessibility testing. Speech tests verify voice-controlled interfaces, alternative input methods, adjustable speech recognition sensitivity, and text-based command options.

While this list isn't exhaustive, the specific tests you automate and run will depend on your accessibility goals. But, before getting into the accessibility standards to think through when running these tests, knowing why it's a requirement for businesses is key. 

In 2024, 95.9% of home pages still failed the Web Content Accessibility Guidelines (WGAC). It becomes a missed opportunity to improve the customer experience, increase customer satisfaction and loyalty, boost their reputation, and improve their search engine rankings.

Why does accessibility testing matter? 

If guaranteeing equitable access to the web for everyone isn’t reason enough to build and run automated accessibility tests, there are others: Attracting and retaining a larger user base, improving SEO, and avoiding legal trouble. 

On top of that, a few quick statistics highlight the upside for companies that prioritize accessibility testing:

• Companies that prioritize accessibility outperform their competitors in shareholder returns by 4x (Accenture)
• Accessible websites rank higher in search engine results pages (ADA)
• WCAG 2.0 compliant products have a 50% higher market performance than their competitors. (Gartner)
• 2,387 website accessibility lawsuits were filed in 2022 (Accessibility.com).
• Products that champion accessibility see 28% higher revenue and 30% higher margin (Accenture)

Digital accessibility is globally required by law

The business benefits of accessibility testing are one thing. But businesses also need to consider the risk of not complying with accessibility legislation. 

In the United States, Section 508 of the Rehabilitation Act requires web and mobile products to provide accessible interfaces to people with disabilities. The Americans with Disabilities Act (ADA), which prohibits discrimination against individuals with disabilities, has been interpreted by courts to apply to websites and digital content. It's led to a rise in lawsuits against businesses that don't have a fully accessible online presence.

Similar legislation exists in Canada, the European Union, the UK, Australia, and Brazil.

The basics of accessibility standards 

The Web Content Accessibility Guidelines (WCAG), which is a W3C standard, is considered the gold standard for accessibility guidelines. The WCAG success criteria are testable statements that confirm a given website content is accessible to a wide range of people with disabilities (PWD). 

Many country’s regulatory bodies including the US, EU, UK, and Canada — use WCAG standards to measure compliance with the law.

WCAG is built on four fundamental principles known as POUR:

• Perceivable. The site must be presentable to users in ways they can perceive. For example, users need to be able to change the presentation to suit their needs, such as large print, braille, speech, translations, or more straightforward language.
• Operable. The site must not require interaction that a user cannot perform. For example, a site cannot require mouse-only inputs. It also needs keyboard shortcuts for people who can’t move a mouse and must provide people with enough time to complete actions. It cannot cause seizures. It must be navigable so that PWDs understand where they are and know how to get to where they are going.
• Understandable. Users must be able to comprehend any information and understand how to use the interface. For example, labels and instructions should provide a clear context for the associated page element (e.g., sign-up or log in buttons). 
• Robust. As in able to weather changes in technologies and user agents through the strictest interpretation of web standards.

Each principle outlines individual success criteria. As of WCAG 2.1 there are 78 success criteria (WCAG 2.2 will have 86). WCAG conformance is scored on a rating scale with each rating requiring compliance with additional success criteria. 

• Level A – Minimum compliance with web accessibility features (30 criteria in 2.1)
• Level AA – Addresses the biggest barriers for most disabled users (50 criteria in 2.1)
• Level AAA – The most complex level of web accessibility (All 78 criteria in 2.1)

It’s important to note that conformance is not the same as certification. To be certified you must be audited by an accredited body.

How to start accessibility testing 

Thankfully, you don’t have to pore over all the different legislation across constituencies to know if your website is compliant. WCAG provides a comprehensive framework for making web content more accessible to people with disabilities.

However, before you start to monitor your conformance with WCAG standards it’s helpful to define a general strategy.

1: Decide on a realistic target 

Choosing a WCAG target doesn’t require an extraordinary amount of analysis into personas and user needs — in most cases, using WCAG as your standard will already encapsulate any testing you would discover on your own. The WCAG standard will also be fully auditable by a third-party should you decide to get certified.

Currently, a solely automated strategy will identify about 80% of the issues that would need WCAG certification. That's because determining how well content is presented and interacted with by people with different disabilities takes human judgment.

2: Choose a testing tool

Popular accessibility testing tools, like Lighthouse and Axe, use WCAG standards to evaluate web accessibility. For example, Lighthouse, a part of Chrome’s DevTools, performs automated accessibility audits and provides actionable insights to improve web content. Axe is an open-source tool that integrates into development workflows for both automated and manual accessibility checks.

At QA Wolf we use the Deque Axe library to power our accessibility testing, and we think you should, too. Axe is compatible with a bunch of different frameworks and has bindings for many of the major programming languages. And it’s open-source. 

3: Run your accessibility tests

You can run your accessibility checks as a static analyzer on the source code, or as end-to-end tests on a rendered page. We recommend the latter as it more accurately reflects how users with disabilities will perceive and interact with your product. Playwright has excellent documentation on implementing accessibility tests as part of your standard regression suite. 

If you are running your accessibility tests as an E2E regression test, we recommend integrating them with the rest of your test suite in the CI/CD pipeline. This will give your developers instant feedback if new code causes an accessibility issue where there wasn’t one before. 

4: Run regular accessibility audits

At WCAG Level AAA, the highest level of WCAG conformance, are success criteria that require human judgment and can’t be automated. For those criteria you’ll need to perform occasional manual accessibility audits. 

Our friends at Deque and AccessibilityWeb have manual auditing tools so you can find violations on your own without the need for third-party consultants. Even if your organization isn’t shooting for certification, manual audit tools can still be extremely valuable in identifying the last 20% of problems that automated tools can’t yet catch.

Automate your accessibility testing process 

Accessibility testing helps you identify and resolve issues that keep people with disabilities from fully engaging with your website and applications. Adhering to accessibility standards, like WCAG, mitigates the legal risks of noncompliance and broadens your potential user base, opening up new avenues for market expansion and customer satisfaction.

You can read more about how we deliver WCAG compliance through automated regression testing, with results that are logged and fully auditable or schedule a demo to get a guided tour.