Hire Software Developers 7
Back to blogs

What Automated Accessibility Testing Can't Catch

November 3, 2026
Signup form whose focused button is completely hidden under a cookie banner, a failure automated accessibility testing misses

What Automated Accessibility Testing Can't Catch

Deque makes the axe accessibility engine, and it publishes a coverage report for automated accessibility testing [A9]. One row is Focus Order, WCAG 2.4.3: auditors logged 9,553 issues, and automation found 0% of them [A10]. Focus Visible: 7,312 issues, 0%. Meaningful Sequence: 3,313 issues, 0% [A10].

That is how a SaaS signup flow passes every scan and still can't be finished from a keyboard. The clearest evidence that scans miss the failures that block users comes from the vendor with the most to gain from automation.

Automated accessibility testing is software, such as axe-core or Lighthouse, checking a page against WCAG rules with no person judging the result. Its makers say results still need manual review, and the W3C says tools "can only assist" in an evaluation [A13, A14, A15].

Not sure the European Accessibility Act covers you? Start with what the EAA actually applies to.

Disclosure: yousource publishes this and sells UX On Demand, which includes an accessibility audit.

Key takeaways

  • Deque, the axe vendor, reports that automated testing finds 57.38% of accessibility issues by volume. The figure is vendor-sourced [A9].
  • In Deque's own table, automation found 0% of Focus Order (2.4.3), Focus Visible (2.4.7) and Meaningful Sequence (1.3.2) issues [A10].
  • Colour contrast is about 30% of Deque's issue count. Without it, the automated share is roughly 46%, by our arithmetic on Deque's data [A11].
  • In the UK GDS audit of 2017, 42 of 143 barriers (29%) were missed by all 10 tools combined [A1, A2].
  • EN 301 549 V4.1.1, published 2 September 2026, references WCAG 2.2, but its main legal effect waits on citation in the EU Official Journal [A36]. (Not legal advice.)

How much can automated accessibility testing catch?

By issue volume, about 57%, according to Deque, a vendor [A9]. Colour contrast props that number up. For focus order, focus visible and meaningful sequence, Deque's own table shows 0% [A10, A11].

Deque set the 57% against the "widely-accepted belief that automated accessibility testing only provides 20-30 percent of coverage", arguing that issue volume is "a more meaningful and accurate metric" than share of success criteria [A12]. Fair, up to a point. Most accessibility bugs by count are mechanical, and a scanner should catch them.

So look at what fills the count. Colour contrast (1.4.3) is 88,714 of Deque's 294,958 issues, about 30%, and automation caught 83.11% of them [A11]. Take contrast out and the automated share drops to roughly 46%: (169,242 − 73,733) / (294,958 − 88,714). That's our arithmetic on Deque's table, not a number Deque publishes [A11].

The rest of the table is uneven. Keyboard (2.1.1): 234 of 9,412 issues, 2.49% [A10]. Non-text Content, the alt-text criterion, reaches 67.57% [A10], though a tool can only confirm that alt text exists, not that it "serves the equivalent purpose" [A21].

Volume coverage is not journey coverage. Forty contrast warnings and one keyboard trap add up to one problem that stops a user, and the scanner missed that one.

The best-known public test of accessibility testing tools is from 2017

The best-known public, independent comparison of accessibility testing tools is nine years old. In a study published in February 2017, the UK Government Digital Service built a deliberately broken page with 143 barriers and ran 10 tools against it, aXe and WAVE among them [A1]. Together they found 71% [A2]. "A total of 42 were missed by all of the tools we tested", or 29% [A2]. Counting errors and warnings only, the best single tool, Tenon, found 37% [A3].

The GDS results page moved on. Last updated on 13 April 2018, it covers a different setup, 142 barriers and 13 tools. Its table names SortSite as the best tool, which "found 40% of the problems we introduced", and Nu Html Checker as the worst, which "only picked up 13% of the barriers" [A5, A6]. It gives no missed-by-all total, so the 29% belongs to 2017.

A 2013 study of six tools by Vigo, Brown and Conway found that "at most, 50% of the success criteria are covered" [A18]. We found no neutral rerun on current tool versions, and the toolmakers don't dispute the gap. Lighthouse says a 100 "does not guarantee that the audited page is accessible" [A14]. In April 2025 the FTC ordered accessiBe to pay $1 million after, in the FTC's words, it claimed its plug-in, accessWidget, "can make any website compliant" with WCAG [A40].

One SaaS signup flow, five failures a scanner won't flag

Picture a typical B2B trial: cookie banner, signup form, login, plan picker, card details. Put the mouse in a drawer.

Focus under the cookie banner (2.4.11). The banner is pinned to the bottom of the viewport. You tab down the form and focus lands on "Create account", entirely hidden beneath it. WCAG 2.2 fails a focused component that is "entirely hidden due to author-created content", and the W3C gives a sticky cookie banner that entirely covers the focused component as an example that fails; partial overlap passes [A27]. A scan reads the page at one moment; it doesn't tab through and check what covers focus at each stop.

A trap in the card form (2.1.2). The payment widget takes focus and won't hand it back. Tab loops through the card fields forever. The rule is that focus "can be moved away from that component using only a keyboard interface" [A24]. You find this by pressing Tab until you can't.

Retyped characters at login (3.3.8). The login CAPTCHA wants six distorted characters typed back, and the password field blocks paste. WCAG 2.2 says "a cognitive function test (such as remembering a password or solving a puzzle) is not required for any step" of authentication, unless an exception applies. It defines that test as "A task that requires the user to remember, manipulate, or transcribe information" [A31]. Whether an exception applies is a call about your auth flow.

The silent toast (4.1.3). Billing details saved; a green "Saved" toast fades in and out. A screen-reader user hears nothing unless the message "can be programmatically determined through role or properties" and reaches them "without receiving focus" [A32]. To a scanner, the toast is a div. Knowing it was a status message takes someone who knows what the flow is meant to say.

Alt text that says nothing (1.1.1). The plan comparison is an image with alt="pricing". The attribute exists, so the check passes. GDS puts "whether alt text descriptions were accurate" among the things that "needed a human being to check" [A7, A21].

WCAG 2.2 testing and EN 301 549: published, not yet cited

EN 301 549 V4.1.1 came out on 2 September 2026 and references WCAG 2.2 Levels A and AA [A36]. Davis Wright Tremaine notes that V4.1.1's "most significant legal effect will not occur until it is cited in the Official Journal of the European Union" [A36]. We found no source confirming that citation yet. According to Legalithm, a compliance-software company, no version of EN 301 549 is cited under the EAA so far, and V3.2.1 is the version cited under the separate public-sector Web Accessibility Directive [A45]. V3.2.1 pegged compliance to WCAG 2.1 [A36]. This is not legal advice; check with counsel for your product and market.

This matters for WCAG 2.2 testing because 2.2's additions lean on judgement: 2.4.11 and 3.3.8 are in the walkthrough above. Target Size (2.5.8) sets a 24 by 24 CSS pixel minimum, then lists five exceptions [A28]. A tool can measure the box. Deciding whether "Inline" or "Essential" applies takes a person. Consistent Help (3.2.6) asks for help mechanisms, where they repeat, to sit in the same relative order across pages [A29], and a single-page scan can't see across pages.

Manual accessibility testing misses things too

An auditor doesn't fix this on their own. In a 2010 study by Brajnik, Yesilada and Harper, 22 experts evaluating pages against WCAG 2.0 produced 20% false positives and missed 32% of the real problems [A19]. The 27 non-experts did worse: 42% false positives and 49% false negatives [A19].

One expert's PDF, written once, is one opinion. Scripted passes, two reviewers on the journeys that matter and a re-run after changes turn "an auditor looked at it" into evidence.

How to split automated and manual accessibility testing

Give CI the mechanical checks on every pull request: contrast, missing labels, missing alt attributes, invalid ARIA. That's where Deque's own numbers are strong [A10, A11]. When axe-core marks a result "incomplete", it's telling you "manual review is needed" [A13], so file a ticket. Lighthouse ships its manual audits "as a checklist" [A14]. Somebody should own it.

Give people the journeys. Pick the three to five flows that earn or lose money (signup, login, checkout, billing) and script each one:

  1. Keyboard only. Can you see focus at every step? Does it move in a sensible order? Can you leave every widget, and is anything sitting on top of it? (2.4.7, 2.4.3, 2.1.2, 2.4.11) [A26, A25, A24, A27]
  2. Screen reader on. Are errors described in text? Are toasts and confirmations announced? Does every image and icon button say what it does? (3.3.1, 4.1.3, 1.1.1) [A30, A32, A21]
  3. Login. Can someone get in without solving a puzzle, retyping characters or recalling something from memory, and can a password manager paste the password? (3.3.8) [A31]

Two people run the script separately and compare. Re-run it whenever a release touches those flows. GDS gave the same advice in 2017: "combine automated tool testing with manual checking, an accessibility audit and user testing" [A4]. It's the split from our manual vs automated testing post: automation owns regression, people own discovery [A44].

That script is yours to run. Separately, if you want an outside audit of contrast, focus order, semantics and labels, yousource's UX On Demand has an Accessibility card, "Bring the app to AA": "Contrast, focus order, semantics, and labels — an audit with fixes designers and devs can action" [A42]. Single stream is $3,495/mo, with a 3-day turnaround per task [A42].

Before any of that, unplug your mouse and sign up for your own product. It takes ten minutes, and it will show you what your last scan report couldn't.

Frequently asked questions

Can automated accessibility testing make a site WCAG compliant?

No. The W3C says tools "can only assist" in determining accessibility [A15]. In 2025 the FTC ordered accessiBe to pay $1 million after it claimed its plug-in could make any website WCAG-compliant [A40].

What percentage of accessibility issues do automated tools find?

Deque, a vendor, reports 57.38% by issue volume [A9]. In the GDS audit of 2017, 10 tools combined found 71% of 143 barriers, and 29% were missed by all of them [A2].

Which WCAG criteria need manual accessibility testing?

Deque's data shows 0% automated detection for Focus Order (2.4.3), Focus Visible (2.4.7) and Meaningful Sequence (1.3.2) [A10]. Keyboard traps (2.1.2), focus hidden by sticky content (2.4.11), puzzle logins (3.3.8), status messages (4.1.3) and alt-text accuracy (1.1.1) need human judgement too [A24, A27, A31, A32, A7].

Does the European Accessibility Act require WCAG 2.2?

Not yet. EN 301 549 V4.1.1 (2 September 2026) references WCAG 2.2, but it takes its main legal effect only once cited in the EU Official Journal, and no version is cited under the EAA so far [A36, A45]. The previous V3.2.1 was based on WCAG 2.1 [A36]. This is not legal advice.

Sources

UX On Demand
back to top
Book 15 min with Albert
Smiling man with short dark hair and glasses wearing a black suit, white shirt, and black tie against blue background.
Tell Albert what you're shipping.
He'll read this before joining the call. Phone number comes next, on the calendar step.
↳ info@you-source.com
↳ 4-hour response
Please wait while we retrieve meeting schedules.
Oops! There's a problem with your request. We're working on fixing it. Please try again later.