Question. W3C maintains a deliberately inaccessible demo page and documents every barrier on it. What share of that list does an AI review catch, and how does that compare with a rules scanner on the same page?
Test. The CityLights home page from W3C's Before and After Demonstration, demo chrome removed, with W3C's inline annotations and evaluation report compiled into 20 documented barrier types (World Wide Web Consortium Web Accessibility Initiative, 2012). Three methods: AI review of the markup, AI review of markup plus a full page screenshot, and an axe-core 4.14.0 scan (Deque Systems, 2026). Two Claude models, ten runs per cell, 40 of 40 delivered, preregistered and fully logged (Progressive Disclosure, 2026).
Observed difference. The scanner found 5 of 20 types: missing alt, missing select label, missing lang, low contrast, unnamed links. Three runs, identical output. The AI reviews averaged 12.5 to 14.2 types per run under the preregistered keyword scoring, 14.7 to 16.5 after correcting three scoring artifacts logged in the record. The AI's surplus was everything that takes judgment: the verbose logo alt, the placeholder phone number alt, Read more links, headings styled as paragraphs, reading order, the select that navigates on change, the script that blurs keyboard focus. The screenshot bought nothing: recall was slightly lower with it, in both models. AI plus scanner together covered all 20.
Limit. One famous page, public since 2005 and plausibly in training data, so recognition may inflate recall. Keyword scoring is approximate. Catching a barrier type once is not finding every instance, and none of this measures fix quality. WebAIM (2026) finds detectable WCAG 2 failures on 95.9 percent of top home pages; detection was never the bottleneck.
Takeaway. Run both. The scanner is the deterministic floor, the AI reads like a junior auditor with opinions, and neither alone is an audit.
Experiment record (essentials)
Preregistered 2026-10-15 before the first model call (protocol.json, sha256 231a240fce7cd6d3fb8ff09f78fbae8094a7365099ac392705825b0eefbabe7d). Stimulus: W3C BAD inaccessible home page, downloaded 2026-10-15 from w3.org (original sha256 6c9d1b0518020e80f7563272a15e3fca324105e2752bee8a4095e09600863512), W3C demo chrome removed by a logged edit script (modified sha256 c2330d462df75023a43781f1da454ae6a95bfe117b2a1f63a5df5fd7e78bc94f); no CityLights markup changed. Ground truth: 20 barrier types compiled from W3C's inline annotations and evaluation report, AAA mirrors merged, page title excluded after the chrome edit. Conditions: 2 methods (markup only; markup plus full page screenshot) x 2 Claude models (harness aliases sonnet and haiku, Claude Agent SDK, provider default sampling; exact snapshot IDs not exposed) x 10 isolated runs, plus axe-core 4.14.0 in Playwright Chromium, three identical runs. 40 of 40 model runs returned parseable JSON; zero parse failures. Preregistered first-match keyword mapping; three systematic mapping foldings (focus blur, select label, images of text) detected in the spot check and reported as a corrected secondary analysis, never silently fixed. Findings outside the documented list (sonnet 3.2 to 3.9 per run, haiku 0.4 to 0.6) are reported as such; the list is not exhaustive, so they are not labeled false alarms. Full numbers in results.json; raw outputs in raw/. No human participants; no claims about user experience or fix quality.