-
Notifications
You must be signed in to change notification settings - Fork 27
[CHORE] Add axe-core WCAG 2.2 AA accessibility tests + DuckDB query tests to CI #420
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
John McCall (lowlydba)
wants to merge
25
commits into
main
Choose a base branch
from
lowlydba/add-a11y-ci
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
c991011
feat: add axe-core WCAG 2.2 AA accessibility tests to CI
lowlydba 28f85f8
docs: add WCAG 2.2 AA badge to README
lowlydba 1b9b2d9
fix: resolve 3 WCAG 2.2 AA violations caught by axe CI
lowlydba e093942
fix: prevent path traversal in a11y test static server (CodeQL js/pat…
lowlydba 81c1e82
chore: move CODEOWNERS and SECURITY.md to .github/
lowlydba bce69b7
feat: add DuckDB query smoke tests to CI
lowlydba 4f71687
fix: correct setup-python version comment (v5.6.0 not v6.2.0)
lowlydba 012b925
chore: consolidate CI into single workflow with are-we-good rollup
lowlydba a79cc31
chore: add 15 minute timeout to DuckDB query tests job
lowlydba 380b07b
test: expand a11y coverage to 7 pages
lowlydba 5818492
fix(a11y): fix WCAG AA violations on community page
lowlydba 397cde8
chore(query-tests): print query SQL and elapsed time per run
lowlydba abeb466
fix(a11y): resolve code-block, link, email, and iframe WCAG violations
lowlydba e67b6ae
chore: remove ponytail comment markers
lowlydba 40425b1
fix(query-tests): stream output live and print setup timing
lowlydba 0beba9d
feat(query-tests): colorize printed SQL to set it apart from log lines
lowlydba 7c1beab
fix(query-tests): print the exact SQL that executes
lowlydba 3f0b946
fix(query-tests): strip 'LOAD ...; --noqa' preamble so queries get LI…
lowlydba 1bc9f03
perf(query-tests): use LIMIT 0 instead of LIMIT 1
lowlydba 37e7f49
fix(query-tests): unwrap COPY queries with leading comments, add --dr…
lowlydba 775da80
fix(divisions): wire up orphaned count queries, drop dead SQL fragments
lowlydba d112c54
fix(divisions): shorten new count-tab labels so they don't truncate
lowlydba cc6276a
fix(query-tests): use AWS S3 path for Detroit and Seattle building qu…
lowlydba 100ed22
perf(query-tests): wrap COPY selects in LIMIT 0 for multi-statement s…
lowlydba eb46c7a
fix(a11y): harden static server and dedupe Tabs default
lowlydba File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| --- | ||
| name: CI | ||
|
|
||
| on: | ||
| pull_request: | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: ci-${{ github.event.number || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| unit-tests: | ||
| name: Unit tests | ||
| runs-on: ubuntu-slim | ||
| steps: | ||
| - name: Checkout repo | ||
| uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Node.js | ||
| uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0 | ||
| with: | ||
| node-version-file: package.json | ||
|
|
||
| - name: Configure sustainable npm | ||
| uses: lowlydba/sustainable-npm@31d51025884f424f58f22e4e6578178bb4e79632 # v3.0.0 | ||
|
|
||
| - name: Install NPM dependencies | ||
| run: npm ci | ||
|
|
||
| - name: Run tests | ||
| run: npm test | ||
|
|
||
| a11y: | ||
| name: Accessibility (WCAG 2.2 AA) | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout repo | ||
| uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Node.js | ||
| uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0 | ||
| with: | ||
| node-version-file: package.json | ||
|
|
||
| - name: Configure sustainable npm | ||
| uses: lowlydba/sustainable-npm@31d51025884f424f58f22e4e6578178bb4e79632 # v3.0.0 | ||
|
|
||
| - name: Install NPM dependencies | ||
| run: npm ci | ||
|
|
||
| - name: Build | ||
| run: npm run build | ||
|
|
||
| - name: Install Playwright browser | ||
| run: npx playwright install chromium --with-deps | ||
|
|
||
| - name: Run accessibility tests | ||
| run: npm run test:a11y | ||
|
|
||
| query-tests: | ||
| name: DuckDB query smoke tests | ||
| uses: ./.github/workflows/query-tests.yml | ||
| permissions: | ||
| contents: read | ||
|
|
||
| are-we-good: | ||
| name: are-we-good | ||
| runs-on: ubuntu-slim | ||
| needs: [unit-tests, a11y, query-tests] | ||
| if: always() | ||
| steps: | ||
| - uses: lowlydba/are-we-good@bb8ee9e793e4233fac1992bb880e2a28bed7f42f # v1.0.3 | ||
| with: | ||
| jobs: ${{ toJSON(needs) }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| --- | ||
| name: Query Tests | ||
|
|
||
| on: | ||
| workflow_call: # reusable — called from ci.yml on PRs | ||
| schedule: | ||
| - cron: '0 6 * * 1' # Monday 06:00 UTC — catches data-side breakage between releases | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: query-tests-${{ github.run_id }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| duckdb: | ||
| name: DuckDB query smoke tests | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
| steps: | ||
| - name: Checkout repo | ||
| uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| - name: Install duckdb | ||
| run: pip install duckdb | ||
|
|
||
| - name: Run query smoke tests | ||
| run: python scripts/test_queries.py |
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| /** | ||
| * WCAG 2.2 AA accessibility tests using Node's built-in test runner. | ||
| * Requires a built `build/` directory — run `npm run build` first. | ||
| * Run with: npm run test:a11y | ||
| */ | ||
|
|
||
| import { test, before, after } from 'node:test'; | ||
| import assert from 'node:assert/strict'; | ||
| import { createServer } from 'node:http'; | ||
| import { readFile, access } from 'node:fs/promises'; | ||
| import { extname, join, resolve, relative, isAbsolute } from 'node:path'; | ||
| import { fileURLToPath } from 'node:url'; | ||
| import { chromium } from 'playwright'; | ||
| import AxeBuilder from '@axe-core/playwright'; | ||
|
|
||
| const __dirname = fileURLToPath(new URL('.', import.meta.url)); | ||
| const BUILD_DIR = join(__dirname, '..', 'build'); | ||
| const PORT = 3778; | ||
| const BASE_URL = `http://localhost:${PORT}`; | ||
|
|
||
| const MIME = { | ||
| '.html': 'text/html', | ||
| '.js': 'application/javascript', | ||
| '.mjs': 'application/javascript', | ||
| '.css': 'text/css', | ||
| '.png': 'image/png', | ||
| '.svg': 'image/svg+xml', | ||
| '.woff2': 'font/woff2', | ||
| '.json': 'application/json', | ||
| }; | ||
|
|
||
| let server, browser; | ||
|
|
||
| before(async () => { | ||
| await access(join(BUILD_DIR, 'index.html')).catch(() => { | ||
| throw new Error(`build/index.html not found — run 'npm run build' before running accessibility tests`); | ||
| }); | ||
|
|
||
| // minimal static file server, falls back to index.html for Docusaurus client-side routing | ||
| server = createServer(async (req, res) => { | ||
| const urlPath = (req.url ?? '/').split('?')[0]; | ||
| const filePath = resolve(join(BUILD_DIR, urlPath === '/' ? 'index.html' : urlPath)); | ||
| // relative() is cross-platform; startsWith(BUILD_DIR + '/') broke on Windows backslashes | ||
| const rel = relative(BUILD_DIR, filePath); | ||
| if (rel.startsWith('..') || isAbsolute(rel)) { | ||
| res.writeHead(403); | ||
| res.end('Forbidden'); | ||
| return; | ||
| } | ||
| try { | ||
| const data = await readFile(filePath); | ||
| res.writeHead(200, { 'Content-Type': MIME[extname(filePath)] ?? 'application/octet-stream' }); | ||
| res.end(data); | ||
| } catch { | ||
| // SPA fallback only for route-like requests (no extension); a missing | ||
| // asset (.js/.css/.png) must 404 so broken builds aren't masked. | ||
| if (extname(filePath) === '') { | ||
| try { | ||
| const data = await readFile(join(BUILD_DIR, 'index.html')); | ||
| res.writeHead(200, { 'Content-Type': 'text/html' }); | ||
| res.end(data); | ||
| return; | ||
| } catch { /* fall through to 404 */ } | ||
| } | ||
| res.writeHead(404); | ||
| res.end('Not found'); | ||
| } | ||
| }); | ||
| await new Promise((resolve) => server.listen(PORT, resolve)); | ||
| browser = await chromium.launch(); | ||
| }); | ||
|
|
||
| after(async () => { | ||
| await browser?.close(); | ||
| if (server) await new Promise((resolve) => server.close(resolve)); | ||
| }); | ||
|
|
||
| function formatViolations(violations, label) { | ||
| if (violations.length === 0) return; | ||
| const detail = violations | ||
| .map((v) => | ||
| `[${v.impact}] ${v.id}: ${v.description}\n` + | ||
| v.nodes.map((n) => | ||
| ` • ${n.target}` + | ||
| (n.failureSummary ? `\n ${n.failureSummary.replace(/\n/g, '\n ')}` : '') | ||
| ).join('\n') | ||
| ) | ||
| .join('\n\n'); | ||
| assert.fail(`${violations.length} WCAG 2.2 AA violation(s) — ${label}:\n\n${detail}`); | ||
| } | ||
|
|
||
| async function runAxe(url) { | ||
| const context = await browser.newContext(); | ||
| const page = await context.newPage(); | ||
| try { | ||
| await page.goto(`${BASE_URL}${url}`, { waitUntil: 'domcontentloaded' }); | ||
| await page.waitForSelector('main', { timeout: 10000 }); | ||
| const { violations } = await new AxeBuilder({ page }) | ||
| .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa']) | ||
| .analyze(); | ||
| return violations; | ||
| } finally { | ||
| await context.close(); | ||
| } | ||
| } | ||
|
|
||
| const PAGES = [ | ||
| { path: '/', label: 'homepage' }, | ||
| { path: '/docs/', label: 'docs index' }, | ||
| { path: '/getting-data/', label: 'quickstart (QueryBuilder)' }, | ||
| { path: '/guides/', label: 'guides index' }, | ||
| { path: '/examples/', label: 'examples index' }, | ||
| { path: '/blog/', label: 'blog index' }, | ||
| { path: '/community/', label: 'community (custom table)' }, | ||
| ]; | ||
|
|
||
| for (const { path, label } of PAGES) { | ||
| test(`${label} passes WCAG 2.2 AA`, async () => { | ||
| formatViolations(await runAxe(path), label); | ||
| }); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.