Quality tool: audit, overview & dedupe with HTML dashboard and agent skills#11
Open
MirjamOdile wants to merge 1 commit into
Open
Quality tool: audit, overview & dedupe with HTML dashboard and agent skills#11MirjamOdile wants to merge 1 commit into
MirjamOdile wants to merge 1 commit into
Conversation
…nt skills Adds the read-only quality layer over data/<project>/ and its one mutating counterpart: - ./scrapai audit — coverage + extraction scoring per spider (CSV/MD), compliance lenses (robots/legal snapshots, conflict flags), external-PDF host analysis, and a self-contained HTML dashboard (Coverage · Compliance · PDFs tabs). PDFs follow the framework's row model: URL-only rows with content_type="pdf"; scraped/content% are judged over HTML rows only; same-org vs external is derived from allowed_domains. - ./scrapai overview — per-spider content profile: sections, date span, field coverage, thin/constant/off-domain checks. - ./scrapai dedupe — the ONE mutating command, kept separate so the audit never rewrites data; consolidates crawls/*.jsonl reversibly (originals kept as *.superseded). Ships with three slash-command skills (.claude/commands/: spider-review, spider-align, spider-slow) that consume the audit's status groups, docs (docs/quality.md, docs/skills-overview.md, request write-ups under docs/requests/), and the .gitignore pattern for teams to keep local skills/docs untracked alongside the shipped ones. Engine tests cover corpus scanning + caching, scoring (incl. PDF-row arithmetic and reviewed-stale guards), compliance capture/report, external-PDF partitioning, dashboard rendering, and an end-to-end synthetic-project flow.
iRanadheer
force-pushed
the
pr/05-quality-tool
branch
from
3a4035b to
64d2f8a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A project-wide quality layer over
data/<project>/crawl output — three CLIcommands, an HTML dashboard, three agent skills, docs, and tests. Built and
battle-tested against live production projects before this handover.
Commands
./scrapai audit --project <name>— read-only. Per-spider coverage +extraction scoring (
crawl_audit.csv/audit_<project>.md), compliancelenses (robots/llms/legal-page snapshots with conflict flagging),
external-PDF host analysis, and a self-contained HTML dashboard
(
_audit/dashboard_<project>.html, tabs: Coverage · Compliance · PDFs).Every spider is classified into a status worklist (
ok/manual review/too few pages/incomplete/extraction broken) — the goal, stated onthe report itself: every spider ends in
ok(or deliberatelydiscarded)../scrapai overview --project <name>— read-only. Per-spider contentprofile: sections, date span + per-year histogram + null %, field coverage,
thin/constant/off-domain checks.
./scrapai dedupe --project <name>— the ONE mutating command, keptseparate so the audit never rewrites data. Consolidates
crawls/*.jsonlreversibly (originals kept as
*.superseded).Design points reviewers will care about
links_onlyharvest —URL-only rows with
metadata_json.content_type="pdf". Extraction quality isjudged over HTML rows only; the PDFs lens splits same-org vs external by URL
host against
allowed_domains(finding an org's documents stored off itsown domain is the lens's whole purpose).
_audit/crawl_stats/when a crawl recorded them (no re-fetch, no drift, noCF problems); fetched sitemaps are the fallback.
_audit/audit_sitemap_skip.jsoncorrects theyardstick BEFORE scoring;
_audit/audit_notes.jsonrecords the verdictAFTER (promote/discard). Both are self-documenting (embedded, auto-refreshed
_instructions), both suggest-only for agents, and stale notes are policed(a note can never vouch for broken extraction or an empty corpus).
corpora re-audit in seconds; caches migrate themselves on format change.
dynamic value escaped; markdown/CSVs remain the untouched ground truth.
Also in this PR
.claude/commands/):/spider-review(evidence-based triage/repair of the audit's problem groups),
/spider-align(fleet-wide conformance sweep),
/spider-slow(live-crawl ops viacrawl-status/Pueue) — plus the
.gitignorepattern that tracks shippedskills while keeping teams' local ones untracked.
docs/quality.md(full reference),docs/skills-overview.md, CLAUDE.md§6.9 + README wiring, and
docs/requests/— the per-change design write-upsbehind this PR series (the earlier small PRs reference them).
Verification
~100 dedicated unit tests across corpus scanning + caching, scoring (including
PDF-row arithmetic and the reviewed-stale guards), compliance capture/report,
external-PDF partitioning, dashboard rendering, and an end-to-end synthetic
project flow. Full suite green. Restructures were gated by byte-identical
golden-master runs against real project output.