Ship the three-minute RAG regression proof

[thangldw]

User outcome

A new user runs uvx ragops demo and sees a passing baseline, regressed candidate, named BLOCK reasons, and standalone HTML evidence in under three minutes. A maintainer adds the reusable gate in seven YAML lines and receives the same evidence on a pull request.

Contract impact

• Existing schemas, metrics, default evaluators, and exit codes remain compatible.
• Reusable gates add a bounded standalone HTML artifact.
• Two opt-in evaluators add explicit tagged contracts for current-source IDs and lexical abstention evidence; they are diagnostic unless selected and gated by policy.
• The default-branch publisher remains isolated from untrusted PR execution.

Acceptance criteria

• Exact no-clone command works without credentials.
• PR comment includes metric deltas, block reasons, and HTML artifact link.
• Failure Zoo covers permission leakage, stale source, wrong citation, prompt injection, failure to abstain, and excessive agency.
• Repository self-gates, full tests pass, and visual evidence comes from real output.

Initial BLOCK evidence

This first revision intentionally points the repository self-gate at benchmark-regressed.json. The expected failed check and publisher comment are part of the acceptance evidence. A follow-up commit will restore the passing candidate before merge.

Local evidence

• Ruff passed.
• 208 tests passed with plugin autoload disabled.
• Failure Zoo exits 2 with named specialized findings.
• uvx ragops demo completed in about 2.4 seconds locally.
• Terminal GIF duration: 45.01 seconds.
• Desktop 1440px and mobile 390px render without overflow or console errors.

Decision requested

Review and accept the proposed opt-in tagged evidence evaluator decision before the next public milestone. No tag, package publication, or public release is performed by this PR.

[github-actions]

RAGOps release gate: PASS

RAGOps regression check: PASS

Scenario: jp-troubleshooting-benchmark-v0.2

Metric	Baseline	Candidate	Delta
citation_coverage	1	1	+0
citation_precision	1	1	+0
lexical_groundedness	1	1	+0
avg_latency_ms	966.6667	966.6667	+0
avg_cost_usd	0.0097	0.0097	+0
critical_findings	0	0	+0

Failed gates

None.

---

Canonical evidence: workflow run · HTML report (ragops-report.html) · head 7594f83eaa3d