docs(concept): reviewer redesign — pipeline, architecture, config, quality, harness#293
Open
sebastianwessel wants to merge 25 commits into
Open
docs(concept): reviewer redesign — pipeline, architecture, config, quality, harness#293sebastianwessel wants to merge 25 commits into
sebastianwessel wants to merge 25 commits into
Conversation
Draft specification for realigning QualOps to a precision-first "generate wide, verify adversarially, filter hard, publish little" pipeline, a contracts-first layered code structure, a folder-based configuration model, an evaluation strategy with a noise/precision north star, and an open ADR on the agent-loop harness backend. Evidence-backed by a code audit, industry/literature research, a gap analysis against QualOps' own codereviewer spike, and measured dependency/license facts for the harness options considered. Nothing here is implemented; this is a draft for human review. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
QualOps Code Quality AnalysisStatus: ✅ PASSED - No issues found Summary
No issues found in the analyzed code. 📊 Full ReportPowered by QualOps |
Resolve the open harness ADR (08): the team chose the Vercel AI SDK as the agent-loop backbone and ruled out maintaining an own harness implementation. Both the from-scratch custom loop and adopting/forking @purista/harness are rejected — not on technical merit but because a Python-centric org will not staff a TypeScript harness. @openai/agents and @eggai/configurable-agent are retired. The AgentRunPort abstraction and evaluation record are preserved so the choice stays swappable (Eve at GA) and auditable. Propagated the decision through the glossary/decision-log (01), architecture (03), and roadmap (06). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ept→spec flow Rework 03 (architecture) given the harness decision: - provider layer collapses into @ai-sdk/* — only capabilities/pricing/ accounting survive into llm/model; ~7 bespoke provider files removed - ports (CompletionPort, AgentRunPort) declared in contracts/ports, implemented by the single AI SDK backend in llm/backend/ai-sdk - exclusivity rules, dependency policy, centralization note, and the structural budget updated accordingly; transition sequencing noted (ports wrap current providers in the refactor phase, swap in Phase 2) Reframe the concept README to the concept→spec→implementation→ documentation flow: these are concept-stage proposals, not binding specs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…spec) Create specs/ with the concept→spec→implementation→documentation flow documented, and promote the refactor plan as the first approved spec: - layered target structure, one-definition-per-concept type unification, centralization of duplicated utilities incl. the model-output boundary - defect inventory bucketed A (internal) / B (latent-bug) / C (intended behavior change); behavior-correcting fixes F-1/F-2/F-13 etc. are IN the phase with changelog + release notes, per decision - normative reviewability method: move≠edit, strangler shims, characterization tests, one-singleton-at-a-time, the PR-stack order - exit criteria tied to the structural budget Link concept/06 Phase 0–1 to the graduated spec so they don't drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CLAUDE.md: binding engineering contract (spec-driven flow, layered
structure, no fakes, one-definition-per-concept types, error/logging,
security, side-by-side unit tests + ≥80% coverage, docs-in-sync).
AGENTS.md: pointer to it for AI tools.
- specs/: the post-refactor baseline (current behavior, restructured):
architecture, contracts (unified types/validation), behavior/{pipeline,
configuration,integrations}, quality. Each behavior spec carries a
refactor-acceptance list flagging the bucket B/C corrections and open
discrepancies found while speccing.
- Migrate docs/tdr → specs/adr (history preserved); ADR 0004 marked
superseded by the harness decision; fix ADR index; repoint CONTRIBUTING.
- Remove stale plan.md / progress.md (2025-11 test journal).
Code unchanged — this round is spec/concept/planning only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- specs/documentation.md: root README standard (crisp intro + quick start + TOC into docs/) and docs/ standard (concise, current, nested by reader journey, no stubs); docs/ is the authored source of truth website/ syncs from. IDs (DOC-1..11) with acceptance/verification (AC-1..5) and open items. - Wire the standard into CLAUDE.md §8 and specs/quality.md §7. - Per the spec-driven flow (spec-architect): specs are not self-approved. Downgrade all newly-authored specs from "Approved" to "Draft — pending spec-readiness-review + human approval"; clarify the approval rule in specs/README. Authored with the spec-architect skill. Code unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dissolve the flat specs/adr/ folder into the domain-organized spec tree: - operations/release.md ← ADR 0001 (release/versioning contract; tables + promotion mermaid) - evaluation/eval-cases.md ← ADR 0002 (slice eval-case data contract; schema table) - behavior/review-dialects.md ← ADR 0003 (structured/prose routing; dialect table + selection mermaid); now the single dialect home — pipeline.md/integrations.md point here instead of duplicating - decisions/ (0001–0004 + README): lean records keeping the WHY/alternatives/status, each linking to the domain spec that holds the WHAT. 0004 recorded as superseded by the harness decision (concept/08). Specs are contract-only (no implementation), markdown with tables/mermaid, separated by purpose/domain/responsibility so human reviewers and AI agents can plan/implement gap- and guess-free. Repoint CONTRIBUTING to operations/release.md; add a spec-map mermaid + domain index to specs/README. Code unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deepen the domain nesting so each file is one responsibility (readable by humans, targetable by AI agents): - behavior/pipeline/ — per stage: README (orchestration + acceptance list), intake, review, review-dialects (moved here), fix, reporting, gate - behavior/configuration/ — cli, config-file, action-and-env, custom-agents - behavior/integrations/ — providers, github, gitlab - quality/ — testing, error-handling, logging, security, dependencies (+ README carrying the no-fakes and docs-hygiene principles) Each folder has a README overview. Repointed all inbound links (contracts, architecture, documentation, decisions, evaluation, specs/README index + mermaid map); full link audit passes with zero broken links. Contract-only, markdown with tables/mermaid. Code unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ran spec-readiness-review over the specs/ set. Verdict: NOT ready for approval. Records deterministic results (0 broken links, 0 contradictions; fails on missing framework scaffolding + Owner: TBD + Public API section), 8 open human decisions (4 blocking), and the authoring gaps to close before approval. No source-code changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve all open decisions from the readiness review and mark the baseline spec set human-approved (EggAI, 2026-07-08): - fix stage honors fix.severities/minConfidence + --include-medium (critical now eligible) — bucket C - forge config path unified to .qualops/.qualopsrc.json (GitLab aligns; migration + release note); forge report source = latest session only - gate thresholds config section named `gate` - owner = EggAI across all specs - documentation: website-only, no repo docs/ folder; README short/crisp/precise with no fixed limit; CI doc-checks deferred to the documentation phase - CLAUDE.md/AGENTS.md doc rules updated to website-only Readiness report updated to status: approved with resolved_decisions and accepted_residual_gaps. Scope: specs/, CLAUDE.md, AGENTS.md only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Migrate the GitHub Action setup guide into the website as a native Starlight
page and repoint references (docs/ folder was removed in the prior commit).
- website/src/content/docs/github-action/setup.md (route /github-action/setup/)
- sync-root-docs.mjs: drop the docs/ dependency (keeps CHANGELOG/CONTRIBUTING/
example-prompt syncs); untrack the now-native setup page
- repoint dead links: .qualops example → website setup guide; releases.mdx →
specs/decisions/0001
Note: matching cleanups in .github/workflows/{docs,promote-to-stable,
create-release-pr}.yml (drop docs/** trigger; repoint a comment to
specs/operations/release.md) could not be pushed (token lacks `workflow`
scope) — left for a maintainer to apply.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a researches/ folder for point-in-time research reports (one folder per topic, background evidence — not binding). Migrate PR #149's agent-evaluation-research into researches/agent-evaluation/ verbatim (main report + 4 source dossiers + 3 diagrams + HTML), and add a researches/README index explaining how researches/ relates to concept/, specs/, and website/. Lets PR #149 be closed (its content now lives here). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read every open issue (bodies + comments) and compared against the concept set, approved baseline specs, and current code (claims verified in source): - adopt reshaped: MCP context sources (#83 + #71.3) with D12 security constraints; language-packaging frontmatter from the #70 skill proposal - already covered/superseded: agent init (#71.1), agentic observability (#69 → port trajectory contract), agentic unit tests (#68 → port conformance suite; code being replaced) - reject with evidence: agent chaining (#71.2 — TDR 0005 A/B: worse at 4× cost), marketplace (#71.5 — zero-infrastructure non-goal) - verified stale/resolved: .ts-only intake filter already removed (#70 must-have), Action builds at runtime (#121), docs site shipped (#18), all 15 auto release-failure issues (releases succeeded) - cross-cutting: failure-issue mechanism files 3 dupes per failure and never auto-closes — dedupe/auto-close recommended for the release spec Verdicts pending EggAI confirmation before issue actions are taken. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lace listing The issue's literal ask (hosted agent registry + install command) stays rejected (zero-infrastructure non-goal). Two lightweight substitutes adopted: publish QualOps to the GitHub Actions Marketplace (verified 2026-07-08: not listed at all; action.yml metadata already qualifies — a checkbox on the next stable release, best timed with the README polish), and community reviewer sharing via extends-presets + an in-repo curated folder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…k loop) Answer EggAI's three eval questions, verified against evals/ and the codereviewer spike's eval/ folder: - Tracking: Langfuse is REQUIRED (exits without creds); NO committed history (Langfuse-remote + gitignored logs only). Genuine gap → propose a committed scoreboard + dataset fingerprint AND make Langfuse optional/degradable. - Dataset growth: already concept 05 §3 intent (n=3 is too small); port the spike's per-language negative/no-finding cases (~17%) and category/tier taxonomy (productRecall). Reinforcement + concrete specifics. - Quick loop: already concept 05 §7 (fast gate); port the spike's deterministic OFFLINE matcher → a creds-free eval:quick with deterministic scorers only. Keep Langfuse (better than the spike's ephemeral model); add offline+committed alongside, don't regress. Flags the CHANGELOG advertising unshipped A/B tooling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mapping Refines 10-eval-operations with a multi-language / multi-use-case benchmark review (§4 adopt-vs-build: keep CRB, defer security/CVEfixes for label-noise + reshape cost, build performance/memory/concurrency synthetically) and a source-verified spike-port mapping (§5: field map + the crb-pairwise null-on-empty-expected and loader blockers that make the ported negatives actually count). 05-quality-spec gains the no-finding / spurious-rate scorer note (§2) and a synthetic multi-use-case planted-defect dataset (§3 item 8). Concept only; no spec change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sign # Conflicts: # docs/tdr/README.md
…d decision 0005 - concept/10: the A/B tooling flag was stale post-merge; run-ab.ts, compare-experiments.ts, --repeats and --eval-log all now exist in the tree, so the changelog/reality gap is closed. Remaining eval-ops gaps are the committed scoreboard + optional-Langfuse only. - specs/decisions: add 0005 (intent-based agentic review — Rejected 2026-07-02) so the decision index reflects the TDR main added; the review pipeline stays a flat agentic pass, filtering is the forward path. Full text lives in CHANGELOG + git history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sebastianwessel
marked this pull request as ready for review
July 8, 2026 21:24
…, D13/D14 Folds the destructure/context brainstorm into the concept, evidence-backed by four research passes (new appendix E): - 02 §3.1 rewritten as the deterministic intake stage: hunk classification (format/comment-only never reach an LLM), change-unit grouping (file-level v1, residual-unit fallback, honest 70-85% literature ceiling), per-unit tiering, impact sets (references AND callers), context packs (PageRank ranking, bounded digests, priority-ladder budget, cache-aware layout). - 02 §3.2: per-unit reviewer selection; single-shot holistic default with agent-mode as escalation tier (D13, eval-gated flip). §3.3: cause-anchoring scope semantics. §3.4: verifier slice specified (enclosing declaration + one def-use hop; batch+cache economics). - 01: glossary gains Change-unit / Context pack / Impact set, Tier goes per-unit; decisions D13 (default flip, refines D8) and D14 (destructuring is deterministic and scopes context only — the lane TDR 0005 left open). - 03: intake submodules + dependency justification (ast-grep/tree-sitter in, CodeQL/SCIP/stack-graphs rejected). 06: Phase 2 gains intake destructure v1. 07: three eval-gated backlog rows. 05: context_miss_rate metric. 04: paths-is-the-only-authored-selector note. - appendix/E: condensed evidence dossier (decomposition SOTA, hybrid deterministic+LLM results, verified cost mechanics, spike code-read, honest limits). Cross-consistency reviewed; 7 findings fixed (citations, enum value, glossary gaps, stale vocabulary, metric gap, 04 cross-reference). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e caveat Align docs with decisions made while implementing the structure refactor: - Terminology: "forge" -> "integration" across concept/ and specs/ (glossary term, architecture tree, centralization map, roadmap, behavior/integrations, contracts, refactor plan). The "forge" jargon was unclear. - Centralization map (concept/03 §5 + specs/plans/refactor.md §5): add the behaviorally-divergent-"duplicate" caveat — escapeHtml, estimateTokens, the four location parsers, retry (rethrow vs swallow), glob (dot option) can't be blind-merged under no-behavior-change; concurrency isn't stdlib-pure (logger dep) so it stays out of kernel/. A map row means one home, not one implementation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Exploratory concept 11: @eggai/configurable-agent as the single execution backend behind AgentRunPort/CompletionPort, with reviewer folders compiling to AgentConfigs so QualOps sheds its provider clients, three parallel agent adapters, template engine, and JSON-repair layer. Re-measures the four blockers 08 recorded (AI SDK v7 now, 162 prod packages with 144 attributable to the service-only OTel auto-instrumentations, OTel via /lib verified; license/repository metadata still open and gating), lists prerequisites CA-R1..R10 the runtime must ship first, argues the maintenance-ownership question both ways, and gates any adoption on a released core package, the port conformance suite, and eval-scoreboard parity.
…estimate Answers the feasibility questions raised in review: where the accuracy risk actually concentrates (Anthropic agentic path off the Claude Agent SDK, tolerant->strict structured output — both eval-gated with documented fallbacks), an effort estimate per repo (~3-4 wk runtime, ~4-6 wk QualOps, release-first sequencing), and the stacked-package design that reconciles batteries-included MCP tool packs (e.g. Playwright) with the slim core: opt-in tools-* packages carrying their own security profile, approval defaults, and config-schema fragment; providers as optional peers; QualOps installs zero packs per D12.
States plainly that the accuracy risk profile is identical in both options (both exit the Claude Agent SDK and move to strict structured output), so the decision hinges on glue ownership (~600-900 LoC QualOps-owned wrappers vs ~250 LoC compiler), features beyond the port (compaction, summarization, approval, config artifact, CLI isolation), timeline shape (one repo/no gate vs release-gated two-repo chain), and the organizational break-even (second consumer or feature requirements vs QualOps-only).
…igurable subagents Grounds the comparison on shipped v7 features: subagents are a native pattern available to both options; WorkflowAgent is Vercel-runtime-bound and wrong for CI. Splits roadmap items by shape — pipeline-shaped work (deterministic stages, fan-out, workflows) stays QualOps code under both options per TDR 0005 evidence; run-shaped work (verifier, gated fix proposals via run_paused resume, subagent teams) is where compiled configs compound over call sites. Adds CA-R12 for folder-defined subagent teams.
…sthrough Applies the run-shaped/pipeline-shaped split to context handling: input-side context (context packs, tiering, ledger) is deterministic QualOps code under both options per the concept's own determinism-before-tokens principle; in-run hygiene (summarizing compaction, tool-output summarization) is B's strongest feature edge with fresh evidence that 08's ~50-line wrapper framing underprices it; pruneMessages is free in both; cross-run memory is irrelevant. Records the honest B-gap: prompt-cache control requires a new providerOptions passthrough (CA-R13) or B regresses cost where A keeps Anthropic cache hits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Draft specification for reshaping QualOps around one goal: review a PR and deliver the most reliable feedback possible without polluting the review. Nothing here is implemented — it's a spec for discussion and approval before any code changes.
@purista/harnessvs. Vercel AI SDK vs. Claude Agent SDK vs. others — with measured dependency/license facts, not guesses).Core thesis
The current pipeline measures recall well but has no deterministic noise guardrails, no stable finding identity, no independent verification, a CI gate that doesn't actually gate, and evals that don't measure false positives. The industry's converged answer — generate wide → verify adversarially → filter hard → publish little — is the target shape.
Test plan
concept/README.mdand the linked spec docs🤖 Generated with Claude Code