docs(workflows): primitive references for verification scaling (V11, #2491) - #2524
Merged
Conversation
…o verifier/docs-primitives # Conflicts: # packages/coding-agent/docs/workflows.md # packages/workflows/CHANGELOG.md
…ifier/docs-primitives
Declare finalDecision, scoreTablePath, and reviewReportPath without placeholder values that GitHub Code Quality flagged as dead stores. Every loop path assigns them before return. Assistant-model: Grok 4.6
flora131
marked this pull request as ready for review
August 19, 2026 16:53
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+776
to
+786
| <<<<<<< HEAD | ||
| | `adversarial-verification` | worker → per-criterion fresh rubric verifiers → deterministic mean+veto reducer → findings consolidation / bounded repair; decompose rubrics into named criteria and aggregate by mean plus explicit veto, never a unanimity AND; `criteria.md` uses `## Criteria` with non-empty `### Name {#id}` bodies and the shared `verification-criteria` doors; consolidator cannot approve | `task`; `criteria` (record or criteria.md markdown, defaults to task_fit/evidence/completeness), `verifier_count=3` (1–5), `max_repairs=2` (0–5), `accept_mean=14`, `reask_limit=1`; invalid reports are re-asked within the bound and never become fail votes; normal calls per round: criteria.length × verifier_count | `approved`, `mean_score`, `score_table_path` (`verification-summary-<round>.json` with scores, mean, invalidCount, decision, and usage), `repairs_completed`, `candidate_path`, `review_report_path`, `remaining_work` | | ||
| | `generate-and-filter` | candidate fan-out → dedupe/filter → optional judge → shortlist; use graded per-criterion scores, Bradley–Terry preference from score gaps, and slot-swap repeats when ranking candidates; custom judges can reuse the shared `verification-criteria` module and `criteria.md` rubric format | `prompt`; `num_candidates=8`, `shortlist_size=3`, `use_judge=true`, `max_concurrency=4` | result, shortlist and candidate/filter/judge/final/manifest paths | | ||
| | `tournament` | independent attempts → seeded ring and pivot-round soft scoring with order-balanced pairwise judges → full ranking/comparisons reducer; derive a Bradley–Terry preference from graded score gaps and repeat with A/B slot swaps; `comparisons.json` preserves per-job score or invalid rows, pair aggregates, ranking, and planned/executed budget records | `prompt`; `num_attempts=4`, `max_concurrency=4`, `n_evaluations=2`, `pivots=1`, `seed=0`, optional `criteria` (markdown rubric, record, string list, or `CriterionInput` list)/`models` | result, winner, `attempt_artifact_paths`, `judge_artifact_paths`, `comparisons_path`, `ranking`, `seed` | | ||
| | `loop-until-done` | durable ledger → iteration/evaluator loop → complete or inspectable exhaustion; each scored iteration records `progress` (`score`, `perRepeat`, `trend`, `window`) and the ledger emits `progress_curve`, `final_trend`, and `progress_disclaimer`; trend is advisory and never a kill switch | `prompt`; `max_iterations=5`, `progress_scoring=true`, `progress_repeats=1` | result/status, ledger, iteration/evaluation paths, remaining work, `progress_curve`, `final_trend`, `progress_disclaimer` | | ||
| ======= | ||
| | `adversarial-verification` | worker → per-criterion fresh verifier fan-out → deterministic mean+veto gate → findings consolidation / bounded repair; consolidator cannot approve | `task`; `criteria` (record or criteria.md markdown, defaults to task_fit/evidence/completeness), `verifier_count=3` (1–5), `max_repairs=2` (0–5), `accept_mean=14`, `reask_limit=1`; normal calls per round: criteria.length × verifier_count | `approved`, `mean_score`, `score_table_path`, `repairs_completed`, `candidate_path`, `review_report_path`, `remaining_work` | | ||
| | `generate-and-filter` | candidate fan-out → dedupe/filter → optional judge → shortlist | `prompt`; `num_candidates=8`, `shortlist_size=3`, `use_judge=true`, `max_concurrency=4` | result, shortlist and candidate/filter/judge/final/manifest paths | | ||
| | `tournament` | independent attempts → seeded ring and pivot-round soft scoring → full ranking reducer | `prompt`; `num_attempts=4`, `max_concurrency=4`, `n_evaluations=2`, `pivots=1`, `seed=0`, optional `criteria`/`models` | result, winner, `attempt_artifact_paths`, `judge_artifact_paths`, `comparisons_path`, `ranking`, `seed` | | ||
| | `loop-until-done` | durable ledger → iteration/evaluator loop → complete or inspectable exhaustion | `prompt`; `max_iterations=5` | result/status, ledger, iteration/evaluation paths, remaining work | | ||
| >>>>>>> origin/main |
There was a problem hiding this comment.
The builtin-workflow table publishes literal Git conflict markers and two competing rows each for adversarial-verification, generate-and-filter, tournament, and loop-until-done. Markdown rendering converts the markers and alternate rows into headings, paragraphs, and blockquotes instead of preserving the table, so users receive malformed and contradictory workflow contracts. Resolve the conflict and retain one reconciled row per workflow.
Context Used: AGENTS.md (source)
Artifacts
Exact README source range with unresolved merge-conflict markers
- Executed `sed -n "776,786p" packages/workflows/README.md` in `/home/user/repo`; it shows the three literal conflict markers and both sets of workflow rows, confirming the malformed source.
Markdown rendering check source
- Node check source used to read lines 776-786, count the markers and duplicated workflow rows, and render the exact range with the installed `marked` parser.
Marked rendering of the malformed builtin workflow section
- Executed the supplied Node check in `/home/user/repo`; it reports all three markers, two copies of each affected workflow, and malformed HTML instead of a valid table, confirming the documentation defect.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/workflows/README.md
Line: 776-786
Comment:
**Unresolved workflow-table merge conflict**
The builtin-workflow table publishes literal Git conflict markers and two competing rows each for `adversarial-verification`, `generate-and-filter`, `tournament`, and `loop-until-done`. Markdown rendering converts the markers and alternate rows into headings, paragraphs, and blockquotes instead of preserving the table, so users receive malformed and contradictory workflow contracts. Resolve the conflict and retain one reconciled row per workflow.
**Context Used:** AGENTS.md ([source](https://github.com/bastani-inc/atomic/blob/main/AGENTS.md))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Slice V11 of the LLM-as-a-Verifier adoption program (
specs/2026-08-17-verifier-docs.md§3.2 / §4.1 V11 column): primitive-level references for every shipped verification primitive, added to the D1 pattern sections.Documents (defaults copied from shipped source on this branch, not the spec):
criteria.mdformatpivots1,n_evaluations2,seed0,criteria,models) andcomparisons.jsoncriteria+accept_mean14 + re-ask semanticsrepeatsdefault 1) + loop ledger entriesstop_review_loopremains authoritativeThis branch also merges V8 (
verifier/progress-consumers) and D1 (verifier/docs-patterns) so the docs sit on the full primitive set. Docs-only for the V11 delta; no CHANGELOG (docs guidance is not shipped-package behavior).Base:
verifier/goal-convergence(V10, #2521). Caps Stack V.Evidence
Produced by an implement→review→repair goal run (2 turns; completion/evidence/risk reviewers all
complete;remaining_work: none):npm run check— greendocument tournament inputs,document adversarial verification reports,document progress trend outputs,document review convergence evidence, plus the repair commits that fixed artifact/output names and anchors)Spec contract:
specs/2026-08-17-verifier-docs.md§4.1 V11 column + §4.2 fidelity rules.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Greptile Summary
This change expands the workflow documentation with verification criteria, comparison artifacts, progress trends, and convergence guidance. The builtin workflow reference in
packages/workflows/README.mdis broken: unresolved Git conflict markers and duplicate competing rows cause the Markdown to render outside the intended table. Reconcile the four duplicated workflow rows and remove the markers before merging.Confidence Score: 4/5
Not safe to merge until the malformed builtin workflow table is reconciled.
The affected Markdown was rendered with the installed parser, which demonstrated that the literal conflict markers and alternate rows break the table structure.
Files Needing Attention:
packages/workflows/README.mdlines 776-786 need the conflict markers removed and one accurate row retained for each builtin workflow.What T-Rex did
Comments Outside Diff (1)
General comment
packages/workflows/README.mdlines 776-786 publish literal Git conflict markers plus conflicting duplicate descriptions for four builtin workflows. A real Markdown render does not preserve the table: it emits a heading containing<<<<<<< HEAD, a paragraph containing the alternate rows, and nested blockquotes for>>>>>>> origin/main.HEADandorigin/mainwas committed without selecting and reconciling one version of the four workflow rows.Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "merge origin/main" | Re-trigger Greptile
Context used: