feat(workflows): progress scoring + trend classifier (V7, #2489)

[flora131]

Summary

Slice V7 of the LLM-as-a-Verifier adoption program (specs/2026-08-17-progress-scoring.md §5.1): the progress-scoring module and pure trend classifier.

packages/workflows/builtin/progress-scoring.ts (new):

• build_progress_prompt — skeptical calibration baked in: trust observed output not narration; effort/step count is not progress; declarations of success are zero evidence; scores may plateau or fall; the prompt never reveals eventual success. Scores on VERIFICATION_SCALE 1..20 oriented "would the CURRENT state satisfy the acceptance criteria". Uses the V3 head/tail layout (steps in the shared head, checkpoint list at the tail) for prefix-cache reuse across repeats.
• score_progress(ctx, {problem, steps, checkpoints?, repeats?}) — checkpoints default to interior 2..T−1; repeats default 1 (Q1); one call per repeat scores all checkpoints; an invalid repeat contributes nothing; a checkpoint with zero valid scores is null, never invented; out-of-range checkpoint and empty prefix throw.
• classify_trend(series, config?) — pure hysteresis classifier (named defaults: window 3, ±1.5) returning {trend: rising|flat|regressing, evidence} with no action variant in the type — nothing in this module can terminate, fail, or block anything. Short series = flat.

Consumers (loop-until-done ledger, subagent attention) are slice V8. Base: verifier/prefix-cache-prompts (V3, #2510).

Evidence

Produced by an implement→review→repair goal run (approved in 1 turn; completion/evidence/risk reviewers all complete; remaining_work: none):

• npm run check — green (includes the coding-agent tsgo erasableSyntaxOnly pass)
• npx vitest --run --project unit -t "progress-scoring" — green: calibration rules present in the built prompt, no success leakage, head byte-identical across repeats, extraction fixtures incl. partial and fully-invalid repeats, null checkpoint never invented, hysteresis under alternating noise, short-series flat, low-and-flat classified flat not regressing
• Size cap: 231 changed source lines (< 500; tests uncapped — 486 total insertions)
• CHANGELOG: packages/workflows/CHANGELOG.md under ## [Unreleased] ### Added

Spec contract: specs/2026-08-17-progress-scoring.md §5.1 (Q1: repeats default 1).

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Greptile Summary

This change adds checkpoint-based workflow progress scoring, repeat aggregation, and trend classification. Execution checks reproduced two issues in packages/workflows/builtin/progress-scoring.ts: earlier checkpoints are scored with later workflow outcomes in view, and invalid repeat counts can silently return empty results, execute an unexpected number of times, or continue indefinitely. These behaviors should be corrected before merging.

Confidence Score: 3/5

Not safe to merge until checkpoint input is bounded to the requested point in the workflow and repeat counts are validated.

Two independently reproduced correctness problems remain. Both were exercised through the exported scoring utility with captured prompts and mocked task execution.

Files Needing Attention: packages/workflows/builtin/progress-scoring.ts needs checkpoint-scoped prompt construction and validation that repeats is a finite positive integer. test/unit/progress-scoring.test.ts should cover both behaviors.

T-Rex Logs

What T-Rex did

• • T-Rex produced a finding-proof for a posted P1 finding, with artifacts that help verify prompt exposure containment and related prompt executions during baseline and production runs.
• • T-Rex produced a second finding-proof for a P1 finding, with artifacts detailing a mocked score_progress repeat-validation harness, finite invalid repeat execution, infinity repeat bounded execution, and existing progress-scoring unit-test output.
• • T-Rex performed contract validation showing baseline and production tests for the progress prompt harness, with commands run and outcomes indicating baseline limits and production exposure of future steps.
• • T-Rex documented a contract-validation finding on the progress-scoring loop, identifying the unchecked repeats and infinite loop risk and proposing a finite, integer guard plus regression tests, with uploaded harness and command-output evidence.
• • T-Rex produced an additional P1 finding-proof.

_{Ran code and verified through T-Rex}

Comments Outside Diff (2)

1. General comment

   Interior checkpoint scoring prompt leaks future workflow observations

   • Bug
     • For requested checkpoint 2, the real score_progress task prompt contains the later observations FUTURE-THREE: acceptance suite passed after the final fix and FUTURE-FOUR: deployment outcome succeeded. Those outcomes can inflate or otherwise influence the score for the earlier state, contradicting checkpoint semantics.
   • Cause
     • scoreRepeat calls build_progress_prompt({ ...input, checkpoints }) without deriving a checkpoint-bounded step prefix. build_progress_prompt then serializes all input.steps into the prompt's shared <steps> head; the checkpoint list is merely an instruction, not a data boundary.
   • Fix
     • Generate a bounded observation prefix for each scored checkpoint (or otherwise make each checkpoint's visible step range explicit and enforce it structurally). If batching remains necessary, the prompt must serialize separate checkpoint-scoped sections without later-step content for each score.

   _{Ran code and verified through T-Rex}

2. General comment

   score_progress does not validate repeats before using it as a loop bound

   • Bug
     • Passing Infinity to repeats invoked the mocked task at least five times and did not complete within two seconds; the process was terminated by timeout (exit 124). Passing zero and -2 produced zero calls and null scores. Passing 2.5 produced three calls and three per-repeat results, rather than rejecting an invalid repeat count.
   • Cause
     • At packages/workflows/builtin/progress-scoring.ts:153, input.repeats is used without validation, and the loop at lines 155-156 accepts any JavaScript number satisfying repeat < repeats. Further, errors from each task call are converted into null repeat results by scoreRepeat at lines 140-142, so task failures cannot terminate an Infinity loop.
   • Fix
     • Before constructing perRepeat, require repeats to be a finite positive integer (for example, reject unless Number.isFinite(repeats), Number.isInteger(repeats), and repeats >= 1); throw a RangeError otherwise. Add focused regression tests for Infinity, zero, negative, and fractional repeat values.

   _{Ran code and verified through T-Rex}

Prompt To Fix All With AI

### Issue 1
packages/workflows/builtin/progress-scoring.ts:135
**Future workflow observations leak into checkpoint scores**

`scoreRepeat` passes the full `input.steps` array to the prompt builder for every requested checkpoint. When scoring checkpoint 2, the executed prompt included later acceptance-suite and deployment outcomes from steps 3 and 4. The checkpoint list labels which score to return but does not limit the evidence available to the verifier, so later outcomes can alter the score for an earlier state and distort the resulting trend. Construct checkpoint-bounded observation prefixes, or separate structurally bounded prompt sections, so later steps cannot influence earlier checkpoint scores.

### Issue 2
packages/workflows/builtin/progress-scoring.ts:153-156
**Invalid repeat counts can run scoring indefinitely**

`repeats` is used directly as the loop bound without requiring a finite positive integer. The executed harness showed that `Infinity` keeps scheduling scoring calls until externally terminated; task errors do not end the loop because each repeat converts them into null results. Zero and negative values return null scores without scoring, while `2.5` executes three repeats. Reject non-finite, non-integer, and non-positive repeat counts before entering the loop.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "merge origin/main" | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

[greptile-apps]

Future workflow observations leak into checkpoint scores

scoreRepeat passes the full input.steps array to the prompt builder for every requested checkpoint. When scoring checkpoint 2, the executed prompt included later acceptance-suite and deployment outcomes from steps 3 and 4. The checkpoint list labels which score to return but does not limit the evidence available to the verifier, so later outcomes can alter the score for an earlier state and distort the resulting trend. Construct checkpoint-bounded observation prefixes, or separate structurally bounded prompt sections, so later steps cannot influence earlier checkpoint scores.

Artifacts

Interior checkpoint prompt exposure harness source

• Vitest source used to compare a checkpoint-bounded prompt with the prompt captured from the real score_progress task call; it asserts the relevant future-data presence and absence conditions.

Checkpoint-bounded baseline prompt execution

• The baseline harness run exited 0 and printed a checkpoint-2 prompt containing only steps 1 and 2, with no future acceptance or deployment outcome; this establishes the intended bounded scope.

Production score_progress prompt execution with future data

• The production-path harness run exited 0 and printed the captured task prompt for checkpoint 2, which includes future steps 3 and 4; this confirms the data leak.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/workflows/builtin/progress-scoring.ts
Line: 135

Comment:
**Future workflow observations leak into checkpoint scores**

`scoreRepeat` passes the full `input.steps` array to the prompt builder for every requested checkpoint. When scoring checkpoint 2, the executed prompt included later acceptance-suite and deployment outcomes from steps 3 and 4. The checkpoint list labels which score to return but does not limit the evidence available to the verifier, so later outcomes can alter the score for an earlier state and distort the resulting trend. Construct checkpoint-bounded observation prefixes, or separate structurally bounded prompt sections, so later steps cannot influence earlier checkpoint scores.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Invalid repeat counts can run scoring indefinitely

repeats is used directly as the loop bound without requiring a finite positive integer. The executed harness showed that Infinity keeps scheduling scoring calls until externally terminated; task errors do not end the loop because each repeat converts them into null results. Zero and negative values return null scores without scoring, while 2.5 executes three repeats. Reject non-finite, non-integer, and non-positive repeat counts before entering the loop.

Artifacts

Mocked score_progress repeat-validation execution harness

• Authored Bun/TypeScript harness invokes score_progress with zero, negative, fractional, and Infinity repeats using a mocked task, with a bounded failure mode for the Infinity demonstration.

Finite invalid repeat execution output

• Captured command output shows zero and negative repeats made no task calls and a fractional repeat of 2.5 made three calls, demonstrating unchecked repeat semantics.

Infinity repeat bounded execution output

• Captured timeout-wrapped execution reached the mock task's fifth call and was terminated with exit code 124, demonstrating the unbounded Infinity loop.

Existing progress-scoring unit-test output

• Captured existing focused unit-test run passed 10 tests, showing the repository's current coverage does not catch invalid repeat values.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/workflows/builtin/progress-scoring.ts
Line: 153-156

Comment:
**Invalid repeat counts can run scoring indefinitely**

`repeats` is used directly as the loop bound without requiring a finite positive integer. The executed harness showed that `Infinity` keeps scheduling scoring calls until externally terminated; task errors do not end the loop because each repeat converts them into null results. Zero and negative values return null scores without scoring, while `2.5` executes three repeats. Reject non-finite, non-integer, and non-positive repeat counts before entering the loop.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.