feat(workflows): budget duration enforcement + soft landing (B2, #2212) - #2536
Conversation
Add the duration checkpoint controller consumed at stage and tool boundaries: elapsedRunMs semantics, warn-once, one-turn wrap-up, and the exhaustion report carrying dimension, reading, ceiling, and frontier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only the enforcement door may produce the status; stage, tool, and model output cannot set or clear it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Persist and restore run budget state so a resumed run never gets fresh meters and repeat exhaustion needs no second wrap-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Check before stage dispatch, before a tool node, and after either completes; never mid-stream. Child scopes soft-land their own subtree while the parent continues, and the root wins simultaneous exhaustion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Emit the warn-once lifecycle notice and report per-dimension reading, ceiling, and percent in workflow status. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The interrupted edit left a truncated import block that failed to parse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The existing warn-once test injected its own onWarning spy, leaving the shipped store.recordNotice wiring in run.ts unexercised. Drive six stage boundaries past the threshold and assert a single deduped warning notice that does not stop the run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shipped effective config must carry the documented 80 percent warning threshold so definition and run budgets warn through the extension path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only an already-live stage turn may consume the one-time wrap-up allowance. Tool, child-workflow, startup, and before-dispatch exhaustion now stop with no new stage and no false wrap-up state; the system rechecks after the workflow body so caught budget errors cannot clear the blocked outcome. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A workflow body cannot swallow the system-owned exhaustion stop. Resume startup reports the actual failed or resumed stage when available, while post-body checkpoints keep caught budget errors on the blocked rail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Describe the 80 percent default, live-turn-only wrap-ups, no-live-turn boundary behavior, and raising a ceiling through workflow resume. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The budget warning was written only to the store notice array, which no renderer reads, so R9 never reached a user. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ownership keyed on wrap-up delivery left a boundary-only exhaustion reporting running forever. Key it on an engine-set marker instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A display name was persisted where a stage id belongs, so resuming a boundary-only exhaustion was rejected as insufficient_state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add durable regressions for startup-checkpoint blocked status, the full same-budget resume then raised-budget chain, and failedStageId identity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Remove non-semantic whitespace and use the shared WorkflowBudget type in the durable-resume interface. Preserve every B2 repair while recording that the remaining source cap overage cannot be safely golfed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Teach the shared resume gate about the engine-owned budget stop instead of relying on the extension-only escape hatch. Exercise the shipped workflow resume action with a raised budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…letion A post-body duration check was newly exhausting successful workflows and discarding their outputs. Preserve the caught-error guarantee by rethrowing only a system-owned stop already raised at an earlier boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Persisted checkpoint readings are still used for ended runs, while live runs compute their current elapsed duration and percentage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Capture the wrap-up turn's own model usage without tree-wide metering, and consume the one-turn allowance only when a summary is actually delivered. A boundary abort may still own the stop without burning an undelivered wrap-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drive the agent-facing resume action through createExtensionRuntime and assert that a raised-budget continuation is launched and completes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A blind 450ms sleep hid a two-phase settle and made the assertion depend on an idle machine. Poll the observable end state instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shipped config always materializes a budget object, so testing for presence left the scan running for every unbudgeted same-id dispatch. Test the ceiling instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A child ceiling bounds that child run's own boundaries, not its subtree; only the root scope reaches descendants. Also restore the code spans the reserved-status paragraph lost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
| const warning = run.budgetState?.warning; | ||
| if (warning === undefined) return; | ||
| const key = budgetWarningKey(run); | ||
| if (state.deliveredTerminalRuns.has(key) || state.pendingTerminalRuns.has(key)) return; |
There was a problem hiding this comment.
Budget warning retries bypass delivery deduplication
A rejected warning delivery puts its key in retryableTerminalRuns, but this admission guard only checks delivered and pending keys. If an unrelated store invalidation occurs before the retry timer fires, the unchanged warning is delivered again immediately, creating duplicate warning cards, duplicate reconciliation turns, and an additional retry timer. Include retryableTerminalRuns.has(key) in this guard, as the terminal and control notice paths already do.
Artifacts
Executable duplicate budget-warning delivery repro source
- A narrow Vitest test uses the real store and lifecycle installation to reject delivery, invalidate the snapshot, and assert duplicate admission before retry; the takeaway is that the suspected path is executable.
Passing output for duplicate budget-warning delivery repro
- The real Vitest command completed with one passing test and exit code 0 after asserting the retryable state, second send, and second retry timer; the takeaway is that duplicate delivery occurs.
Temporary test cleanup and artifact hash output
- The cleanup command confirmed the temporary test no longer exists under test/unit and recorded hashes for the saved proof files; the takeaway is that product and permanent test sources were not modified.
Ran code and verified through T-Rex
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/workflows/src/extension/lifecycle-notifications.ts
Line: 318
Comment:
**Budget warning retries bypass delivery deduplication**
A rejected warning delivery puts its key in `retryableTerminalRuns`, but this admission guard only checks delivered and pending keys. If an unrelated store invalidation occurs before the retry timer fires, the unchanged warning is delivered again immediately, creating duplicate warning cards, duplicate reconciliation turns, and an additional retry timer. Include `retryableTerminalRuns.has(key)` in this guard, as the terminal and control notice paths already do.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Replacement for #2527 after the stacked-PR base branch was deleted. Same branch, now targeting main (B1 already merged).
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Greptile Summary
This PR adds configurable workflow duration budgets, warning notices, budget-exhaustion handling, and persisted restoration. A reproduced notification-delivery issue remains in the budget-warning path: after a warning delivery fails, an unrelated workflow update can send the same warning again before its scheduled retry, creating duplicate warning cards and reconciliation work.
Confidence Score: 4/5
The change is not ready to merge until failed budget-warning deliveries remain deduplicated during their retry window.
The duplicate-delivery path was reproduced using the real lifecycle notification installation, store invalidation mechanism, rejected delivery handling, and retry scheduling.
Files Needing Attention: packages/workflows/src/extension/lifecycle-notifications.ts needs a retryable-delivery check in the budget-warning admission guard, plus regression coverage for an invalidation occurring before retry.
What T-Rex did
Comments Outside Diff (1)
General comment
sendMessagerejects, the budget-warning key is added toretryableTerminalRuns. A subsequent unrelated store invalidation re-inspects the unchanged warning and sends it again immediately, resulting in duplicate warning cards/reconciliation turns and independent retry timers.emitBudgetWarningOnceatpackages/workflows/src/extension/lifecycle-notifications.ts:318checks onlydeliveredTerminalRunsandpendingTerminalRuns. It omitsretryableTerminalRuns, unlike terminal and control notice guards.state.retryableTerminalRuns.has(key)to the budget-warning guard, matching the terminal/control admission checks, then retain a regression test covering rejection followed by invalidation before the retry timer fires.Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "merge origin/main" | Re-trigger Greptile