Skip to content

feat(workflows): budget duration enforcement + soft landing (B2, #2212) - #2536

Merged
flora131 merged 51 commits into
mainfrom
budgets/duration-enforcement
Aug 19, 2026
Merged

feat(workflows): budget duration enforcement + soft landing (B2, #2212)#2536
flora131 merged 51 commits into
mainfrom
budgets/duration-enforcement

Conversation

@flora131

@flora131 flora131 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Replacement for #2527 after the stacked-PR base branch was deleted. Same branch, now targeting main (B1 already merged).


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Greptile Summary

This PR adds configurable workflow duration budgets, warning notices, budget-exhaustion handling, and persisted restoration. A reproduced notification-delivery issue remains in the budget-warning path: after a warning delivery fails, an unrelated workflow update can send the same warning again before its scheduled retry, creating duplicate warning cards and reconciliation work.

Confidence Score: 4/5

The change is not ready to merge until failed budget-warning deliveries remain deduplicated during their retry window.

The duplicate-delivery path was reproduced using the real lifecycle notification installation, store invalidation mechanism, rejected delivery handling, and retry scheduling.

Files Needing Attention: packages/workflows/src/extension/lifecycle-notifications.ts needs a retryable-delivery check in the budget-warning admission guard, plus regression coverage for an invalidation occurring before retry.

T-Rex T-Rex Logs

What T-Rex did

  • T-Rex produced a finding-comment-proof for a posted P1 finding.
  • T-Rex produced a second finding-comment-proof for another posted P1 finding.
  • T-Rex performed a general-contract-validation-proof to verify the budget-warning retryable guard scenario, including the before-test TypeScript repro and the after-test cleanup.

View all artifacts

T-Rex Ran code and verified through T-Rex

Comments Outside Diff (1)

  1. General comment

    P1 Failed budget warnings can be delivered twice before their scheduled retry

    • Bug
      • After sendMessage rejects, the budget-warning key is added to retryableTerminalRuns. A subsequent unrelated store invalidation re-inspects the unchanged warning and sends it again immediately, resulting in duplicate warning cards/reconciliation turns and independent retry timers.
    • Cause
      • emitBudgetWarningOnce at packages/workflows/src/extension/lifecycle-notifications.ts:318 checks only deliveredTerminalRuns and pendingTerminalRuns. It omits retryableTerminalRuns, unlike terminal and control notice guards.
    • Fix
      • Add state.retryableTerminalRuns.has(key) to the budget-warning guard, matching the terminal/control admission checks, then retain a regression test covering rejection followed by invalidation before the retry timer fires.

    T-Rex Ran code and verified through T-Rex

Prompt To Fix All With AI
### Issue 1
packages/workflows/src/extension/lifecycle-notifications.ts:318
**Budget warning retries bypass delivery deduplication**

A rejected warning delivery puts its key in `retryableTerminalRuns`, but this admission guard only checks delivered and pending keys. If an unrelated store invalidation occurs before the retry timer fires, the unchanged warning is delivered again immediately, creating duplicate warning cards, duplicate reconciliation turns, and an additional retry timer. Include `retryableTerminalRuns.has(key)` in this guard, as the terminal and control notice paths already do.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "merge origin/main" | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.

flora131 and others added 30 commits August 18, 2026 10:04
Add the duration checkpoint controller consumed at stage and tool
boundaries: elapsedRunMs semantics, warn-once, one-turn wrap-up, and the
exhaustion report carrying dimension, reading, ceiling, and frontier.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only the enforcement door may produce the status; stage, tool, and model
output cannot set or clear it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Persist and restore run budget state so a resumed run never gets fresh
meters and repeat exhaustion needs no second wrap-up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Check before stage dispatch, before a tool node, and after either
completes; never mid-stream. Child scopes soft-land their own subtree
while the parent continues, and the root wins simultaneous exhaustion.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Emit the warn-once lifecycle notice and report per-dimension reading,
ceiling, and percent in workflow status.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The interrupted edit left a truncated import block that failed to parse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The existing warn-once test injected its own onWarning spy, leaving the
shipped store.recordNotice wiring in run.ts unexercised. Drive six stage
boundaries past the threshold and assert a single deduped warning notice
that does not stop the run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shipped effective config must carry the documented 80 percent warning
threshold so definition and run budgets warn through the extension path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only an already-live stage turn may consume the one-time wrap-up allowance.
Tool, child-workflow, startup, and before-dispatch exhaustion now stop with
no new stage and no false wrap-up state; the system rechecks after the
workflow body so caught budget errors cannot clear the blocked outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A workflow body cannot swallow the system-owned exhaustion stop. Resume
startup reports the actual failed or resumed stage when available, while
post-body checkpoints keep caught budget errors on the blocked rail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Describe the 80 percent default, live-turn-only wrap-ups, no-live-turn
boundary behavior, and raising a ceiling through workflow resume.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The budget warning was written only to the store notice array, which no
renderer reads, so R9 never reached a user.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ownership keyed on wrap-up delivery left a boundary-only exhaustion
reporting running forever. Key it on an engine-set marker instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A display name was persisted where a stage id belongs, so resuming a
boundary-only exhaustion was rejected as insufficient_state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add durable regressions for startup-checkpoint blocked status, the full
same-budget resume then raised-budget chain, and failedStageId identity.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Remove non-semantic whitespace and use the shared WorkflowBudget type in
the durable-resume interface. Preserve every B2 repair while recording
that the remaining source cap overage cannot be safely golfed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Teach the shared resume gate about the engine-owned budget stop instead
of relying on the extension-only escape hatch. Exercise the shipped
workflow resume action with a raised budget.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…letion

A post-body duration check was newly exhausting successful workflows and
discarding their outputs. Preserve the caught-error guarantee by
rethrowing only a system-owned stop already raised at an earlier boundary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Persisted checkpoint readings are still used for ended runs, while live
runs compute their current elapsed duration and percentage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Capture the wrap-up turn's own model usage without tree-wide metering, and
consume the one-turn allowance only when a summary is actually delivered.
A boundary abort may still own the stop without burning an undelivered
wrap-up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
flora131 and others added 21 commits August 18, 2026 16:40
Drive the agent-facing resume action through createExtensionRuntime and
assert that a raised-budget continuation is launched and completes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A blind 450ms sleep hid a two-phase settle and made the assertion depend
on an idle machine. Poll the observable end state instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shipped config always materializes a budget object, so testing for
presence left the scan running for every unbudgeted same-id dispatch.
Test the ceiling instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A child ceiling bounds that child run's own boundaries, not its subtree;
only the root scope reaches descendants. Also restore the code spans the
reserved-status paragraph lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
bastani 🟢 Ready View Preview Aug 19, 2026, 4:58 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@flora131
flora131 merged commit 2252fd0 into main Aug 19, 2026
14 checks passed
const warning = run.budgetState?.warning;
if (warning === undefined) return;
const key = budgetWarningKey(run);
if (state.deliveredTerminalRuns.has(key) || state.pendingTerminalRuns.has(key)) return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Budget warning retries bypass delivery deduplication

A rejected warning delivery puts its key in retryableTerminalRuns, but this admission guard only checks delivered and pending keys. If an unrelated store invalidation occurs before the retry timer fires, the unchanged warning is delivered again immediately, creating duplicate warning cards, duplicate reconciliation turns, and an additional retry timer. Include retryableTerminalRuns.has(key) in this guard, as the terminal and control notice paths already do.

Artifacts

Executable duplicate budget-warning delivery repro source

  • A narrow Vitest test uses the real store and lifecycle installation to reject delivery, invalidate the snapshot, and assert duplicate admission before retry; the takeaway is that the suspected path is executable.

Passing output for duplicate budget-warning delivery repro

  • The real Vitest command completed with one passing test and exit code 0 after asserting the retryable state, second send, and second retry timer; the takeaway is that duplicate delivery occurs.

Temporary test cleanup and artifact hash output

  • The cleanup command confirmed the temporary test no longer exists under test/unit and recorded hashes for the saved proof files; the takeaway is that product and permanent test sources were not modified.

View artifacts

T-Rex Ran code and verified through T-Rex

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/workflows/src/extension/lifecycle-notifications.ts
Line: 318

Comment:
**Budget warning retries bypass delivery deduplication**

A rejected warning delivery puts its key in `retryableTerminalRuns`, but this admission guard only checks delivered and pending keys. If an unrelated store invalidation occurs before the retry timer fires, the unchanged warning is delivered again immediately, creating duplicate warning cards, duplicate reconciliation turns, and an additional retry timer. Include `retryableTerminalRuns.has(key)` in this guard, as the terminal and control notice paths already do.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant