Retries are not designed yet: model limits must be known, counted, and waited out

[EdmondDantes]

What was measured

Four concurrent runs against one OpenAI org (30k TPM for gpt-4o). A run died with:

Rate limit reached for gpt-4o … Used 29055, Requested 4335. Please try again in 6.78s.

BackoffAgentRetryPolicy is deliberately simple: 4 attempts, exponential 0.5s→1s→2s, honor
retryAfterMs when the error carries one. It never carried one here: AgentErrors::retryAfterMs()
reads only the Retry-After header (integer seconds), which Anthropic sends and OpenAI's TPM 429
does not — OpenAI states the wait in x-ratelimit-reset-tokens ("6.78s", "1m30s" formats) and in
the message text. So the policy waited ~3.5s total against a ~7s window, every attempt re-collided,
and the run failed with the correct wait time sitting unread in the response.

Bouncing off 429s is also the only mechanism today: nothing counts what we send, so two hungry
runs saturate the window continuously and each one's independent retries re-collide by design.

What "doing retries properly" has to decide

1. The wait is data, not a guess. Provider-specific extraction belongs in AgentErrors (the
normalization point): retry-after (seconds and HTTP-date), OpenAI x-ratelimit-reset-tokens /
x-ratelimit-reset-requests, message text as last resort. The policy stays provider-agnostic and
just receives retryAfterMs.

2. Model limits are config, counting is ours. Every reply already reports usage (it is in the
trace), and per-model config exists (config/model-pricing.php). Add per-model limits (TPM, RPM —
they differ per model and per org tier) and keep a sliding-window account of tokens/requests sent
per (provider, model). Then the client knows the window is full before the provider says so —
429 becomes the corrective signal, not the scheduler. Open question: estimating a request's token
cost before sending (prompt is countable, completion is not — reserve max_tokens? a running
average? overshoot and let 429 correct?).

3. Waiting strategy — who waits, how long, in what order.

• Budget in wall-clock, not attempts: a TPM window is time-based, so the bound should be a total
  wait budget (the existing 60s ceiling generalized), with attempts unlimited inside it.
• One shared gate per (provider, model) in the process: concurrent coroutines queue on it instead
  of racing. FIFO to avoid starvation; jitter on wake.
• Patience is the caller's, not the policy's: a detached run can wait a minute, claw -i inline
  triage should give up early and record analysis-failed. The request should carry the patience,
  the policy should respect it.
• The wait must be visible: a trace record per wait (who, why, how long), or a slow run reads as a
  hung one — this session's failures were only diagnosable by reading raw SQL.

4. What a failure means afterwards. A run that exhausts its wait budget should fail the way
strategy outcomes already do (outcome_reason carried the 429 text correctly today) — and a
re-triage of that ticket should be able to see "rate limit" and choose to just retry later rather
than escalate the strategy.

Constraints

• AgentRetryPolicyInterface is already separated from the send loop (AbstractAgent) — the seam
  exists; this is about what flows through it.
• Per the project rule, prefer making the wrong thing inexpressible: a shared per-model gate that
  every agent call passes through beats a convention that callers should throttle themselves.

Acceptance sketch

• The measured case — OpenAI 429, no Retry-After header, "try again in 6.78s" in body — succeeds
  on retry.
• Two concurrent runs over one 30k TPM limit both complete: slower, never dead.
• A trace of a throttled run shows where the time went.

[EdmondDantes]

One more constraint, and it changes the shape: the account must live in shared memory across processes, not in the process.

The org limit is one budget spent by everyone at once: the dashboard server's detached runs, every claw -i triage, every claw run — separate processes. An in-process gate only orders coroutines inside one of them; two processes would each see a half-empty window and jointly blow it, which is exactly what happened live (CLI triages and server runs collided on the same 30k TPM).

Where the shared account lives:

• SQLite is the honest choice here — the project already runs on it and the hard lessons (WAL once, BEGIN IMMEDIATE, busy timeout) are already paid for. But the scope is new: the limit is per (provider, model) org-wide, not per project, so it cannot live in a project db — it needs an app-home-level file (beside the project dbs, not inside one).
• Alternatives don't survive contact: APCu is per-process in CLI; raw shm has no tooling and no crash recovery; routing every send through the dashboard server as the single accountant breaks the moment the server is down while a CLI run works.

Mechanics sketch: minute-bucket counters, one row per (provider, model, minute), atomic UPSERT increment — cheap, contention-friendly, self-expiring; prune on read. The writes are tiny and WAL absorbs them.

The interesting open question is reservation: counting after the reply is too late — two processes both "fit" into the same remaining window before either has settled. So a sender reserves before the call (prompt tokens are countable; completion is not — reserve max_tokens or a running average) and settles with actuals after. Reservation over-blocks when max_tokens is generous; settling corrects the account but not the wait another process already served. That trade-off — optimistic (429s correct us) vs pessimistic (reserve up front) — is the core design decision, and it may reasonably be optimistic-first with reservation added only if collisions stay frequent.

Failure mode when the shared file is unavailable: fall back to local-only accounting and say so in the trace — degraded coordination, never a refusal to run.