diff --git a/devlog/_plan/260908_bug6_manual_stack/000_plan.md b/devlog/_plan/260908_bug6_manual_stack/000_plan.md index c27b042af9..972d5f3445 100644 --- a/devlog/_plan/260908_bug6_manual_stack/000_plan.md +++ b/devlog/_plan/260908_bug6_manual_stack/000_plan.md @@ -6,7 +6,7 @@ - Trigger: owner request on 2026-09-08 to use one stack, repeated PABCD, Astra high delegates, no local suites, no-verify pushes, and merge through dev. - Goal: Go/xAI child-result compatibility, separated V2 guidance and server-owned presets, and correctly scoped reset-credit recovery. - Non-goals: releases, main/preview, account changes, real credit consumption, unrelated cleanup, native GitHub stacks, local product tests/install/typecheck/build. -- Verifier: independent source audits and GitHub `ci.yml` at each candidate head; final dispatch `lane=all`. Docs-only verification checks numbered artifacts and whitespace without running product code. Every activation fixture and observable result is specified in the phase designs. +- Verifier: independent source audits and GitHub `ci.yml` at each candidate head; final dispatch `lane=all`. Roadmap-only verification checks numbered artifacts and whitespace without running product code. Public docs-site changes additionally require remote `cd docs-site && bun install --frozen-lockfile && bun run build` success; artifact/whitespace checks alone do not certify documentation builds. Every activation fixture and observable result is specified in the phase designs. - Stop: every named item has a fresh terminal disposition, all nonempty layers have landed through reviewed PRs, final hosted gates pass, and fetched ancestry plus landed-tree comparison prove integration. - Artifacts: this numbered unit, ignored `.tmp/bug6-01a07e9d/`, and session-bound `.codexclaw` ledger/receipts. New unpublished security analysis stays in scratch only. - Outcomes: DONE requires all evidence; NOOP requires proof the full named contract already landed; unresolved work remains pending; genuine external blockers are reported without inventing proof. @@ -41,7 +41,11 @@ Before each merge refresh head/base, membership, reviews, required checks and ac ## Continuity ledger -- wp0 P: live source intake and complete decade designs in progress; no product changes. -- wp0 A: independent Astra high reviewer returned PASS, zero blockers. Full source appendices remain in ignored scratch. An absent REST stack field means unknown membership, not proven absence; inspect the stacks endpoint before delivery. +- wp0 P completed: live source intake and all eight decade designs were prepared; no product changes. +- wp0 A: independent Astra high reviewer returned PASS, zero blockers. Full source appendices remain in ignored scratch. The final integration refresh returned an empty stacks response for bottom PR #3986; refresh every PR before delivery. A missing field or failed request alone is not evidence of absence. - wp0 B/C handoff: all eight numbered roadmap documents are complete. Structural validation passed with 30 pre-existing user files preserved. Next cycle is wp1 Go residual implementation. Candidate cycles c1–c6 require their scoped audited delta and matching-head PR CI; c7 retains all six terminal dispositions and final integration proof. - Remote documentation verification uses isolated `macmini-cf` scratch, not the deploy-docs workflow. Existing Node 24.20.0 is available under the remote user's nvm tree; select the repository-pinned Bun in that scratch environment and record actual versions. No live service or account state is touched. + +## Integration entry + +All six source work-phases are verified: newPR3986/3991/3992/3993/4002 candidates passed their exact-head PRCI and source audits; #3965 independently landed and is a verified NOOP for a new PR. The actual product stack is those five newPRs. Previous wp6 D certifies6904ecd9c with CI34188893148. Final full-matrix integration and source closeout remain wp7. diff --git a/devlog/_plan/260908_bug6_manual_stack/050_credit_alias.md b/devlog/_plan/260908_bug6_manual_stack/050_credit_alias.md index bca638a08b..edbafbe6b8 100644 --- a/devlog/_plan/260908_bug6_manual_stack/050_credit_alias.md +++ b/devlog/_plan/260908_bug6_manual_stack/050_credit_alias.md @@ -12,6 +12,16 @@ Retain `Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>` and o ## Verification -The public three-file patch is the diff authority: https://github.com/lidge-jun/opencodex/pull/3965/files . Each negative fixture begins pending, so it observes the changed failure-settlement path instead of rechecking an already ambiguous row. Existing no-operationId and ordinary terminal paths remain regression controls. Hosted CI runs the auth and ledger suites; local tests/typecheck/build/install are NOT RUN by owner instruction. A source/security reviewer verifies the exact carried head before merge. Existing source-PR CI failure is historical and must not be described as passing. +The final immutable three-file source range is `abb46a1599ec0d0bbfbe03905114178df92e67f5..62412d38606851f7cace76360f3c5737db9cae20`; the landed equivalent is `abb46a1599ec0d0bbfbe03905114178df92e67f5..402be7c1f88283eb8465c3aec8437ccecd2542ec`. These final pins supersede the initial intake head above. A live source-head mismatch requires renewed comparison before carry; the mutable PR files page is navigation, not patch authority. Each negative fixture begins pending, so it observes the changed failure-settlement path instead of rechecking an already ambiguous row. Existing no-operationId and ordinary terminal paths remain regression controls. Hosted CI runs the auth and ledger suites; local tests/typecheck/build/install are NOT RUN by owner instruction. A source/security reviewer verifies the exact carried head before merge. Existing source-PR CI failure is historical and must not be described as passing. All additional unpublished security analysis lives in ignored `.tmp/bug6-01a07e9d/credit-plan.md` and later audit artifacts. It must not be copied into this public unit. + +## wp5 P refresh + +Previous wp4 D certified PR3993 head727683f44 with CI34185870948, source/security/GUI audits, QA and remote docs. Proceed canonical alias carry. Live refresh supersedes the prepared pin: #3965 merged at03:17:07Z with head62412d386 and merge402be7c1f; prepared bdb9f4bfe+9eb44cfb4 passed two source/security audits. All three target files on actual predecessor727683f44 equal preparedbase d1f61e933; intervening V2/GUI/test-harness deltas do not modify this owner. Keep exact3filecarry and original attribution. No real credentials/resetcredits and no localproductcommands. Actual adoption equality and hostedCI remain required. + +## Verified landed disposition + +Current origin/dev402be7c1f contains the #3965 merge402be7c1f. Its three target files exactly equal prepared candidate9eb44cfb4 (git diff exit0); PR CI34181771859 passed at exacthead62412d386, including Linux4/macOS2/gates. Thus the source item is already landed, not a new product fix. The initial A narrative retained the old OPEN assumption; this fresh source/API evidence corrects it before B. + +NOOP for a new PR. Adopt the identical two contribution commits locally only as the prerequisite for wp6; preserve provenance and contributor credit. The final new recovery PR targets existing layer4 and explains the already-landed alias dependency in its base-relative diff. No duplicate fifth PR is created and no original branch is rewritten. c5 closes on live merged-state/CI/ancestry/file equality evidence; wp6 and final cumulative integration still run their full gates. The single product stack has five new PRs plus this independently landed sixth source item. diff --git a/devlog/_plan/260908_bug6_manual_stack/060_credit_recovery.md b/devlog/_plan/260908_bug6_manual_stack/060_credit_recovery.md index 9f7f1cfd23..244cfe8693 100644 --- a/devlog/_plan/260908_bug6_manual_stack/060_credit_recovery.md +++ b/devlog/_plan/260908_bug6_manual_stack/060_credit_recovery.md @@ -17,3 +17,25 @@ The complete before/after design, exact current source anchors, threat model, re Only the matching account's eligible pre-existing cooldown may be recovered after confirmed reset and fresh supporting evidence. Ordinary successful requests, uncertain results and replay do not gain broader recovery authority. Existing unrelated scopes and caller selections remain intact. The private appendix enumerates the full mocked positive/negative matrix and claim cleanup requirements. Run no local product commands. Hosted CI must execute the affected auth, cooldown, quota and provenance suites; independent security review remains required. PR #3848 overlaps the flight interface: refresh before B and integrate any landed change without absorbing its unrelated registration behavior. New code belongs to this owned stack; do not modify other open PRs. Record privacy-safe outcome evidence here only after publication. + +## wp6 P refresh + +Previous wp5 D verified #3965 already landed in dev402be7c1f, exacthead62412d386CIpassed, and locally adopted identical prerequisite (926b3719f); no duplicatePR5. This recovery PR targets existing layer4 #3993 and identifies the already-landed alias prerequisite in its relative diff. Source3973 remains open. Fresh inventory found overlapping contributorPR3995 (e172453052bf7bbc4a0ae5aa24592982c0c64b15) and independent fallbackPR3997; the latter resolves3996 and is outside this goal. The earlier no-overlap narrative was incorrect and is superseded before B. + +Prepared recovery e6e081c09 plus repair a87a3f624 passed independent security and behavior audits. The private repair synthesis and updated handoff under ignored scratch resolve main-publication ordering and positive refresh provenance; never copy security working analysis into this public unit. All six target preimages on actual predecessor926b3719f equal auditedbase9eb44cfb4. Revalidate the unchanged candidate across intervening V2/GUI/testharness context, then adopt. All mocked regressions, current-head hostedCI, privacy, finalfullcohort proof and source-item closeout remain required. No localproductcommands or realcreditactions. + +## Concurrent source reconciliation in P/A + +Review new3995 against the prepared candidate before adoption. Preserve originalcontributor credit and include its useful language/CLI docs or regression cases when source comparison warrants. Existing prepared recovery provides bounded claims/publication/provenance invariants; no competing implementation is accepted solely from prior green claims. Comparative security/behavior source reviews are in progress, all notes remain scratch. No productdelta forwp6 has been adopted yet. + +## Consolidated source decision + +Retain audited recovery e6e081c09+a87a3f624 and consolidate contributorPR3995 rather than creating competing deliveries. Comparative security review retains its PASS; detailed algorithm findings remain private in credit3995Comparison.md. Keep pause/reauth eligibility and existing background lease ownership conservative and document that recovery can remain pending under those conditions. #3997/#3996 stays outside scope. + +Additional MODIFY paths: docs-site/src/content/docs/ko/reference/management-api.md and docs-site/src/content/docs/reference/cli/providers-accounts.md, carrying the matching contributor guidance with parity to the final conditional recovery contract. This expands six unique files to eight. Do not duplicate the fuller English API paragraph. Adapt PR3995 tests into the existing auth-api test: two cold-main reset/already_redeemed cases without prior listing/reconciliation, bogus consume99 versus freshWHAM1; strengthen the existing saturation test with pre-existing shared cooldown, one consume, zero usage and retainedcooldown; adapt the two-old-flight/current-generation convergence scenario to assert fresh fourthdispatch completes before oldresponses, then oldresponses cannotoverwritefreshquota or recoveredcooldown. Preserve and await every deferred fixture cleanup. No new testfile, account-store schema or CLI runtime change. + +Carry sourcee172453052 with Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com> in the adaptation commit and final PR body; describe exactly which tests/docs are adopted. B includes local candidate adoption and these bounded test/doc additions; independent interdiff review and exacthead hostedCI remain mandatory. No local tests/install/typecheck/build or realcredits. + +## C fixture foldback + +CI34188041321 caught a shared401-recovery budget leaking between fake-home cases: prior manual-a selfrefresh spends generation2, and the next case creates a different generation2 in a newhome but doesnotreset the module budget. The early spent-budget refusal prevents the intended external-replacement replay. MODIFY only the existing auth-api test: import/call resetQuotaRecoveryForTests in beforeEach/afterEach, assert empty budget at the negative-case start, observe real force-refresh provenance, and KEEP expectedfreshremaining2, replay URLs and cooldown-preservation assertions. No production relaxation. Also use existing watchdogMs(10000) and60souter ceiling for the new convergence fixture; its current run passed, so this is convention/contended-runner safety, not increasing a failing behavioral timeout. Source/interdiff review and newexactheadCI are required. diff --git a/devlog/_plan/260908_bug6_manual_stack/070_integration.md b/devlog/_plan/260908_bug6_manual_stack/070_integration.md index 2b546b12cd..ae22513448 100644 --- a/devlog/_plan/260908_bug6_manual_stack/070_integration.md +++ b/devlog/_plan/260908_bug6_manual_stack/070_integration.md @@ -17,7 +17,7 @@ Depends on wp1–wp6. The owner explicitly requested a single manual branch chai 3. Read each PR's current head/base and native `stack` field. A native membership conflict is inspected without mutating membership. Our newly created ordinary PRs must remain manual. 4. Inspect `gh pr checks` and matching workflow runs. Before landing obtain final candidate `ci.yml` `workflow_dispatch` with `lane=all` as well as required PR checks. Bind conclusions to `head_sha`, event and run attempt. Retry failed jobs only after investigating the actual failure and ensuring it does not hide a product regression. 5. For the preset UI, download the hosted `dashboard-preview-*` artifact from the verified head. Verify `build-commit.txt` and `build-gui-tree.txt`; serve the prebuilt bundle with synthetic API fixtures on a disposable loopback port; observe preset activation/restoration and server-switch behavior in a browser; capture/read the screenshot. No local product compilation. Existing browser driver only, no installation. -6. Refresh MAINTAINERS.md, live actor permission, reviewer objections and security evidence. Record maintainer integration in the owned PR body. Land only the bottom PR with `--match-head-commit`; retarget the next child to dev and verify exact resulting integration head/CI. Never merge an upper PR into its parent branch as if that landed it in dev. +6. Refresh MAINTAINERS.md, live actor permission, reviewer objections and security evidence. Record maintainer integration in the owned PR body. The repository deletes merged head branches: inspect direct children and retarget our next child to dev immediately before merging its parent, so automatic deletion cannot close it. Land only the bottom PR with `--match-head-commit`; verify the resulting integration head and CI evidence before advancing. Never merge an upper PR into its parent branch as if that landed it in dev. Do not change repository settings or unrelated children. 7. Fetch dev after each merge and prove the merged commit is an ancestor. At final integration compare actual trees against the final certified candidate, including any explicitly reviewed concurrent dev changes. 8. Refresh each original item and mark closed only if its entire user-visible bug is resolved by the landed tree. Preserve unresolved residuals as open; report the exact residual rather than treating overlap as duplication. @@ -33,3 +33,27 @@ Depends on wp1–wp6. The owner explicitly requested a single manual branch chai ## Validation limits Local product tests, installs, typechecks and builds: NOT RUN by owner instruction. Hosted tests and independent source audits provide product evidence; docs-only filesystem/link/whitespace checks provide document evidence. Neither substitutes for the other. + +## wp7 P refresh + +Previous wp6 D:6904ecd9c passed CI34188893148 and source/security/interdiff audits; coldmain, busy, same-tick replacement and converged-flight regressions passed; docs425pages plus renderedlink/KOparity passed. Latestdev402be7c1f is pinned for integration. Read-only merge-tree predicts conflicts only in reference/management-api.md and codex-auth-api.test.ts because dev already contains the canonicalalias prefix. Both dev versions exactly equal our adopted alias predecessor9eb44cfb4; resolve those two to our current versions, which include that prefix plus the audited recovery. No pre-existing userfile overlaps the incomingdevdelta. + +Merge402be7c1f into the topbranch with hooksdisabled, preserving all other incomingfiles. Record exactresolvedtree, recheckcurrentdev, publishnoverify, and dispatchci.yml lane=all on that exacthead. NewPRchain is3986→3991→3992→3993→4002;3965alreadymerged. All original candidateheads retain passingPRCI; any rewrittenhead gets freshproof. Refresh each target/head/membership/review/CI before its separately authorized ordinary merge. Use merge commits to preserve ancestry/attribution, retarget each nextchild todev, and prove resolved integration content is covered by the certifiedtop. Current-headrequiredchecks and source/security duties remain separate. + +After productlanding, close only satisfied sourceitems, including superseded3995 aftercore3973lands; preserve3997. Archive this unit with an evidence-only closing PR if needed, so completedrecords do not change the tested product tree. Verify that closingdelta is documentation-only and receives its properCI; retain exact product-tree equivalence to the full-matrix candidate rather than attributing skipped productjobs to passingexecution. No release/main/preview/deployment changes. + +## Final CI repair amendment + +Full run34190287787 at f1b436324 failed Windows3/6: the first restart-help correctness test returned an unobserved exit status after its fixed10s synchronous subprocess bound. Other observed shards passed; wait for the complete run before deciding whether any additional repair is needed. The source investigation does not establish a Bun defect or a startup latency cause. + +Modify only `tests/cli/cli-restart-health.test.ts` for this repair. Replace synchronous spawning with awaited Bun.spawn, existing captureTestOutput and watchdogMs(10000). Preserve all eight command tests, arguments, private homes, output assertions and legitimate health exit1. Independently bound execution, TERM grace5s, KILL reap2s and output drain1s; clear timers and keep timeout, signal, rejected observation, incomplete output and unreaped child as failures even after eventual exit0/1. Use an outer cleanup envelope below the existing60s CI ceiling. Keep child ownership and avoid deleting an unreaped child's private home. Emit safe stage/PID/exit diagnostics without inherited environment or credentials. Add small controlled wrapper regressions for sticky timeout, incomplete output at0/1, unreaped child and spawn/observation errors; reuse capture-owner coverage for its internals. No production CLI or workflow timeout changes, assertion removal, skip, retry loop or local product execution. + +An independent plan audit precedes implementation; an independent patch audit precedes publication. Publish the repaired top head with hooks disabled and --no-verify, obtain fresh PR checks and a new full lane=all dispatch on that exact SHA, and verify the original Windows lane. Passing results establish that head's observed outcomes, not the historical root cause. The final docs build runs remotely on the integrated docs tree. Preserve failed-run evidence and all prior user files. + +## Last-layer ancestry repair + +Four lower PRs landed with exact predicted trees; current dev is `74f62f9c2914ead2fba474aa97734e322251bd46`. Candidate `f80f39d20e8395901d3b62758d118ea3a559a9f4` passed PR CI34193213502 attempt2 and full CI34193218874 attempt2 (26 named jobs and execution steps). Each investigated macOS retry was limited to its failed/cancelled job; prior failures remain recorded. + +GitHub refuses the last PR as conflicting even after its base metadata was refreshed to actual dev; ordinary GraphQL and REST merges both refused. Local merge-tree remains clean and equals candidate tree `1b3bd117702b86da5810b1ad21988a2c60df6d17`. The histories have two merge bases, `727683f44e9f1daa9b6b1e2dbf93167e4ce30cc1` and `402be7c1f88283eb8465c3aec8437ccecd2542ec`; this explains the need to simplify ancestry without claiming a proven server implementation cause. + +Merge that exact current dev into the owned top branch with hooks disabled. Before adding this record, require exact candidate-tree equality; no product files may change. Include only this integration record in the merge commit. Independently audit both parents and the exact record-only tree delta, preserve all 30 user files, push no-verify, and obtain fresh PR and full lane=all CI on the new head before merging #4002. Preserve the earlier successful candidate evidence as historical; do not label it new-head execution. All source closures and final archive duties remain pending until landing. diff --git a/devlog/_plan/260908_bug6_manual_stack/071_delivery.md b/devlog/_plan/260908_bug6_manual_stack/071_delivery.md new file mode 100644 index 0000000000..c1b8fc1ee2 --- /dev/null +++ b/devlog/_plan/260908_bug6_manual_stack/071_delivery.md @@ -0,0 +1,40 @@ +# Candidate verification and landing ledger + +## Verified candidates before final integration + +| Source | Owned PR | Candidate head | Successful PR CI | +| --- | --- | --- | --- | +| #3838 placement/stateless residual | #3986 | d1f61e933b0cde3df3862baed65546a5cf81066f | 34178540141 | +| #3907 string child-result residual | #3991 | 00eb47886690e7b24b0eed69b6d870c33ceade62 | 34180674115 | +| #3944 V2 guidance | #3992 | 3ceef0121712b290c3d4443e9fc3f0a04cecead6 | 34181398746; target check rerun 34181398713 | +| #3951 server-owned preset | #3993 | 727683f44e9f1daa9b6b1e2dbf93167e4ce30cc1 | 34185870948 | +| #3965 canonical alias | independently merged | 62412d38606851f7cace76360f3c5737db9cae20 | 34181771859; merge 402be7c1f88283eb8465c3aec8437ccecd2542ec | +| #3973 / #3995 recovery consolidation | #4002 | 6904ecd9cdbbd6b393e32f5e4c393a705eb3a0d2 | 34188893148 | + +Each new candidate received independent source review; authentication/recovery changes also received explicit source security review. C6's two initial publication/lineage findings were repaired before adoption. The failed test-budget fixture retained its expected response/replay assertions and gained proper per-test isolation; the repaired negative reached actual external-replacement provenance in hosted execution. No local product suites, install, typecheck or build were run. + +PR CI skips the full Windows and macOS-control jobs by workflow design; these rows are not evidence those jobs passed. The final pinned `lane=all` dispatch remains mandatory before landing. + +## GUI and documentation + +The preset GUI tree is `b0bc09ba867906375e52cf0180caa4ea4ea95bea`. Hosted artifact 10039810403 from run 34183701289 supplied the rendered bundle. Main and two independent reviewers inspected desktop light/dark and Korean 320/390px captures; viewport metrics also cover 768/1024px. The narrow editor layout was repaired after observing clipping. Exact custom saving, restore-without-write, clear, missing/malformed recommendations, error/retry, a distinct server recommendation, and keyboard focus were exercised against an isolated synthetic API. No real account or native configuration was used. + +Screenshot-only evidence commit: `924327cdd71a14a0aea1936e4c5e1f6b6b660438`. Its immutable images are linked in #3993. The image branch is not another product PR. GUI source and artifact bytes remain the same after the subsequent test-only correction. Owned browser/fixture ports 9239 and 18744 were verified closed. + +Documentation builds ran only in an isolated `macmini-cf` scratch directory using Bun 1.4.0 and Node 24.20.0. The latest candidate build produced 425 pages; the CLI management link and Korean reset contract were checked in generated HTML. An initial incomplete-transfer attempt is excluded from passing evidence. No documentation deployment occurred. + +## Final integration plan and pending evidence + +Pin dev `402be7c1f88283eb8465c3aec8437ccecd2542ec` and merge it into the top candidate. The two preview conflicts are duplicate canonical-alias prefixes in the English management reference and auth API tests. Dev's versions equal the already-adopted alias predecessor, so retaining the complete candidate versions preserves both that contribution and recovery additions. Other incoming dev changes remain intact. Existing user-file paths have no overlap with the incoming dev delta. + +Before each ordinary merge, refresh actor permission, head/base, native membership, reviewer objections and checks. Use merge commits and retarget children bottom-up. Final full-matrix run, actual merge SHAs, source closures, and tree/ancestry proof will be appended after those actions occur; none is claimed by this planning snapshot. + +Attribution retained: jpierrevd for #3838 intent; luvs01 for #3944/#3951/#3919/#3965 and adapted #3995 coverage/documentation. Lossy mixed-ciphertext filtering from #3838 is deliberately declined; current fail-closed behavior remains. #3997/#3996 is independent and stays outside this delivery. + +## Integrated candidate and failed full dispatch + +Integration commit `f1b436324d335a64789e7899a4ed491183a9c216` retains all incoming dev changes. The independently inspected resolution matched predicted tree `4e1a2458e243da43a32d49664364e88d3473da32` before the three delivery-record updates; all 30 pre-existing user files remained unchanged. PR CI `34190212954` passed. Full dispatch `34190287787` completed with 24 successful jobs, one failed Windows3/6 test job and a failed aggregate. The failure was the first `restart --help` test's null subprocess status at its fixed10s synchronous bound; later help cases passed. This failed run is not landing evidence. + +The final docs archive from this integrated head has SHA-256 `a51cdbd83f409472defcb7758873734edba167f116a17869ec345366e0e9063d`. Remote frozen install and build passed with 425 pages; the rendered CLI recovery link resolves to the API section, and English/Korean reset replay text is present. + +The audited repair is confined to the CLI test harness, preserving all original command assertions and private homes while making exit, termination and capture failures explicit. A fresh exact-head full dispatch remains required after the repair. Neither a historical root cause nor absence of future timing failures is claimed. diff --git a/docs-site/src/content/docs/ko/reference/management-api.md b/docs-site/src/content/docs/ko/reference/management-api.md index a9a54c69c3..086dc5aa49 100644 --- a/docs-site/src/content/docs/ko/reference/management-api.md +++ b/docs-site/src/content/docs/ko/reference/management-api.md @@ -253,6 +253,20 @@ OpenAI도 같은 규칙을 따르며, 스위치를 켠다고 별도의 922k 모 | `POST /api/codex-auth/login/cancel` | Codex 로그인 흐름을 취소합니다 | — | | `GET /api/codex-auth/login-status` | 흐름 또는 account 로그인 상태를 조회합니다. 새 계정 완료 시 복구가 필요할 때만 `catalogRefreshPending: true`를 포함합니다. | 알 수 없는 흐름은 `expired`로 보고되며, 활성 흐름이 없으면 `idle`로 보고됩니다 | +수동 소비가 `reset`으로 확인되면 같은 계정의 새 usage를 조회하여 기존 shared reset-derived +쿨다운을 즉시 복구할 수 있습니다. 복구는 조건부입니다. 계정이 일시 정지되었거나 재인증이 +필요하거나 다른 진행 중인 probe가 쿨다운을 소유하면 쿨다운은 유지됩니다. reset 이전에 시작한 +조회, 불완전하거나 소진된 usage, 신원이 바뀐 계정, 더 최근의 quota 실패로는 복구하지 않습니다. +오래된 main usage 응답은 더 최근에 반영한 관측을 덮어쓰지 않습니다. credential 갱신을 거쳤다면 +해당 인증에서 이어진 갱신인지 확인되어야 하며, 외부에서 교체된 credential은 같은 계정이어도 +복구 근거가 되지 않습니다. 명시적 `Retry-After`, Spark/Reserve 쿨다운, pause·pin·선택 +설정도 보존됩니다. `already_redeemed`와 저장된 결과 재생은 새 reset을 증명하지 않습니다. + +`reset` 또는 `already_redeemed`가 확인된 뒤 usage 조회가 실패하거나 바쁘더라도 소비 응답은 +HTTP 200과 원래 `code`를 유지합니다. 새 잔여 수를 얻지 못하면 `remaining`을 생략합니다. +이는 소비 결과의 확인이며 라우팅 가능 상태를 보장하지 않습니다. usage를 다시 조회하십시오. +usage 조회 실패를 재시도하기 위해 reset credit을 다시 소비하지 마십시오. + 새 account의 config row는 저장되었지만 credential setup을 완료하지 못하면 OAuth `login-status`는 `status: "error"`를 보고하며 `code: "codex_credential_persistence_failed"`, `accountId`, `needsReauth: true`, 필요한 경우 diff --git a/docs-site/src/content/docs/reference/cli/providers-accounts.md b/docs-site/src/content/docs/reference/cli/providers-accounts.md index 82c30a2c30..21760eef0f 100644 --- a/docs-site/src/content/docs/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/reference/cli/providers-accounts.md @@ -440,6 +440,14 @@ security find-generic-password -w openrouter | ocx account add-key openrouter -- Inspect Codex reset credits for an account. Consuming a credit is destructive and requires both `--consume` and `--yes`. +After a confirmed `reset`, fresh usage can recover the same account's eligible existing +shared reset-derived cooldown. Paused accounts, accounts needing reauthentication and +cooldowns owned by an in-flight probe remain excluded from this recovery. A failed or busy +usage refresh after confirmed consumption does not require another credit: check usage +again instead of repeating `--consume`. Consume success does not guarantee routability; +see the [management API recovery contract](/reference/management-api/#codex-authentication-delegation) +for reset/replay, freshness and scope limits. + ### `ocx account main ` Manage named native Codex main-login profiles without changing OpenCodex account-pool routing: diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index dafb51dc7a..c0dd38f1fb 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -434,6 +434,24 @@ requests with the original ID or a known alias replay the stored result without consume request. A previously unseen ID supplied after settlement starts a new explicit redemption; clients retrying an existing action should keep its ID. +After a confirmed manual `reset`, OpenCodex checks fresh usage for that same account +and can reconcile its eligible pre-existing shared reset-derived cooldown immediately. +Paused accounts, accounts requiring reauthentication and cooldowns already owned by an +in-flight probe remain excluded from this recovery; their cooldowns are retained. Usage +started before the reset, incomplete or exhausted usage, a changed account, and a newer +quota failure do not qualify. Older main-account usage responses cannot replace a newer +published observation. If usage needs credential refresh, recovery requires that refresh's +confirmed lineage; an externally replaced credential does not qualify merely because it +belongs to the same account. Explicit `Retry-After`, Spark/Reserve cooldowns, pause +settings, pins and the selected account are preserved. `already_redeemed` and durable +replay do not prove a new reset and do not gain this recovery behavior. + +A failed or busy usage refresh after a confirmed `reset` or `already_redeemed` does not +turn the completed consumption into an error: the response remains HTTP 200 with its +consume `code`, omitting `remaining` when no fresh count was obtained. This response +confirms the consume outcome, not that the account is now routable. Refresh usage to +check availability; do not consume another credit to retry a failed usage refresh. + If a new account config row is saved but credential setup cannot finish, OAuth `login-status` reports `status: "error"` with `code: "codex_credential_persistence_failed"`, `accountId`, `needsReauth: true`, and optional diff --git a/src/codex/auth-api.ts b/src/codex/auth-api.ts index 7ced31b3df..2e2a775867 100644 --- a/src/codex/auth-api.ts +++ b/src/codex/auth-api.ts @@ -40,6 +40,10 @@ import { } from "./account-priority"; import { claimDueCodexQuotaRecoveryProbes, + claimManualResetCooldowns, + settleManualResetCooldown, + type ManualResetCooldownClaim, + type ManualResetRefreshLineage, clearCodexAccountCooldown, clearThreadAccountMapForAccount, getEffectiveActiveCodexAccountId, @@ -98,6 +102,8 @@ import { getMainAccountInfoCache, getMainQuotaCredentialGeneration, isMainAccountIdentityGenerationLive, + isMainQuotaWriterLive, + type MainQuotaWriter, matchesMainQuotaCredential, observeMainQuotaCredential, setMainAccountCredentialPresence, @@ -387,6 +393,8 @@ interface ResetCreditAuth { chatgptAccountId: string; nativeMainLease?: AdmissionLease; nativeMainSharedClaimHeld?: true; + poolGeneration?: number; + mainProof?: MainResetQuotaProof; } async function withResetCreditAuth( @@ -407,10 +415,15 @@ async function withResetCreditAuth( if (!tokens) { return { ok: false, response: jsonResponse({ error: "Main Codex account not logged in" }, 401) }; } + reconcileMainCodexAccountRuntimeState(); + const physicalId = extractAccountId(tokens.id_token, tokens.access_token) ?? tokens.account_id; + const writer = physicalId === tokens.account_id + ? observeMainQuotaCredential(tokens.access_token, tokens.account_id) : undefined; return { ok: true, value: await operation({ isMain: true, + ...(writer ? { mainProof: { writer, credentialGeneration: getMainQuotaCredentialGeneration() } } : {}), accessToken: tokens.access_token, chatgptAccountId: tokens.account_id, nativeMainLease, @@ -439,6 +452,7 @@ async function withResetCreditAuth( ok: true, value: await operation({ isMain: false, + poolGeneration: cred.generation, accessToken: cred.accessToken, chatgptAccountId: cred.chatgptAccountId, }), @@ -776,8 +790,14 @@ async function readMainAuthErrorCode(resp: Response): Promise { } } +interface MainResetQuotaProof { + writer: MainQuotaWriter; + credentialGeneration: number; +} + interface MainAccountInfoFetchResult { info: MainAccountInfo; + resetRecoveryProof?: MainResetQuotaProof & { dispatchSequence: number }; /** Ephemeral result of this attempt, omitted when no WHAM request was made. */ quotaRefresh?: CodexQuotaRefreshOutcome; /** Internal dispatch fence for diagnostics only; never copied into a public DTO or cache. */ @@ -914,6 +934,7 @@ async function fetchMainAccountInfoWhileOwned( let quotaPhase: "request" | "body" | "decode" | "publish" = "request"; let quotaRefreshGeneration = captureMainAccountIdentityGeneration(); try { + const dispatchSequence = ++quotaDispatchSequence; const resp = await fetch("https://chatgpt.com/backend-api/wham/usage", { headers: { Authorization: `Bearer ${tokens.access_token}`, "ChatGPT-Account-Id": tokens.account_id }, signal: quotaSignal, @@ -923,6 +944,10 @@ async function fetchMainAccountInfoWhileOwned( const terminalAuthFailure = await isTerminalMainAuthResponse(resp, isMainAccountTokenVerifiablyLive()); const retried = await retryMainAccountInfoIfIdentityChanged(requestAccountId, retriesRemaining, nativeMainLease, explicitRefresh); if (retried) return retried; + if (dispatchSequence < mainQuotaPublishedSequence) { + return { info: getMainAccountInfoCache() ?? EMPTY_MAIN_ACCOUNT_INFO, + credentialChecked: true, hasCredential: true }; + } if (terminalAuthFailure) { // Account for this attempt's own synchronous invalidation, never prior external drift. const diagnosticStillLive = isMainAccountIdentityGenerationLive(quotaRefreshGeneration); @@ -945,6 +970,12 @@ async function fetchMainAccountInfoWhileOwned( if (data === null || typeof data !== "object" || Array.isArray(data)) { throw new Error("Invalid WHAM usage object"); } + // Check after body/retry awaits and before any cache, credits, policy or + // Reserve publication. Returning cached state supplies no fresh recovery proof. + if (dispatchSequence < mainQuotaPublishedSequence) { + return { info: getMainAccountInfoCache() ?? EMPTY_MAIN_ACCOUNT_INFO, + credentialChecked: true, hasCredential: true }; + } quotaPhase = "publish"; // A delayed response from a replaced bearer cannot revoke a newer Reserve grant, // even in the same workspace or after an A→B→A credential transition. @@ -985,6 +1016,7 @@ async function fetchMainAccountInfoWhileOwned( if (result.quota) { setAccountQuotaFromParsed(MAIN_CODEX_ACCOUNT_ID, result.quota, writerGeneration, mainQuotaWriter, policyQuota); } + mainQuotaPublishedSequence = dispatchSequence; return { info: result, quotaRefresh: { status: quota ? "ok" : "not_reported" }, @@ -992,6 +1024,11 @@ async function fetchMainAccountInfoWhileOwned( credentialChecked: true, hasCredential: true, ...(quota ? { freshQuota: quota } : {}), + ...(quota && mainQuotaWriter && isMainQuotaWriterLive(mainQuotaWriter) + && mainQuotaCredentialGeneration === getMainQuotaCredentialGeneration() + && matchesMainQuotaCredential(tokens.access_token, tokens.account_id) + ? { resetRecoveryProof: { writer: mainQuotaWriter, credentialGeneration: mainQuotaCredentialGeneration, dispatchSequence } } + : {}), ...(freshResetCredits !== undefined ? { freshResetCredits } : {}), }; } catch (error) { @@ -1011,6 +1048,8 @@ async function fetchMainAccountInfoWhileOwned( } interface PoolQuotaResult { + /** Actual refresh result attached only to the successful usage replay. */ + resetRefreshLineage?: ManualResetRefreshLineage; quota: StoredAccountQuota | null; needsReauth: boolean; /** Credential generation whose cache or network result this DTO state belongs to. */ @@ -1025,15 +1064,25 @@ interface PoolQuotaResult { freshResetCredits?: number; quotaProbeSkipped?: true; /** Positive evidence captured immediately before an upstream WHAM dispatch. */ - quotaProbeAttempted?: { at: number; credentialGeneration: number }; + quotaProbeAttempted?: { at: number; credentialGeneration: number; dispatchSequence: number }; } +// Process-local ordering, never a timestamp or a serialized account identifier. +let quotaDispatchSequence = 0; +// Shared native-main ownership permits concurrent usage readers. Only a later +// successfully published response advances this fence; failed reads do not win. +let mainQuotaPublishedSequence = 0; + interface PoolQuotaProbeEvidence { + onDispatch?: (sequence: number) => void; + mayPublish?: () => boolean; attempted?: NonNullable; } function markQuotaProbeAttempted(evidence: PoolQuotaProbeEvidence, credentialGeneration: number): void { - evidence.attempted = { at: Date.now(), credentialGeneration }; + const dispatchSequence = ++quotaDispatchSequence; + evidence.attempted = { at: Date.now(), credentialGeneration, dispatchSequence }; + evidence.onDispatch?.(dispatchSequence); } function withQuotaProbeEvidence( @@ -1045,6 +1094,8 @@ function withQuotaProbeEvidence( interface PoolQuotaRefreshFlight { state: { + dispatchSequence?: number; + superseded?: boolean; startCredentialGeneration?: number; resolvedCredentialGeneration?: number; }; @@ -1280,9 +1331,18 @@ async function recoverPoolQuotaFrom401(ctx: { } return { quota: existing ?? null, needsReauth: false, credentialGeneration: refreshed.generation }; } - return await commitPoolQuotaResponse(replay, { + const result = await commitPoolQuotaResponse(replay, { accountId, existing, configuredPlan, generation: refreshed.generation, writerGeneration, + mayPublish: ctx.quotaProbeEvidence.mayPublish, }); + return result.freshCredentialGeneration === refreshed.generation ? { + ...result, + resetRefreshLineage: { + fromGeneration: rejectedGeneration, + toGeneration: refreshed.generation, + provenance: refreshed.provenance, + }, + } : result; } /** Backoff after a refresh failure that proved nothing about the credential. */ @@ -1314,10 +1374,14 @@ async function commitPoolQuotaResponse( configuredPlan: string | undefined; generation: number; writerGeneration: number; + mayPublish?: () => boolean; }, ): Promise { const { accountId, existing, configuredPlan, generation, writerGeneration } = ctx; const data = (await resp.json()) as WhamUsageResponse; + if (ctx.mayPublish?.() === false) { + return { quota: getAccountQuota(accountId), needsReauth: false, credentialGeneration: generation }; + } const freshPlan = nonEmptyPlan(data.plan_type) ?? undefined; const quota = parseUsageQuota({ ...data, plan_type: freshPlan ?? configuredPlan }); const freshResetCredits = quota?.resetCredits; @@ -1350,10 +1414,10 @@ async function fetchFreshPoolAccountQuota( configuredPlan?: string, onCredentialGeneration?: (generation: number) => void, getValidToken: typeof getValidCodexToken = getValidCodexToken, + quotaProbeEvidence: PoolQuotaProbeEvidence = {}, ): Promise { const writerGeneration = captureConfigGeneration(); let requestCredentialGeneration = readCodexAccountRecord(accountId)?.generation; - const quotaProbeEvidence: PoolQuotaProbeEvidence = {}; try { const { accessToken, chatgptAccountId, generation } = await getValidToken(accountId); requestCredentialGeneration = generation; @@ -1387,6 +1451,7 @@ async function fetchFreshPoolAccountQuota( } const committed = await commitPoolQuotaResponse(resp, { accountId, existing, configuredPlan, generation, writerGeneration, + mayPublish: quotaProbeEvidence.mayPublish, }); return withQuotaProbeEvidence(committed, quotaProbeEvidence); } catch (e) { @@ -1417,9 +1482,10 @@ async function fetchPoolAccountQuota( forceRefresh = false, configuredPlan?: string, getValidToken: typeof getValidCodexToken = getValidCodexToken, + afterDispatchSequence?: number, ): Promise { const existing = getAccountQuota(accountId); - if (!forceRefresh && existing && Date.now() - existing.updatedAt < POOL_CACHE_TTL) { + if (afterDispatchSequence === undefined && !forceRefresh && existing && Date.now() - existing.updatedAt < POOL_CACHE_TTL) { return { quota: existing, needsReauth: false, @@ -1434,11 +1500,18 @@ async function fetchPoolAccountQuota( const current = flights && [...flights].find(flight => { const generation = flight.state.resolvedCredentialGeneration ?? flight.state.startCredentialGeneration; - return generation !== undefined && isCodexAccountGenerationLive(accountId, generation); + return !flight.state.superseded + && (afterDispatchSequence === undefined || (flight.state.dispatchSequence ?? 0) > afterDispatchSequence) + && generation !== undefined && isCodexAccountGenerationLive(accountId, generation); }); if (current) return current.promise; if (poolQuotaFlightCount() >= MAX_POOL_QUOTA_FLIGHTS) throw new PoolQuotaProbeBusyError(); + // A post-reset request must not let an older same-account response overwrite its evidence. + // Flags live only as long as the bounded flights; no retained per-account sequence map. + if (afterDispatchSequence !== undefined) { + for (const flight of flights ?? []) flight.state.superseded = true; + } const state: PoolQuotaRefreshFlight["state"] = { startCredentialGeneration: record?.generation, }; @@ -1448,6 +1521,10 @@ async function fetchPoolAccountQuota( configuredPlan, generation => { state.resolvedCredentialGeneration = generation; }, getValidToken, + { + onDispatch: sequence => { state.dispatchSequence = sequence; }, + mayPublish: () => state.superseded !== true, + }, ); const flight: PoolQuotaRefreshFlight = { state, promise: refresh }; const activeFlights = flights ?? new Set(); @@ -1463,6 +1540,74 @@ async function fetchPoolAccountQuota( } } +function manualResetAuthStillLive(accountId: string, auth: ResetCreditAuth): boolean { + if (!auth.isMain) { + const record = readCodexAccountRecord(accountId); + return auth.poolGeneration !== undefined + && isCodexAccountGenerationLive(accountId, auth.poolGeneration) + && record?.credential?.chatgptAccountId === auth.chatgptAccountId; + } + const tokens = readCodexTokens(); + return !!auth.mainProof && !!tokens + && tokens.access_token === auth.accessToken && tokens.account_id === auth.chatgptAccountId + && isMainQuotaWriterLive(auth.mainProof.writer) + && auth.mainProof.credentialGeneration === getMainQuotaCredentialGeneration() + && matchesMainQuotaCredential(auth.accessToken, auth.chatgptAccountId); +} + +/** A confirmed spend remains successful even when its optional usage observation fails. */ +async function refreshAfterManualReset( + config: OcxConfig, + accountId: string, + auth: ResetCreditAuth, + claims: ManualResetCooldownClaim[], + didReset: boolean, +): Promise { + const afterDispatchSequence = quotaDispatchSequence; + try { + if (!manualResetAuthStillLive(accountId, auth)) return undefined; + if (auth.isMain) { + const result = await fetchMainAccountInfoAttempt(true, 1, auth.nativeMainLease, + auth.nativeMainSharedClaimHeld === true, false); + const proof = result.resetRecoveryProof; + const recovered = didReset && manualResetAuthStillLive(accountId, auth) + && !!proof && !!auth.mainProof + && proof.dispatchSequence > afterDispatchSequence + && proof.credentialGeneration === auth.mainProof.credentialGeneration + && proof.writer.identityKey === auth.mainProof.writer.identityKey + && proof.writer.identityGeneration === auth.mainProof.writer.identityGeneration + && isCompleteCodexQuotaRecoverySnapshot(result.freshQuota ?? null, result.info.plan); + for (const claim of claims) settleManualResetCooldown(getRuntimeConfig(config), claim, recovered); + return manualResetAuthStillLive(accountId, auth) ? result.freshResetCredits : undefined; + } + const account = configuredPoolAccount(getRuntimeConfig(config), accountId); + if (!account) return undefined; + // Reuse the just-authenticated consume credential for the first usage request. + // getValidCodexToken can silently advance a generation without exposing refresh + // provenance. A 401 here instead uses the existing classified refresh/replay path. + const resetToken: typeof getValidCodexToken = async () => { + if (auth.poolGeneration === undefined || !manualResetAuthStillLive(accountId, auth)) { + throw new CodexCredentialGenerationConflictError(); + } + return { accessToken: auth.accessToken, chatgptAccountId: auth.chatgptAccountId, generation: auth.poolGeneration }; + }; + const result = await fetchPoolAccountQuota(accountId, true, account.plan, didReset ? resetToken : getValidCodexToken, + didReset ? afterDispatchSequence : undefined); + const record = readCodexAccountRecord(accountId); + const recovered = didReset && record?.credential?.chatgptAccountId === auth.chatgptAccountId + && (result.quotaProbeAttempted?.dispatchSequence ?? 0) > afterDispatchSequence + && isCompleteCodexQuotaRecoverySnapshot(result.freshQuota ?? null, result.freshPlan ?? account.plan); + for (const claim of claims) settleManualResetCooldown(getRuntimeConfig(config), claim, recovered, { + credentialGeneration: result.freshCredentialGeneration, + refreshLineage: result.resetRefreshLineage, + }); + return record?.credential?.chatgptAccountId === auth.chatgptAccountId ? result.freshResetCredits : undefined; + } catch { + // The upstream reset already happened. A failed refresh must not invite another spend. + return undefined; + } +} + let primeInFlight: Promise | null = null; /** * Last prime attempt per pool account. A failed WHAM lookup stores no quota, so @@ -2382,71 +2527,69 @@ export async function handleCodexAuthAPI( } else { idempotencyKey = crypto.randomUUID(); } - let resp: Response; + const claims = manualResetAuthStillLive(accountId, auth) + ? claimManualResetCooldowns(getRuntimeConfig(config), accountId, Date.now(), auth.poolGeneration) : []; try { - resp = await fetch( - "https://chatgpt.com/backend-api/wham/rate-limit-reset-credits/consume", - { - method: "POST", - headers: { - Authorization: `Bearer ${auth.accessToken}`, - "ChatGPT-Account-Id": auth.chatgptAccountId, - "Content-Type": "application/json", + let resp: Response; + try { + resp = await fetch( + "https://chatgpt.com/backend-api/wham/rate-limit-reset-credits/consume", + { + method: "POST", + headers: { + Authorization: `Bearer ${auth.accessToken}`, + "ChatGPT-Account-Id": auth.chatgptAccountId, + "Content-Type": "application/json", + }, + body: JSON.stringify({ redeem_request_id: idempotencyKey }), + signal: AbortSignal.timeout(10_000), }, - body: JSON.stringify({ redeem_request_id: idempotencyKey }), - signal: AbortSignal.timeout(10_000), - }, - ); - } catch (error) { - // Dispatch outcome unknown: the credit may or may not have been spent. - // Mark ambiguous so a replay of this same id is never treated as new. - if (identity) markManualResetCreditOperationAmbiguous(identity); - throw error; - } - if (!resp.ok) { - await resp.body?.cancel().catch(() => {}); - if (identity) markManualResetCreditOperationAmbiguous(identity); - return jsonResponse({ error: `Upstream error ${resp.status}` }, resp.status); - } - const result = safeResetCreditConsumeDto(await resp.json()); - if (identity) { - // Narrow explicitly rather than casting: `safeResetCreditConsumeDto` - // normalizes anything unrecognized to "unknown", and settling that - // would come back as a mismatch and leave the row pending anyway. - // Settlement failure never downgrades the user-visible outcome: the - // spend already happened upstream, and reporting failure would invite - // a manual retry -- the exact double-spend this unit removes. - if (result.code === "reset" || result.code === "already_redeemed" - || result.code === "nothing_to_reset" || result.code === "no_credit") { - settleManualResetCreditOperation(identity, result.code); - } else { - markManualResetCreditOperationAmbiguous(identity); + ); + } catch (error) { + // Dispatch outcome unknown: the credit may or may not have been spent. + // Mark ambiguous so a replay of this same id is never treated as new. + if (identity) markManualResetCreditOperationAmbiguous(identity); + throw error; } - } - // After a successful redeem (or an idempotent already_redeemed), refresh WHAM usage - // and return remaining only when that refresh freshly parsed available_count. - // Do not fall back to a preserved cached resetCredits (failed/omitted refresh). - if (result.code === "reset" || result.code === "already_redeemed") { - let freshResetCredits: number | undefined; - if (auth.isMain) { - ({ freshResetCredits } = await fetchMainAccountInfoAttempt( - true, - 1, - auth.nativeMainLease, - auth.nativeMainSharedClaimHeld === true, - )); - } else { - const account = configuredPoolAccount(getRuntimeConfig(config), accountId); - ({ freshResetCredits } = await fetchPoolAccountQuota(accountId, true, account?.plan)); + if (!resp.ok) { + await resp.body?.cancel().catch(() => {}); + if (identity) markManualResetCreditOperationAmbiguous(identity); + return jsonResponse({ error: `Upstream error ${resp.status}` }, resp.status); } - return jsonResponse({ - code: result.code, - ...(typeof freshResetCredits === "number" && Number.isFinite(freshResetCredits) - ? { remaining: freshResetCredits } - : {}), - }); + const result = safeResetCreditConsumeDto(await resp.json()); + if (identity) { + // Narrow explicitly rather than casting: `safeResetCreditConsumeDto` + // normalizes anything unrecognized to "unknown", and settling that + // would come back as a mismatch and leave the row pending anyway. + // Settlement failure never downgrades the user-visible outcome: the + // spend already happened upstream, and reporting failure would invite + // a manual retry -- the exact double-spend this unit removes. + if (result.code === "reset" || result.code === "already_redeemed" + || result.code === "nothing_to_reset" || result.code === "no_credit") { + settleManualResetCreditOperation(identity, result.code); + } else { + markManualResetCreditOperationAmbiguous(identity); + } + } + // After a successful redeem (or an idempotent already_redeemed), refresh WHAM usage + // and return remaining only when that refresh freshly parsed available_count. + // Do not fall back to a preserved cached resetCredits (failed/omitted refresh). + if (result.code === "reset" || result.code === "already_redeemed") { + const freshResetCredits = await refreshAfterManualReset( + config, accountId, auth, claims, result.code === "reset", + ); + return jsonResponse({ + code: result.code, + ...(typeof freshResetCredits === "number" && Number.isFinite(freshResetCredits) + ? { remaining: freshResetCredits } + : {}), + }); + } + return jsonResponse(result); + } finally { + // Release only this invocation's leases, including every ambiguous/error outcome. + for (const claim of claims) settleManualResetCooldown(getRuntimeConfig(config), claim, false); } - return jsonResponse(result); }); return operation.ok ? operation.value : operation.response; } catch (e) { diff --git a/src/codex/routing.ts b/src/codex/routing.ts index dbf9cab086..5d8cc17d15 100644 --- a/src/codex/routing.ts +++ b/src/codex/routing.ts @@ -1,6 +1,6 @@ import { randomUUID } from "node:crypto"; import { saveConfigPreservingClaudeCode } from "../config"; -import { isCodexAccountGenerationLive, readCodexAccountRecord } from "./account-store"; +import { isCodexAccountGenerationLive, readCodexAccountRecord, type CodexRefreshProvenance } from "./account-store"; import { codexAccountLogLabel } from "./account-label"; import { NATIVE_RESERVE_MODEL } from "./catalog/native-models"; import { isCodexAccountPaused } from "./account-pause"; @@ -642,6 +642,80 @@ export function claimDueCodexQuotaRecoveryProbes( }); } +type CooldownRecoveryLease = Pick; + +export type ManualResetCooldownClaim = + | { kind: "pool"; probe: CodexQuotaRecoveryProbeClaim } + | { kind: "main"; probe: CooldownRecoveryLease }; + +function manualResetAccountEligible(config: OcxConfig, accountId: string): boolean { + return !isCodexAccountPaused(config, accountId) && !isAccountNeedsReauth(accountId) + && (accountId === MAIN_CODEX_ACCOUNT_ID + || (config.codexAccounts ?? []).some(account => account.id === accountId && isSelectableCodexPoolAccount(account))); +} + +/** Explicit reset bypasses probe pacing, never another owner's lease or quota scope. */ +export function claimManualResetCooldowns( + config: OcxConfig, + accountId: string, + now = Date.now(), + expectedPoolGeneration?: number, +): ManualResetCooldownClaim[] { + if (!manualResetAccountEligible(config, accountId)) return []; + const record = accountId === MAIN_CODEX_ACCOUNT_ID ? undefined : readCodexAccountRecord(accountId); + if (accountId !== MAIN_CODEX_ACCOUNT_ID && (!record?.credential || record.deletedAt != null)) return []; + if (record && expectedPoolGeneration !== undefined && record.generation !== expectedPoolGeneration) return []; + const claims: ManualResetCooldownClaim[] = []; + for (const scope of [undefined, "shared"] as const) { + const health = scope ? scopedHealthFor(accountId, scope) : upstreamHealth.get(accountId); + if (!health || health.cooldownSource !== "reset-derived" || health.probeLeaseId !== undefined + || !Number.isFinite(health.cooldownUntil) || !(health.cooldownUntil! > now)) continue; + const leaseId = randomUUID(); + const cooldownGeneration = health.cooldownGeneration ?? 0; + const next = { ...health, probeLeaseId: leaseId, probeLeaseGeneration: cooldownGeneration, lastProbeAt: now }; + if (scope) setScopedHealth(accountId, scope, next); + else upstreamHealth.set(accountId, next); + const probe = { accountId, scope, leaseId, cooldownGeneration }; + claims.push(record ? { kind: "pool", probe: { + ...probe, credentialGeneration: record.generation, credentialReplacedAt: record.replacedAt, + } } : { kind: "main", probe }); + } + return claims; +} + +export type ManualResetRefreshLineage = Readonly<{ + fromGeneration: number; + toGeneration: number; + provenance: CodexRefreshProvenance; +}>; + +type ManualResetQuotaProof = CodexQuotaRecoveryProbeProof & { + refreshLineage?: ManualResetRefreshLineage; +}; + +/** Main proof is checked by the already-owned auth operation, never by a Pool record. */ +export function settleManualResetCooldown( + config: OcxConfig, + claim: ManualResetCooldownClaim, + recovered: boolean, + proof: ManualResetQuotaProof = {}, + now = Date.now(), +): boolean { + if (!recovered) return settleCooldownRecoveryLease(claim.probe, false, now); + const eligible = manualResetAccountEligible(config, claim.probe.accountId); + if (claim.kind === "main") return settleCooldownRecoveryLease(claim.probe, eligible, now); + const lineage = proof.refreshLineage; + // Equal wall-clock replacement stamps do not establish ancestry. Manual +1 + // recovery additionally needs the actual forced-refresh result for this edge. + const ownedGeneration = proof.credentialGeneration === claim.probe.credentialGeneration + || (proof.credentialGeneration === claim.probe.credentialGeneration + 1 + && lineage?.fromGeneration === claim.probe.credentialGeneration + && lineage.toGeneration === proof.credentialGeneration + && (lineage.provenance === "self-refresh" || lineage.provenance === "joined-lineage")); + return settleCodexQuotaRecoveryProbe(claim.probe, eligible && ownedGeneration, proof, now); +} + /** Settle one background recovery claim without mutating account-wide outcome state. */ export function settleCodexQuotaRecoveryProbe( claim: CodexQuotaRecoveryProbeClaim, @@ -665,9 +739,16 @@ export function settleCodexQuotaRecoveryProbe( : proofGeneration === claim.credentialGeneration + 1 && currentRecord?.replacedAt === claim.credentialReplacedAt && isCodexAccountGenerationLive(claim.accountId, proofGeneration)); - const fenced = (health.cooldownGeneration ?? 0) === claim.cooldownGeneration - && (health.probeLeaseGeneration ?? 0) === claim.cooldownGeneration - && generationFenced; + return settleCooldownRecoveryLease(claim, recovered && generationFenced, now); +} + +function settleCooldownRecoveryLease(claim: CooldownRecoveryLease, recovered: boolean, now: number): boolean { + const health = claim.scope ? scopedHealthFor(claim.accountId, claim.scope) : upstreamHealth.get(claim.accountId); + if (!health || health.probeLeaseId !== claim.leaseId) return false; + const fenced = (claim.scope === undefined || claim.scope === "shared") + && health.cooldownSource === "reset-derived" + && (health.cooldownGeneration ?? 0) === claim.cooldownGeneration + && (health.probeLeaseGeneration ?? 0) === claim.cooldownGeneration; if (!recovered || !fenced) { const released = withProbeLeaseReleased(health, now); if (claim.scope) setScopedHealth(claim.accountId, claim.scope, released); diff --git a/structure/08_openai-provider-tiers.md b/structure/08_openai-provider-tiers.md index 834cbd46ee..91627acaf0 100644 --- a/structure/08_openai-provider-tiers.md +++ b/structure/08_openai-provider-tiers.md @@ -72,6 +72,20 @@ requests keep their captured credential. An all-paused pool fails closed. The dashboard's bulk pause action refreshes all account quotas and mutates only accounts whose plan-relevant window is freshly confirmed at exactly 100%; unknown and failed refreshes are skipped. +A confirmed manual reset-credit consumption may immediately reconcile that account's +eligible pre-existing ordinary reset-derived cooldown after a complete, non-exhausted usage +observation started after the reset. Paused or reauthentication-required accounts and +cooldowns held by another in-flight probe remain excluded; their cooldowns are retained. +Recovery owns the specific cooldown and authenticates +main and added Pool accounts through their respective credential contracts. Main usage +publication keeps the latest successfully published observation authoritative. Pool recovery +across a credential refresh requires the actual self/joined refresh lineage, not matching +replacement timestamps. It preserves +newer failures, independent Spark/Reserve scopes, explicit Retry-After, pause, pin and +selection state. Replay and `already_redeemed` are not new-reset evidence. Failed usage +recovery leaves the cooldown in place and preserves the confirmed consume success; +retrying usage must not require another credit. + `codexQuotaAutoRefresh` is a separate default-off spending intent. For each explicitly enabled account/window, the one-minute state sweep compares the cached upstream reset timestamp, sends the existing minimal non-stored warmup through that exact account once the timestamp is due, then diff --git a/tests/cli/cli-restart-health.test.ts b/tests/cli/cli-restart-health.test.ts index ac2dfae50d..3b9498707f 100644 --- a/tests/cli/cli-restart-health.test.ts +++ b/tests/cli/cli-restart-health.test.ts @@ -1,10 +1,11 @@ import { describe, expect, test } from "bun:test"; -import { spawnSync } from "node:child_process"; -import { mkdtempSync, writeFileSync } from "node:fs"; +import { existsSync, mkdtempSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { watchdogMs } from "../helpers/ci-watchdog"; +import { captureTestOutput } from "../../scripts/test"; const repoRoot = dirname(fileURLToPath(new URL("../../package.json", import.meta.url))); const cliPath = join(repoRoot, "src", "cli", "index.ts"); @@ -16,20 +17,144 @@ const cliPath = join(repoRoot, "src", "cli", "index.ts"); * network/no-proxy/argument-validation ready tests live as injected tests in * tests/cli/cli-ready.test.ts (no real loopback/home). */ -function runCli(args: string[], env: Record = {}) { - return spawnSync(process.execPath, [cliPath, ...args], { - cwd: repoRoot, - env: { ...process.env, ...env }, - encoding: "utf8", - timeout: 10000, - }); +// These are correctness watchdogs, not startup latency assertions. Scale only execution. +const CLI_BUDGET = { execution: watchdogMs(10_000), term: 5_000, reap: 2_000, drain: 1_000 }; +const CLI_TEST_TIMEOUT = CLI_BUDGET.execution + CLI_BUDGET.term + CLI_BUDGET.reap + CLI_BUDGET.drain + 3_000; +type CliChild = Pick, "pid" | "exited" | "signalCode" | "stdout" | "stderr" | "kill">; +type CliSpawn = (argv: string[], options: { + cwd: string; env: NodeJS.ProcessEnv; stdout: "pipe"; stderr: "pipe"; +}) => CliChild; +type CliState = { + id: string; startedAt: number; pid: number | null; reaped: boolean; + status: number | null; signal: NodeJS.Signals | null; + stdout: string; stderr: string; complete: boolean; +}; +const cliHomes = new Map(); + +function cliStage(state: CliState, stage: string): void { + console.warn(`[cli-probe:${state.id}] ${stage} elapsedMs=${Date.now() - state.startedAt} pid=${state.pid}`); +} + +function errorTag(error: unknown): string { + const name = error instanceof Error ? error.name : "UnknownError"; + const code = error && typeof error === "object" && "code" in error ? String(error.code) : ""; + // Error messages can contain argv or environment. Log only conventional name/code tags. + return `${/^[A-Za-z]+$/.test(name) ? name : "Error"}${/^[A-Z0-9_]+$/.test(code) ? `:${code}` : ""}`; +} + +class CliHarnessError extends Error { + constructor(readonly failures: string[], readonly outcome: CliState) { + super(`[cli-probe:${outcome.id}] ${failures.join(", ")} pid=${outcome.pid} status=${outcome.status} signal=${outcome.signal} reaped=${outcome.reaped} complete=${outcome.complete}`); + this.name = "CliHarnessError"; + } +} + +async function waitForCliExit(exited: Promise, milliseconds: number): Promise { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + exited.then(() => true), + new Promise(resolve => { timer = setTimeout(() => resolve(false), milliseconds); }), + ]); + } finally { + clearTimeout(timer); + } +} + +async function runCli(args: string[], env: Record = {}, control?: { + spawn: CliSpawn; budget: typeof CLI_BUDGET; +}): Promise<{ status: number; stdout: string; stderr: string }> { + const state = cliHomes.get(env.OPENCODEX_HOME); + if (!state) throw new Error("CLI probe requires an owned isolated home"); + const budget = control?.budget ?? CLI_BUDGET; + const spawn: CliSpawn = control?.spawn ?? ((argv, options) => Bun.spawn(argv, options)); + const failures: string[] = []; + let child: CliChild | undefined; + let exited: Promise | undefined; + let capture: ReturnType | undefined; + let boundary = "spawn"; + try { + cliStage(state, "03 spawn requested"); + child = spawn([process.execPath, cliPath, ...args], { + cwd: repoRoot, env: { ...process.env, ...env }, stdout: "pipe", stderr: "pipe", + }); + state.pid = child.pid; // Establish ownership before any observation or capture can fail. + const owned = child; + exited = owned.exited.then(status => { + state.reaped = true; + state.status = status; + state.signal = owned.signalCode ?? null; + cliStage(state, `08 exit status=${status} signal=${state.signal}`); + }).catch(error => { + failures.push(`exit-observation-error:${errorTag(error)}`); + cliStage(state, `08 ${failures[failures.length - 1]}`); + }); + cliStage(state, "04 child owned"); + boundary = "capture"; + capture = captureTestOutput(owned.stdout, owned.stderr); + boundary = "execution"; + if (!await waitForCliExit(exited, budget.execution)) { + failures.push("execution-timeout"); + cliStage(state, "05 execution timeout"); + } + } catch (error) { + failures.push(`${boundary}-error:${errorTag(error)}`); + } finally { + if (child && !state.reaped) { + cliStage(state, "06 TERM"); + try { child.kill("SIGTERM"); } catch (error) { cliStage(state, `06 TERM error=${errorTag(error)}`); } + if (exited) await waitForCliExit(exited, budget.term); + if (!state.reaped) { + cliStage(state, "07 KILL"); + try { child.kill("SIGKILL"); } catch (error) { cliStage(state, `07 KILL error=${errorTag(error)}`); } + if (exited) await waitForCliExit(exited, budget.reap); + } + if (!state.reaped) failures.push("reap-timeout"); + } + if (capture) { + try { Object.assign(state, await capture.finish(budget.drain)); } + catch (error) { failures.push(`capture-error:${errorTag(error)}`); } + cliStage(state, `09 capture complete=${state.complete}`); + if (!state.complete) failures.push("incomplete-output"); + } + } + if (!state.reaped || state.status === null || !Number.isInteger(state.status)) failures.push("exit-not-observed"); + if (state.signal !== null) failures.push("signal-exit"); + // Never turn timeout/incomplete capture into status 1: health legitimately expects 1. + if (failures.length) throw new CliHarnessError([...failures], { ...state }); + return { status: state.status!, stdout: state.stdout, stderr: state.stderr }; } function isolatedHome(prefix: string): string { - return mkdtempSync(join(tmpdir(), prefix)); + const state: CliState = { + id: prefix, startedAt: Date.now(), pid: null, reaped: false, + status: null, signal: null, stdout: "", stderr: "", complete: false, + }; + cliStage(state, "01 home setup"); + const dir = mkdtempSync(join(tmpdir(), prefix)); + cliHomes.set(dir, state); + return dir; +} + +function cleanupCliHome(dir: string, primaryFailed = false): void { + const state = cliHomes.get(dir); + if (!state) throw new Error("Cannot clean an unowned CLI home"); + if (state.pid !== null && !state.reaped) { + cliStage(state, "10 home retained: child unreaped"); + return; + } + try { + removeTreeWithRetry(dir); + cliHomes.delete(dir); + cliStage(state, "10 home removed"); + } catch (error) { + cliStage(state, `10 cleanup error=${errorTag(error)}`); + if (!primaryFailed) throw error; + } } function writeIsolatedConfig(dir: string): void { + cliStage(cliHomes.get(dir)!, "02 config setup"); writeFileSync(join(dir, "config.json"), JSON.stringify({ port: 19999, providers: { openai: { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward" } }, @@ -38,104 +163,227 @@ function writeIsolatedConfig(dir: string): void { }), "utf8"); } +describe("CLI subprocess lifecycle", () => { + const budget = { execution: 10, term: 10, reap: 10, drain: 10 }; + const scenarios: Array<{ + name: string; mode: "exit" | "timeout" | "unreaped" | "spawn-error" | "exit-error"; + status: number | null; signal?: NodeJS.Signals; open?: boolean; + failures: string[]; signals: NodeJS.Signals[]; reaped: boolean; retained?: boolean; + }> = [ + { name: "returns exit 0", mode: "exit", status: 0, failures: [], signals: [], reaped: true }, + { name: "returns health exit 1", mode: "exit", status: 1, failures: [], signals: [], reaped: true }, + { name: "preserves exit 23", mode: "exit", status: 23, failures: [], signals: [], reaped: true }, + { name: "timeout stays failed after TERM yields exit 0", mode: "timeout", status: 0, + failures: ["execution-timeout"], signals: ["SIGTERM"], reaped: true }, + { name: "open output after exit 0 fails", mode: "exit", status: 0, open: true, + failures: ["incomplete-output"], signals: [], reaped: true }, + { name: "open output after exit 1 fails", mode: "exit", status: 1, open: true, + failures: ["incomplete-output"], signals: [], reaped: true }, + { name: "unreaped child retains its home after TERM and KILL", mode: "unreaped", status: null, + failures: ["execution-timeout", "reap-timeout", "exit-not-observed"], + signals: ["SIGTERM", "SIGKILL"], reaped: false, retained: true }, + { name: "spawn error is not command exit 1", mode: "spawn-error", status: null, + failures: ["spawn-error:Error:ENOENT", "exit-not-observed"], signals: [], reaped: false }, + { name: "rejected observation is not reaping", mode: "exit-error", status: null, + failures: ["exit-observation-error:Error:EPIPE", "reap-timeout", "exit-not-observed"], + signals: ["SIGTERM", "SIGKILL"], reaped: false, retained: true }, + { name: "signal exit is not a completed command", mode: "exit", status: 0, signal: "SIGTERM", + failures: ["signal-exit"], signals: [], reaped: true }, + ]; + + for (const scenario of scenarios) test(scenario.name, async () => { + const dir = isolatedHome(`ocx-cli-control-${scenario.name.replace(/[^a-z0-9]+/gi, "-")}-`); + const state = cliHomes.get(dir)!; + let resolveExit!: (status: number) => void; + let rejectExit!: (error: Error) => void; + const exited = new Promise((resolve, reject) => { resolveExit = resolve; rejectExit = reject; }); + const signals: Array = []; + let cancelled = false; + const child: CliChild = { + pid: 424242, exited, signalCode: scenario.signal ?? null, + stdout: new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode("CLI_CONTROL_STDOUT\n")); + if (!scenario.open) controller.close(); + }, + cancel() { cancelled = true; }, + }), + stderr: new ReadableStream({ + start(controller) { controller.enqueue(new TextEncoder().encode("CLI_CONTROL_STDERR\n")); controller.close(); }, + }), + kill(signal) { + signals.push(signal); + if (scenario.mode === "timeout") resolveExit(0); + }, + }; + const spawn: CliSpawn = (argv, options) => { + expect(argv).toEqual([process.execPath, cliPath, "health"]); + expect(options.cwd).toBe(repoRoot); + expect(options.env.OPENCODEX_HOME).toBe(dir); + if (scenario.mode === "spawn-error") throw Object.assign(new Error("fixture"), { code: "ENOENT" }); + if (scenario.mode === "exit-error") rejectExit(Object.assign(new Error("fixture"), { code: "EPIPE" })); + if (scenario.mode === "exit") resolveExit(scenario.status!); + return child; + }; + try { + const result: unknown = await runCli(["health"], { OPENCODEX_HOME: dir }, { spawn, budget }) + .then(value => value, error => error); + if (scenario.failures.length) { + expect(result).toBeInstanceOf(CliHarnessError); + if (!(result instanceof CliHarnessError)) throw new Error("Expected CLI harness failure"); + expect(result.failures).toEqual(scenario.failures); + if (scenario.open) expect(result.outcome.stdout).toBe("CLI_CONTROL_STDOUT\n"); + } else { + expect(result).toEqual({ status: scenario.status, stdout: "CLI_CONTROL_STDOUT\n", stderr: "CLI_CONTROL_STDERR\n" }); + } + expect(state.status).toBe(scenario.status); + expect(state.pid).toBe(scenario.mode === "spawn-error" ? null : 424242); + expect(state.signal).toBe(scenario.signal ?? null); + expect(state.reaped).toBe(scenario.reaped); + expect(state.complete).toBe(scenario.mode !== "spawn-error" && !scenario.open); + expect(signals).toEqual(scenario.signals); + expect(cancelled).toBe(Boolean(scenario.open)); + cleanupCliHome(dir, scenario.failures.length > 0); + expect(existsSync(dir)).toBe(Boolean(scenario.retained)); + expect(cliHomes.has(dir)).toBe(Boolean(scenario.retained)); + } finally { + // The seam never launched an OS process; only this test owns the retained fake home. + cliHomes.delete(dir); + removeTreeWithRetry(dir); + } + }); +}); + describe("ocx restart", () => { - test("restart --help prints usage", () => { + test("restart --help prints usage", async () => { const dir = isolatedHome("ocx-restart-help-"); + let failed = false; try { - const result = runCli(["restart", "--help"], { OPENCODEX_HOME: dir }); + const result = await runCli(["restart", "--help"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(0); expect(result.stdout).toContain("ocx restart"); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); - test("help restart shows restart help entry", () => { + test("help restart shows restart help entry", async () => { const dir = isolatedHome("ocx-restart-help-entry-"); + let failed = false; try { - const result = runCli(["help", "restart"], { OPENCODEX_HOME: dir }); + const result = await runCli(["help", "restart"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(0); expect(result.stdout).toContain("Stop the proxy and restart"); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); }); describe("ocx health", () => { - test("health --help prints usage", () => { + test("health --help prints usage", async () => { const dir = isolatedHome("ocx-health-help-"); + let failed = false; try { - const result = runCli(["health", "--help"], { OPENCODEX_HOME: dir }); + const result = await runCli(["health", "--help"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(0); expect(result.stdout).toContain("ocx health"); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); - test("help health shows health help entry", () => { + test("help health shows health help entry", async () => { const dir = isolatedHome("ocx-health-help-entry-"); + let failed = false; try { - const result = runCli(["help", "health"], { OPENCODEX_HOME: dir }); + const result = await runCli(["help", "health"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(0); expect(result.stdout).toContain("Check proxy health"); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); - test("health exits 1 with no proxy running (isolated home)", () => { + test("health exits 1 with no proxy running (isolated home)", async () => { const dir = isolatedHome("ocx-health-"); - writeIsolatedConfig(dir); + let failed = false; try { - const result = runCli(["health"], { OPENCODEX_HOME: dir }); + writeIsolatedConfig(dir); + const result = await runCli(["health"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(1); expect(result.stdout).toContain("not healthy"); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); - test("health --json exits 1 with valid JSON when no proxy", () => { + test("health --json exits 1 with valid JSON when no proxy", async () => { const dir = isolatedHome("ocx-health-json-"); - writeIsolatedConfig(dir); + let failed = false; try { - const result = runCli(["health", "--json"], { OPENCODEX_HOME: dir }); + writeIsolatedConfig(dir); + const result = await runCli(["health", "--json"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(1); const parsed = JSON.parse(result.stdout); expect(parsed.ok).toBe(false); expect(parsed.pid).toBeNull(); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); }); describe("ocx ready", () => { // Only the help-routing subprocess checks live here. The default-probe, // --json, --wait, --timeout, and argument-validation cases are injected tests // in tests/cli/cli-ready.test.ts (no real loopback/home). - test("ready --help prints usage (exit 0)", () => { + test("ready --help prints usage (exit 0)", async () => { const dir = isolatedHome("ocx-ready-help-"); + let failed = false; try { - const result = runCli(["ready", "--help"], { OPENCODEX_HOME: dir }); + const result = await runCli(["ready", "--help"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(0); expect(result.stdout).toContain("ocx ready"); expect(result.stdout).toContain("--wait"); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); - test("help ready shows the ready help entry", () => { + test("help ready shows the ready help entry", async () => { const dir = isolatedHome("ocx-ready-help-entry-"); + let failed = false; try { - const result = runCli(["help", "ready"], { OPENCODEX_HOME: dir }); + const result = await runCli(["help", "ready"], { OPENCODEX_HOME: dir }); expect(result.status).toBe(0); expect(result.stdout).toContain("post-sync readiness"); + } catch (error) { + failed = true; + throw error; } finally { - removeTreeWithRetry(dir); + cleanupCliHome(dir, failed); } - }); + }, CLI_TEST_TIMEOUT); }); diff --git a/tests/codex-integration/codex-auth-api.test.ts b/tests/codex-integration/codex-auth-api.test.ts index 610ce7d9e4..3b704d6809 100644 --- a/tests/codex-integration/codex-auth-api.test.ts +++ b/tests/codex-integration/codex-auth-api.test.ts @@ -24,20 +24,28 @@ import { getCodexAccountCredential, listCodexAccountIds, readCodexAccountRecord, + removeCodexAccountCredential, saveCodexAccountCredential, } from "../../src/codex/account-store"; import * as accountStoreModule from "../../src/codex/account-store"; import * as reserveAvailabilityModule from "../../src/codex/reserve-availability"; import { getMainAccountInfoCache, observeMainQuotaCredential } from "../../src/codex/main-account-cache"; import { openManualResetCreditOperation } from "../../src/codex/reset-credit-operation-ledger"; +import { quotaRecoveryRecordForTests, resetQuotaRecoveryForTests } from "../../src/codex/quota-401-recovery"; +import { watchdogMs } from "../helpers/ci-watchdog"; import { clearCodexUpstreamHealth, + clearCodexUpstreamHealthForAccount, + getCodexQuotaHealthSnapshot, + claimManualResetCooldowns, + settleManualResetCooldown, clearThreadAccountMap, getCodexUpstreamHealth, recordCodexUpstreamOutcome, resetCodexRoutingForManualSelection, resolveCodexAccountForThread, } from "../../src/codex/routing"; +import { pinnedCodexAccountId, setCodexAccountPin } from "../../src/codex/account-priority"; import { clearPoolRotationState } from "../../src/codex/pool-rotation"; import { clearCodexWebSocketRegistry, @@ -48,6 +56,8 @@ import type { OcxConfig } from "../../src/types"; import type { WsData } from "../../src/server/ws-bridge"; import { handleNativeProfileAPI } from "../../src/codex/native-profile-api"; import type { NativeProfileManager } from "../../src/codex/native-profile-manager"; +import { getMainPolicyQuota } from "../../src/codex/quota"; +import { getMainAccountHardLockStatus } from "../../src/codex/main-account-hard-lock"; import { MAIN_CODEX_ACCOUNT_ID, setMainAccountPlan } from "../../src/codex/main-account"; import { reconcileCodexPlansFromTokens, resetJwtPlanNotesForTests } from "../../src/codex/plan-from-token"; import { @@ -543,6 +553,7 @@ beforeEach(() => { clearCodexWebSocketRegistry(); resetMainCodexAccountIdentityTrackingForTests(); resetJwtPlanNotesForTests(); + resetQuotaRecoveryForTests(); }); afterEach(async () => { @@ -558,6 +569,7 @@ afterEach(async () => { clearPoolRotationState(); clearCodexWebSocketRegistry(); globalThis.fetch = previousFetch; + resetQuotaRecoveryForTests(); if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousOpencodexHome; if (previousCodexHome === undefined) delete process.env.CODEX_HOME; @@ -972,13 +984,27 @@ describe("codex-auth API", () => { } }); - test("busy pool-quota probe maps reset-credit refresh to 503 server_busy with Retry-After 1", async () => { + test("busy usage observation preserves confirmed reset success without a retry directive", async () => { const config = makeConfig(); seedPoolAccount(config, { id: "quota-reset-busy", email: "busy@example.test" }); + // Adapted from #3995 (e172453052bf7bbc4a0ae5aa24592982c0c64b15). + recordCodexUpstreamOutcome(config, "quota-reset-busy", 429, { + now: Date.now(), resetAt: Date.now() + 3_600_000, modelId: "gpt-5.6-sol", fixedAccount: true, + }); + const cooldown = getCodexQuotaHealthSnapshot("quota-reset-busy", "shared"); + expect(cooldown).not.toBeNull(); const cleanup = seedCodexAuthAdmissionForTests({ quotaFlights: 16 }); - globalThis.fetch = (async (input: RequestInfo | URL) => String(input).includes("/consume") - ? Response.json({ code: "reset" }) - : previousFetch(input)) as typeof fetch; + let consumeCalls = 0; let usageCalls = 0; + const urls: string[] = []; + globalThis.fetch = (async (input: RequestInfo | URL) => { + const url = String(input); urls.push(url); + if (url === "https://chatgpt.com/backend-api/wham/rate-limit-reset-credits/consume") { + consumeCalls += 1; + return Response.json({ code: "reset" }); + } + if (url === "https://chatgpt.com/backend-api/wham/usage") usageCalls += 1; + throw new Error("unexpected mock URL"); + }) as typeof fetch; try { const req = new Request("http://localhost/api/codex-auth/reset-credits/consume", { method: "POST", @@ -986,9 +1012,16 @@ describe("codex-auth API", () => { body: JSON.stringify({ accountId: "quota-reset-busy" }), }); const response = await handleCodexAuthAPI(req, new URL(req.url), config); - expect(response?.status).toBe(503); - expect(response?.headers.get("Retry-After")).toBe("1"); - expect(await response?.json()).toMatchObject({ code: "server_busy" }); + expect(response?.status).toBe(200); + expect(response?.headers.get("Retry-After")).toBeNull(); + expect(await response?.json()).toEqual({ code: "reset" }); + expect(consumeCalls).toBe(1); + expect(usageCalls).toBe(0); + expect(urls).toEqual(["https://chatgpt.com/backend-api/wham/rate-limit-reset-credits/consume"]); + expect(getCodexQuotaHealthSnapshot("quota-reset-busy", "shared")).toEqual(cooldown); + const claims = claimManualResetCooldowns(config, "quota-reset-busy"); + try { expect(claims).toHaveLength(1); } + finally { for (const claim of claims) settleManualResetCooldown(config, claim, false); } } finally { cleanup(); } @@ -3048,6 +3081,42 @@ describe("codex-auth API", () => { } }); + // Adapted from luvs01's #3995, e172453052bf7bbc4a0ae5aa24592982c0c64b15. + test.each(["reset", "already_redeemed"])("cold main %s returns fresh WHAM credits without a prior lookup", async code => { + writeFileSync(join(TEST_CODEX_HOME, "auth.json"), JSON.stringify({ + tokens: { access_token: "cold-main-reset-token", account_id: "cold-main-reset-account" }, + })); + // Intentionally no listing, reconciliation, writer observation or quota seed. + let consumeCalls = 0; let usageCalls = 0; + const urls: string[] = []; + const originalFetch = globalThis.fetch; + globalThis.fetch = (async (input: RequestInfo | URL) => { + const url = String(input); urls.push(url); + if (url === "https://chatgpt.com/backend-api/wham/rate-limit-reset-credits/consume") { + consumeCalls += 1; + return Response.json({ code, remaining: 99 }); + } + if (url === "https://chatgpt.com/backend-api/wham/usage") { + usageCalls += 1; + return Response.json({ plan_type: "team", rate_limit: { secondary_window: { used_percent: 12 } }, + rate_limit_reset_credits: { available_count: 1 } }); + } + throw new Error("unexpected mock URL"); + }) as typeof fetch; + try { + const req = new Request("http://localhost/api/codex-auth/reset-credits/consume", { + method: "POST", headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ accountId: MAIN_CODEX_ACCOUNT_ID }), + }); + const response = await handleCodexAuthAPI(req, new URL(req.url), makeConfig()); + expect(response?.status).toBe(200); + expect(await response?.json()).toEqual({ code, remaining: 1 }); + expect(consumeCalls).toBe(1); expect(usageCalls).toBe(1); + expect(urls).toEqual(["https://chatgpt.com/backend-api/wham/rate-limit-reset-credits/consume", + "https://chatgpt.com/backend-api/wham/usage"]); + } finally { globalThis.fetch = originalFetch; } + }); + test("reset-credit consume returns remaining from fresh main WHAM credits", async () => { writeFileSync(join(TEST_CODEX_HOME, "auth.json"), JSON.stringify({ tokens: { access_token: "main-reset-ok", account_id: "acct-main-reset-ok" }, @@ -5454,3 +5523,539 @@ describe("codex-auth helpers", () => { expect(isAccountNeedsReauth(id)).toBe(false); }); }); + + +describe("manual reset cooldown recovery (#3973)", () => { + const USAGE = "https://chatgpt.com/backend-api/wham/usage"; + const CONSUME = "https://chatgpt.com/backend-api/wham/rate-limit-reset-credits/consume"; + const OP = "be810596-310c-4c21-95cb-e47f984398a0"; + function gate() { + let release!: () => void; + const promise = new Promise(resolve => { release = resolve; }); + return { promise, release }; + } + function usage(percent = 12) { + return { plan_type: "team", rate_limit: { secondary_window: { used_percent: percent } }, + rate_limit_reset_credits: { available_count: 2 } }; + } + function setup() { + const config = makeConfig({ activeCodexAccountId: "manual-a", accountPoolStrategy: "fill-first" }); + seedPoolAccount(config, { id: "manual-a", email: "manual@example.test", plan: "team" }); + setCodexAccountPin(config, "manual-a"); + cool(config, "manual-a"); + return config; + } + function cool(config: OcxConfig, id: string, modelId = "gpt-5.6-sol", now = Date.now()) { + recordCodexUpstreamOutcome(config, id, 429, { now, resetAt: now + 3_600_000, modelId, fixedAccount: true }); + } + function consume(config: OcxConfig, id = "manual-a", operationId = OP) { + const req = new Request("http://localhost/api/codex-auth/reset-credits/consume", { + method: "POST", headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ accountId: id, operationId }), + }); + return handleCodexAuthAPI(req, new URL(req.url), config); + } + function mock(consumeResponse: () => Response | Promise, usageResponse: () => Response | Promise) { + const urls: string[] = []; + globalThis.fetch = (async input => { + const url = String(input); + urls.push(url); + if (url === CONSUME) return consumeResponse(); + if (url === USAGE) return usageResponse(); + throw new Error("unexpected mock URL"); + }) as typeof fetch; + return urls; + } + + test.each(["reset", "already_redeemed", "nothing_to_reset", "no_credit", "unknown"])( + "only a new reset recovers, preserving pin/selection and other scopes: %s", async code => { + const config = setup(); + cool(config, "manual-a", "gpt-5.3-codex-spark"); + cool(config, "manual-a", "gpt-reserve"); + const spark = getCodexQuotaHealthSnapshot("manual-a", "spark"); + const reserve = getCodexQuotaHealthSnapshot("manual-a", "reserve"); + const urls = mock(() => Response.json({ code }), () => Response.json(usage())); + const result = await consume(config); + expect(result?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared") === null).toBe(code === "reset"); + expect(getCodexQuotaHealthSnapshot("manual-a", "spark")).toEqual(spark); + expect(getCodexQuotaHealthSnapshot("manual-a", "reserve")).toEqual(reserve); + expect(config.activeCodexAccountId).toBe("manual-a"); + expect(pinnedCodexAccountId(config)).toBe("manual-a"); + expect(urls).toEqual(code === "reset" || code === "already_redeemed" ? [CONSUME, USAGE] : [CONSUME]); + if (code === "reset") { + cool(config, "manual-a"); + const replay = await consume(config); + expect(await replay?.json()).toEqual({ code: "reset", replayed: true }); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).not.toBeNull(); + expect(urls).toEqual([CONSUME, USAGE]); + } + }, + ); + + test.each(["credits-only", "exhausted", "short-exhausted", "tertiary-only", "empty", "non-2xx", "malformed", "timeout"])( + "confirmed reset stays successful but incomplete/failed observation retains cooldown: %s", async kind => { + const config = setup(); + const urls = mock(() => Response.json({ code: "reset" }), () => { + if (kind === "timeout") throw new DOMException("fixture", "TimeoutError"); + if (kind === "non-2xx") return new Response("fixture", { status: 503 }); + if (kind === "malformed") return new Response("not-json"); + if (kind === "empty") return Response.json({}); + if (kind === "credits-only") return Response.json({ rate_limit_reset_credits: { available_count: 2 } }); + if (kind === "tertiary-only") return Response.json({ plan_type: "team", rate_limit: { tertiary_window: { used_percent: 5 } } }); + if (kind === "short-exhausted") return Response.json({ ...usage(), rate_limit: { + primary_window: { used_percent: 100, limit_window_seconds: 18_000 }, secondary_window: { used_percent: 12 }, + } }); + return Response.json(usage(100)); + }); + expect((await consume(config))?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).not.toBeNull(); + const nextClaims = claimManualResetCooldowns(config, "manual-a"); + expect(nextClaims).toHaveLength(1); + for (const claim of nextClaims) settleManualResetCooldown(config, claim, false); + expect(await (await consume(config))?.json()).toEqual({ code: "reset", replayed: true }); + expect(urls).toEqual([CONSUME, USAGE]); + }, + ); + + test("recovery never follows a physical-account match to another local alias", async () => { + const config = setup(); + seedPoolAccount(config, { id: "manual-alias", email: "alias@example.test", plan: "team", chatgptAccountId: "acct-manual-a" }); + cool(config, "manual-alias"); + const untouched = getCodexQuotaHealthSnapshot("manual-alias", "shared"); + const urls = mock(() => Response.json({ code: "reset" }), () => Response.json(usage())); + expect((await consume(config))?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toBeNull(); + expect(getCodexQuotaHealthSnapshot("manual-alias", "shared")).toEqual(untouched); + expect(urls).toEqual([CONSUME, USAGE]); + }); + + test.each(["team", "go", "free"])("monthly governing usage can recover %s", async plan => { + const config = setup(); + const urls = mock(() => Response.json({ code: "reset" }), () => Response.json({ plan_type: plan, + rate_limit: { primary_window: { used_percent: 4, limit_window_seconds: 2_628_000 } }, + })); + expect((await consume(config))?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toBeNull(); + expect(urls).toEqual([CONSUME, USAGE]); + }); + + test.each(["throw", "non-2xx", "unknown"])("ambiguous consume releases only its own cooldown claim: %s", async failure => { + const config = setup(); + const urls = mock(() => { + if (failure === "throw") throw new Error("fixture"); + return failure === "non-2xx" ? new Response("fixture", { status: 503 }) : Response.json({ code: "unknown" }); + }, () => Response.json(usage())); + const response = await consume(config); + expect(response?.status).toBe(failure === "throw" ? 500 : failure === "non-2xx" ? 503 : 200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).not.toBeNull(); + const claims = claimManualResetCooldowns(config, "manual-a"); + expect(claims).toHaveLength(1); + for (const claim of claims) settleManualResetCooldown(config, claim, false); + expect(urls).toEqual([CONSUME]); + }); + + test("post-reset 401 refresh carries the successful replay's dispatch and credential proof", async () => { + const config = setup(); const generation = readCodexAccountRecord("manual-a")!.generation; + const urls: string[] = []; let reads = 0; + globalThis.fetch = (async input => { + const url = String(input); urls.push(url); + if (url === CONSUME) return Response.json({ code: "reset" }); + if (url === USAGE) return ++reads === 1 ? new Response("{}", { status: 401 }) : Response.json(usage()); + if (url === "https://auth.openai.com/oauth/token") return Response.json({ + access_token: "refreshed-access", refresh_token: "refreshed-refresh", expires_in: 3600, + }); + throw new Error("unexpected mock URL"); + }) as typeof fetch; + expect((await consume(config))?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toBeNull(); + expect(readCodexAccountRecord("manual-a")!.generation).toBe(generation + 1); + expect(urls).toEqual([CONSUME, USAGE, "https://auth.openai.com/oauth/token", USAGE]); + }); + + test("same-tick external G+1 adopted by 401 replay cannot settle manual recovery", async () => { + const now = Date.now(); const clock = spyOn(Date, "now").mockReturnValue(now); + const firstUsage = gate(); const release401 = gate(); + let pending: ReturnType | undefined; + const forceRefresh = accountStoreModule.forceRefreshCodexPoolToken; + let observedProvenance: string | undefined; + const refreshSpy = spyOn(accountStoreModule, "forceRefreshCodexPoolToken").mockImplementation(async (id, options) => { + const result = await forceRefresh(id, options); + observedProvenance = result.provenance; + return result; + }); + try { + expect(quotaRecoveryRecordForTests("manual-a")).toBeUndefined(); + const config = setup(); + const original = getCodexAccountCredential("manual-a")!; + // Establish a non-undefined replacement stamp before the manual claim. + saveCodexAccountCredential("manual-a", original); + const before = readCodexAccountRecord("manual-a")!; + expect(before.replacedAt).toBe(now); + let reads = 0; + const urls = mock(() => Response.json({ code: "reset" }), async () => { + if (++reads === 1) { firstUsage.release(); await release401.promise; return new Response("{}", { status: 401 }); } + return Response.json(usage()); + }); + pending = consume(config); + await firstUsage.promise; + saveCodexAccountCredential("manual-a", { ...original, accessToken: "external-access", refreshToken: "external-refresh" }); + const replacement = readCodexAccountRecord("manual-a")!; + expect(replacement.generation).toBe(before.generation + 1); + expect(replacement.replacedAt).toBe(before.replacedAt); + release401.release(); + expect(await (await pending)?.json()).toEqual({ code: "reset", remaining: 2 }); + expect(observedProvenance).toBe("external-replacement"); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).not.toBeNull(); + // No OAuth call: forceRefresh adopted the time-valid external replacement. + expect(urls).toEqual([CONSUME, USAGE, USAGE]); + const claims = claimManualResetCooldowns(config, "manual-a"); + expect(claims).toHaveLength(1); + for (const claim of claims) settleManualResetCooldown(config, claim, false); + } finally { + release401.release(); + try { if (pending) await pending; } + finally { refreshSpy.mockRestore(); clock.mockRestore(); } + } + }); + + test("manual 401 can join a genuine owned refresh and retain its +1 lineage", async () => { + const config = setup(); const before = readCodexAccountRecord("manual-a")!; + const firstUsage = gate(); const release401 = gate(); const oauthStarted = gate(); const releaseOAuth = gate(); const joined = gate(); + const forceRefresh = accountStoreModule.forceRefreshCodexPoolToken; + let refreshCalls = 0; + let joinedProvenance: string | undefined; + const spy = spyOn(accountStoreModule, "forceRefreshCodexPoolToken").mockImplementation(async (id, options) => { + const result = forceRefresh(id, options); + const isJoiner = ++refreshCalls === 2; + if (isJoiner) joined.release(); + const resolved = await result; + if (isJoiner) joinedProvenance = resolved.provenance; + return resolved; + }); + const urls: string[] = []; let reads = 0; + globalThis.fetch = (async input => { + const url = String(input); urls.push(url); + if (url === CONSUME) return Response.json({ code: "reset" }); + if (url === USAGE) { + if (++reads === 1) { firstUsage.release(); await release401.promise; return new Response("{}", { status: 401 }); } + return Response.json(usage()); + } + if (url === "https://auth.openai.com/oauth/token") { + oauthStarted.release(); await releaseOAuth.promise; + return Response.json({ access_token: "joined-access", refresh_token: "joined-refresh", expires_in: 3600 }); + } + throw new Error("unexpected mock URL"); + }) as typeof fetch; + const pending = consume(config); + let owner: ReturnType | undefined; + try { + await firstUsage.promise; + owner = accountStoreModule.forceRefreshCodexPoolToken("manual-a", { + rejectedGeneration: before.generation, rejectedAccessToken: before.credential!.accessToken, + }); + await oauthStarted.promise; + release401.release(); await joined.promise; + releaseOAuth.release(); + expect((await owner).provenance).toBe("self-refresh"); + expect((await pending)?.status).toBe(200); + expect(joinedProvenance).toBe("joined-lineage"); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toBeNull(); + expect(readCodexAccountRecord("manual-a")!.generation).toBe(before.generation + 1); + expect(urls).toEqual([CONSUME, USAGE, "https://auth.openai.com/oauth/token", USAGE]); + } finally { + release401.release(); releaseOAuth.release(); + if (owner) await owner; await pending; + spy.mockRestore(); + } + }); + + test.each(["consume", "usage"])("new 429 during %s survives the old reset claim", async stage => { + const config = setup(); + const started = gate(); const finish = gate(); + let later: ReturnType; + mock(async () => { + if (stage === "consume") { started.release(); await finish.promise; } + return Response.json({ code: "reset" }); + }, async () => { + if (stage === "usage") { started.release(); await finish.promise; } + return Response.json(usage()); + }); + const pending = consume(config); + try { + await started.promise; + cool(config, "manual-a", "gpt-5.6-sol", Date.now() + 1); + later = getCodexQuotaHealthSnapshot("manual-a", "shared"); + } finally { finish.release(); } + expect((await pending)?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toEqual(later!); + }); + + test.each(["replace", "remove", "readd", "pause", "recreate"])("usage cannot recover after %s", async change => { + const config = setup(); const started = gate(); const finish = gate(); + mock(() => Response.json({ code: "reset" }), async () => { + started.release(); await finish.promise; return Response.json(usage()); + }); + const pending = consume(config); + try { + await started.promise; + if (change === "replace") saveCodexAccountCredential("manual-a", { + accessToken: "replacement", refreshToken: "replacement-refresh", expiresAt: Date.now() + 3_600_000, + chatgptAccountId: "replacement-account", + }); + if (change === "remove") config.codexAccounts = []; + if (change === "readd") { + removeCodexAccountCredential("manual-a"); + saveCodexAccountCredential("manual-a", { accessToken: "readded-access", refreshToken: "readded-refresh", + expiresAt: Date.now() + 3_600_000, chatgptAccountId: "acct-manual-a" }); + } + if (change === "pause") config.pausedCodexAccountIds = ["manual-a"]; + if (change === "recreate") { clearCodexUpstreamHealthForAccount("manual-a"); cool(config, "manual-a"); } + } finally { finish.release(); } + expect((await pending)?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).not.toBeNull(); + if (change === "pause") expect(config.pausedCodexAccountIds).toEqual(["manual-a"]); + }); + + test.each([false, true])("old usage cannot prove reset or overwrite a newer observation (old finishes first=%s)", async oldFirst => { + const config = setup(); const oldStarted = gate(); const oldFinish = gate(); + const freshStarted = gate(); const freshFinish = gate(); let reads = 0; + const urls = mock(() => Response.json({ code: "reset" }), async () => { + reads += 1; + if (reads === 1) { oldStarted.release(); await oldFinish.promise; return Response.json(usage(99)); } + freshStarted.release(); await freshFinish.promise; return Response.json(usage(12)); + }); + const frozenNow = Date.now(); + const clock = spyOn(Date, "now").mockReturnValue(frozenNow); + const old = listCodexAuthAccounts(config, true); + let reset: ReturnType | undefined; + try { + await oldStarted.promise; + reset = consume(config); + await freshStarted.promise; + if (oldFirst) { + oldFinish.release(); await old; + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).not.toBeNull(); + } + freshFinish.release(); + expect((await reset)?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toBeNull(); + oldFinish.release(); await old; + expect(getAccountQuota("manual-a")?.weeklyPercent).toBe(12); + expect(urls).toEqual([USAGE, CONSUME, USAGE]); + } finally { + oldFinish.release(); freshFinish.release(); + await old; if (reset) await reset; + clock.mockRestore(); + } + }); + + // Adapt #3995/e172453052's two-flight convergence to fresh-before-old scheduling. + test("reset publishes a fourth usage request before two old current-generation flights complete", async () => { + const config = setup(); + const oldCredential = getCodexAccountCredential("manual-a")!; + const oldGeneration = readCodexAccountRecord("manual-a")!.generation; + const firstStarted = gate(); const release401 = gate(); const secondStarted = gate(); const secondFinish = gate(); + const replayStarted = gate(); const replayFinish = gate(); const freshStarted = gate(); + const latches = [firstStarted, release401, secondStarted, secondFinish, replayStarted, replayFinish, freshStarted]; + const pending: Promise[] = []; + const urls: string[] = []; const usageBearers: Array = []; + let usageCalls = 0; let consumeCalls = 0; let completedOldResponses = 0; + let rejectDeadline!: (error: Error) => void; + const deadline = new Promise((_resolve, reject) => { rejectDeadline = reject; }); + // Failure bound only: success is synchronized on dispatch latches, never elapsed time. + const timeout = setTimeout(() => rejectDeadline(new Error("mock dispatch did not reach its expected phase")), watchdogMs(10_000)); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); urls.push(url); + if (url === CONSUME) { consumeCalls += 1; return Response.json({ code: "reset" }); } + if (url !== USAGE) throw new Error("unexpected mock URL"); + usageBearers.push(new Headers(init?.headers).get("Authorization")); + switch (++usageCalls) { + case 1: + firstStarted.release(); await release401.promise; + return new Response("{}", { status: 401 }); + case 2: + secondStarted.release(); await secondFinish.promise; completedOldResponses += 1; + return Response.json({ ...usage(88), rate_limit_reset_credits: { available_count: 66 } }); + case 3: + replayStarted.release(); await replayFinish.promise; completedOldResponses += 1; + return Response.json({ ...usage(99), rate_limit_reset_credits: { available_count: 77 } }); + case 4: + freshStarted.release(); return Response.json(usage(12)); + default: throw new Error("unexpected mock usage dispatch"); + } + }) as typeof fetch; + try { + const first = listCodexAuthAccounts(config, true); pending.push(first); + void first.catch(rejectDeadline); + await Promise.race([firstStarted.promise, deadline]); + // A fresh external generation starts its own ordinary flight while P's old 401 is held. + saveCodexAccountCredential("manual-a", { ...oldCredential, accessToken: "converged-access", refreshToken: "converged-refresh" }); + expect(readCodexAccountRecord("manual-a")!.generation).toBe(oldGeneration + 1); + const second = listCodexAuthAccounts(config, true); pending.push(second); + void second.catch(rejectDeadline); + await Promise.race([secondStarted.promise, deadline]); + release401.release(); + await Promise.race([replayStarted.promise, deadline]); + // Both old flights now use the current generation; neither response has completed. + expect(usageBearers).toEqual([`Bearer ${oldCredential.accessToken}`, "Bearer converged-access", "Bearer converged-access"]); + expect(completedOldResponses).toBe(0); + const reset = consume(config); pending.push(reset); + void reset.catch(rejectDeadline); + await Promise.race([freshStarted.promise, deadline]); + const response = await Promise.race([reset, deadline]); + expect(response?.status).toBe(200); + expect(await response?.json()).toEqual({ code: "reset", remaining: 2 }); + expect(completedOldResponses).toBe(0); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toBeNull(); + const fresh = structuredClone(getAccountQuota("manual-a")); + expect(fresh).toMatchObject({ weeklyPercent: 12, resetCredits: 2 }); + secondFinish.release(); replayFinish.release(); + await Promise.all([first, second]); + expect(completedOldResponses).toBe(2); + expect(getAccountQuota("manual-a")).toEqual(fresh); + expect(getCodexQuotaHealthSnapshot("manual-a", "shared")).toBeNull(); + expect(consumeCalls).toBe(1); expect(usageCalls).toBe(4); + expect(usageBearers).toEqual([`Bearer ${oldCredential.accessToken}`, "Bearer converged-access", "Bearer converged-access", "Bearer converged-access"]); + expect(urls).toEqual([USAGE, USAGE, USAGE, CONSUME, USAGE]); + } finally { + clearTimeout(timeout); + for (const latch of latches) latch.release(); + const results = await Promise.allSettled(pending); + globalThis.fetch = originalFetch; + for (const result of results) if (result.status === "rejected") throw result.reason; + } + }, 60_000); + + test("main Q-first/P-last publication preserves post-reset cache, credits and hard-lock readiness", async () => { + const config = makeConfig({ codexMainAccountHardLock: true }); + const accessToken = "ordered-main-token"; const accountId = "ordered-main-account"; + writeFileSync(join(TEST_CODEX_HOME, "auth.json"), JSON.stringify({ tokens: { access_token: accessToken, account_id: accountId } })); + reconcileMainCodexAccountRuntimeState(); + const writer = observeMainQuotaCredential(accessToken, accountId)!; + setAccountQuotaFromParsed(MAIN_CODEX_ACCOUNT_ID, { weeklyPercent: 100, resetCredits: 5 }, captureConfigGeneration(), writer); + expect(getMainAccountHardLockStatus(config).state).toBe("blocked"); + cool(config, MAIN_CODEX_ACCOUNT_ID); + const oldStarted = gate(); const oldFinish = gate(); let reads = 0; + const urls = mock(() => Response.json({ code: "reset" }), () => { + if (++reads === 1) return new Response(new ReadableStream({ + async start(controller) { + oldStarted.release(); await oldFinish.promise; + controller.enqueue(new TextEncoder().encode(JSON.stringify({ ...usage(100), + rate_limit_reset_credits: { available_count: 7 } }))); + controller.close(); + }, + }), { headers: { "Content-Type": "application/json" } }); + if (reads === 2) return Response.json(usage(12)); + // Later omission also verifies the private retained-credit slot was not overwritten by P. + return Response.json({ plan_type: "team", rate_limit: { secondary_window: { used_percent: 14 } } }); + }); + const old = fetchMainAccountInfoSnapshot(true); + try { + await oldStarted.promise; + const reset = await consume(config, MAIN_CODEX_ACCOUNT_ID); + expect(await reset?.json()).toEqual({ code: "reset", remaining: 2 }); + expect(getCodexQuotaHealthSnapshot(MAIN_CODEX_ACCOUNT_ID, "shared")).toBeNull(); + const freshCache = structuredClone(getMainAccountInfoCache()); + const freshShared = structuredClone(getAccountQuota(MAIN_CODEX_ACCOUNT_ID)); + const freshPolicy = structuredClone(getMainPolicyQuota()); + expect(freshCache?.quota).toMatchObject({ weeklyPercent: 12, resetCredits: 2 }); + expect(getMainAccountHardLockStatus(config).state).toBe("ready"); + oldFinish.release(); + const stale = await old; + expect(stale.quotaRefresh).toBeUndefined(); + expect(getMainAccountInfoCache()).toEqual(freshCache); + expect(getAccountQuota(MAIN_CODEX_ACCOUNT_ID)).toEqual(freshShared); + expect(getMainPolicyQuota()).toEqual(freshPolicy); + expect(getMainAccountHardLockStatus(config).state).toBe("ready"); + const displayed = (await listCodexAuthAccounts(config, false)).find(account => account.isMain)!; + expect(displayed.quota).toMatchObject({ weeklyPercent: 12, resetCredits: 2 }); + await fetchMainAccountInfoSnapshot(true); + const afterOmission = (await listCodexAuthAccounts(config, false)).find(account => account.isMain)!; + expect(afterOmission.quota?.resetCredits).toBe(2); + expect(getMainAccountHardLockStatus(config).state).toBe("ready"); + expect(urls).toEqual([USAGE, CONSUME, USAGE, USAGE]); + } finally { oldFinish.release(); await old; } + }); + + test("a newer failed main read does not outrank an older successful publication", async () => { + writeFileSync(join(TEST_CODEX_HOME, "auth.json"), JSON.stringify({ + tokens: { access_token: "publication-main-token", account_id: "publication-main-account" }, + })); + reconcileMainCodexAccountRuntimeState(); + const started = gate(); const finish = gate(); let reads = 0; + const urls = mock(() => { throw new Error("consume is not expected"); }, async () => { + if (++reads === 1) { started.release(); await finish.promise; return Response.json(usage()); } + return new Response("fixture unavailable", { status: 503 }); + }); + const old = fetchMainAccountInfoSnapshot(true); + try { + await started.promise; + expect((await fetchMainAccountInfoSnapshot(true)).quotaRefresh).toEqual({ status: "http_error", httpStatus: 503 }); + finish.release(); + expect((await old).quotaRefresh).toEqual({ status: "ok" }); + expect(getMainAccountInfoCache()?.quota).toMatchObject({ weeklyPercent: 12, resetCredits: 2 }); + expect(getMainPolicyQuota()?.weeklyPercent).toBe(12); + expect(getMainAccountHardLockStatus({ codexMainAccountHardLock: true }).state).toBe("ready"); + expect(urls).toEqual([USAGE, USAGE]); + } finally { finish.release(); await old; } + }); + + test("main reset usage does not erase an existing reauth quarantine", async () => { + const config = makeConfig(); + writeFileSync(join(TEST_CODEX_HOME, "auth.json"), JSON.stringify({ + tokens: { access_token: "manual-main-token", account_id: "manual-main-account" }, + })); + reconcileMainCodexAccountRuntimeState(); + cool(config, MAIN_CODEX_ACCOUNT_ID); + markAccountNeedsReauth(MAIN_CODEX_ACCOUNT_ID); + const urls = mock(() => Response.json({ code: "reset" }), () => Response.json(usage())); + expect((await consume(config, MAIN_CODEX_ACCOUNT_ID))?.status).toBe(200); + expect(isAccountNeedsReauth(MAIN_CODEX_ACCOUNT_ID)).toBe(true); + expect(getCodexQuotaHealthSnapshot(MAIN_CODEX_ACCOUNT_ID, "shared")).not.toBeNull(); + expect(urls).toEqual([CONSUME, USAGE]); + }); + + test("conflicting main token/header identity supplies no recovery proof", async () => { + const config = makeConfig(); + const payload = Buffer.from(JSON.stringify({ "https://api.openai.com/auth": { chatgpt_account_id: "token-account" } })).toString("base64url"); + writeFileSync(join(TEST_CODEX_HOME, "auth.json"), JSON.stringify({ + tokens: { access_token: "manual-main-token", id_token: `e30.${payload}.sig`, account_id: "header-account" }, + })); + reconcileMainCodexAccountRuntimeState(); cool(config, MAIN_CODEX_ACCOUNT_ID); + const urls = mock(() => Response.json({ code: "reset" }), () => Response.json(usage())); + expect(await (await consume(config, MAIN_CODEX_ACCOUNT_ID))?.json()).toEqual({ code: "reset" }); + expect(getCodexQuotaHealthSnapshot(MAIN_CODEX_ACCOUNT_ID, "shared")).not.toBeNull(); + expect(urls).toEqual([CONSUME]); + }); + + test.each(["same", "bearer", "other-account", "aba"])("main recovery uses its own live credential proof: %s", async change => { + const config = makeConfig(); + const writeMain = (accountId: string, accessToken = "manual-main-token") => { + writeFileSync(join(TEST_CODEX_HOME, "auth.json"), JSON.stringify({ tokens: { access_token: accessToken, account_id: accountId } })); + }; + writeMain("manual-main-account"); reconcileMainCodexAccountRuntimeState(); + cool(config, MAIN_CODEX_ACCOUNT_ID); + const started = gate(); const finish = gate(); let usageCalls = 0; + mock(() => Response.json({ code: "reset" }), async () => { + usageCalls += 1; + if (usageCalls === 1) { started.release(); await finish.promise; } + return Response.json(usage()); + }); + const pending = consume(config, MAIN_CODEX_ACCOUNT_ID); + try { + await started.promise; + expect(getNativeMainProfileRequestCount()).toBe(1); + if (change === "bearer") writeMain("manual-main-account", "replacement-main-token"); + if (change === "other-account" || change === "aba") { + writeMain("other-main-account"); reconcileMainCodexAccountRuntimeState(); + if (change === "aba") { writeMain("manual-main-account"); reconcileMainCodexAccountRuntimeState(); } + cool(config, MAIN_CODEX_ACCOUNT_ID); + } + } finally { finish.release(); } + expect((await pending)?.status).toBe(200); + expect(getCodexQuotaHealthSnapshot(MAIN_CODEX_ACCOUNT_ID, "shared") === null).toBe(change === "same"); + expect(getNativeMainProfileRequestCount()).toBe(0); + }); +}); diff --git a/tests/codex-integration/codex-cooldown-recovery.test.ts b/tests/codex-integration/codex-cooldown-recovery.test.ts index a196adcfe8..c898d4ee92 100644 --- a/tests/codex-integration/codex-cooldown-recovery.test.ts +++ b/tests/codex-integration/codex-cooldown-recovery.test.ts @@ -7,7 +7,7 @@ import { runCodexCooldownRecoveryProbes, seedCodexAuthAdmissionForTests, } from "../../src/codex/auth-api"; -import { saveCodexAccountCredential } from "../../src/codex/account-store"; +import { readCodexAccountRecord, saveCodexAccountCredential, saveCodexAccountCredentialIfGeneration } from "../../src/codex/account-store"; import { codexQuotaWindowForPlan, getAccountQuota, @@ -20,6 +20,11 @@ import upstreamModels from "../../src/codex/data/upstream-models.json"; import { CODEX_QUOTA_PROBE_INTERVAL_MS, clearCodexUpstreamHealth, + clearCodexUpstreamHealthForAccount, + claimDueCodexQuotaRecoveryProbes, + claimManualResetCooldowns, + settleCodexQuotaRecoveryProbe, + settleManualResetCooldown, getCodexQuotaHealthSnapshot, recordCodexUpstreamOutcome, resolveCodexAccountForThread, @@ -105,6 +110,78 @@ describe("Codex cooldown recovery worker", () => { if (existsSync(TEST_DIR)) removeTreeWithRetry(TEST_DIR); }); + test("manual reset bypasses pacing but does not steal a live background lease", () => { + const config = makeConfig(["a"]); saveCredential("a"); cool(config, "a"); + const manual = claimManualResetCooldowns(config, "a", START + 1); + expect(manual).toHaveLength(1); + expect(claimDueCodexQuotaRecoveryProbes(config, 1, due())).toEqual([]); + settleManualResetCooldown(config, manual[0]!, false, {}, START + 2); + const [background] = claimDueCodexQuotaRecoveryProbes(config, 1, due(START + 2)); + expect(background).toBeDefined(); + expect(claimManualResetCooldowns(config, "a", due(START + 2))).toEqual([]); + expect(settleManualResetCooldown(config, manual[0]!, false, {}, due(START + 2))).toBe(false); + expect(settleCodexQuotaRecoveryProbe(background!, true, { + credentialGeneration: readCodexAccountRecord("a")!.generation, + }, due(START + 2))).toBe(true); + }); + + test("manual recovery rejects an unrelated refresh edge and preserves exact-generation settlement", () => { + const config = makeConfig(["a"]); saveCredential("a"); cool(config, "a"); + const [claim] = claimManualResetCooldowns(config, "a", START); + expect(claim?.kind).toBe("pool"); + const before = readCodexAccountRecord("a")!; + const generation = before.generation; + expect(saveCodexAccountCredentialIfGeneration("a", generation, { + ...before.credential!, accessToken: "fresh-a", refreshToken: "fresh-refresh-a", + })).toBe(true); + expect(readCodexAccountRecord("a")!.replacedAt).toBe(before.replacedAt); + expect(settleManualResetCooldown(config, claim!, true, { + credentialGeneration: generation + 1, + refreshLineage: { fromGeneration: generation - 1, toGeneration: generation + 1, provenance: "self-refresh" }, + }, START)).toBe(false); + expect(getCodexQuotaHealthSnapshot("a", "shared", START)).not.toBeNull(); + // Rejection releases the lease rather than leaving manual recovery stuck. + const [exact] = claimManualResetCooldowns(config, "a", START); + expect(exact).toBeDefined(); + expect(settleManualResetCooldown(config, exact!, true, { credentialGeneration: generation + 1 }, START)).toBe(true); + }); + + test("a replacement between auth and claiming cannot acquire the replacement's cooldown", () => { + const config = makeConfig(["a"]); saveCredential("a"); cool(config, "a"); + const generation = readCodexAccountRecord("a")!.generation; + saveCredential("a", "-replacement"); + expect(claimManualResetCooldowns(config, "a", START, generation)).toEqual([]); + expect(getCodexQuotaHealthSnapshot("a", "shared", START)).not.toBeNull(); + }); + + test("deleted and recreated cooldown with identical generation and clock cannot reuse a manual lease", () => { + const config = makeConfig(["a"]); saveCredential("a"); cool(config, "a"); + const [old] = claimManualResetCooldowns(config, "a", START); + clearCodexUpstreamHealthForAccount("a"); cool(config, "a"); + const [replacement] = claimManualResetCooldowns(config, "a", START); + expect(replacement!.probe.cooldownGeneration).toBe(old!.probe.cooldownGeneration); + expect(replacement!.probe.leaseId).not.toBe(old!.probe.leaseId); + const proof = { credentialGeneration: readCodexAccountRecord("a")!.generation }; + expect(settleManualResetCooldown(config, old!, true, proof, START)).toBe(false); + expect(getCodexQuotaHealthSnapshot("a", "shared", START)).not.toBeNull(); + expect(settleManualResetCooldown(config, replacement!, true, proof, START)).toBe(true); + }); + + test.each(["retry-after", "default", "spark", "reserve", "paused", "missing"])( + "manual reset never claims an ineligible target: %s", kind => { + const config = makeConfig(["a"]); saveCredential("a"); + if (kind === "retry-after") recordCodexUpstreamOutcome(config, "a", 429, { now: START, retryAfter: "3600" }); + else if (kind === "default") recordCodexUpstreamOutcome(config, "a", 429, { now: START }); + else if (kind === "reserve") recordCodexUpstreamOutcome(config, "a", 429, { + now: START, resetAt: START + 3_600_000, modelId: "gpt-reserve", + }); + else cool(config, "a", kind === "spark" ? "spark" : "shared"); + if (kind === "paused") config.pausedCodexAccountIds = ["a"]; + if (kind === "missing") config.codexAccounts = []; + expect(claimManualResetCooldowns(config, "a", START + 1)).toEqual([]); + }, + ); + test("recovers cooled A independently while ordinary routing only selects B", async () => { const config = makeConfig(); saveCredential("a");