diff --git a/docs-site/src/content/docs/fr/reference/configuration/agents.md b/docs-site/src/content/docs/fr/reference/configuration/agents.md index dace4ce567..2dfdd19a90 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/agents.md +++ b/docs-site/src/content/docs/fr/reference/configuration/agents.md @@ -58,7 +58,7 @@ Pour un tour enfant créé, l’ordre de repli est le suivant : Les chaînes de repli propres à un rôle doivent résider dans la configuration d’opencodex. L’ajout de `model_fallback` dans `$CODEX_HOME/agents/*.toml` amène Codex 0.146+ à rejeter le fichier de rôle entier à cause de ce champ inconnu, puis à ignorer le rôle (#1190). Une ancienne ligne `model_fallback` dans le fichier TOML reste lue par souci de rétrocompatibilité, mais `ocx doctor` la signale. -opencodex ignore les candidats désactivés, non routables, en mauvais état, en période de temporisation ou ayant atteint le seuil de quota. L’instantané de disponibilité est mis en cache pendant `subagentModelFallbackPollMs`. Les tâches enfants chiffrées limitent la chaîne aux cibles ChatGPT natives canoniques et aux routes Responses directes avec authentification par clé explicitement approuvées via `allowEncryptedV2AgentTasks: true` ; si aucune ne peut consommer la charge chiffrée, la requête échoue au lieu d’envoyer un texte chiffré illisible à une autre destination. Les combos restent limités aux cibles natives canoniques. +opencodex ignore les candidats désactivés, non routables, en mauvais état, en période de temporisation ou ayant atteint le seuil de quota. L’instantané de disponibilité est mis en cache pendant `subagentModelFallbackPollMs`. Les tâches enfants chiffrées limitent la chaîne aux cibles ChatGPT natives canoniques et aux routes Responses directes avec authentification par clé explicitement approuvées via `allowEncryptedV2AgentTasks: true` ; si aucune ne peut consommer la charge chiffrée, la requête échoue au lieu d’envoyer un texte chiffré illisible à une autre destination. Un combo essaie d’abord une cible native canonique disponible ; si aucune n’est sélectionnable et que `agentTaskRecovery` est activé, un `NEW_TASK` chiffré est récupéré une fois avant l’envoi routé du combo. ```json { @@ -111,7 +111,7 @@ Ce mécanisme ne protège pas contre un autre processus exécuté sous le même N’activez cette option que si la requête authentifiée supplémentaire, la consommation de quota, la présence de texte en clair dans le processus et la dépendance à un service privé sont acceptables. Dans le cas contraire, privilégiez un enfant ChatGPT natif ou une délégation hétérogène v1. -Ce mécanisme de récupération s’applique aux enfants routés directement. Au maximum 32 requêtes de récupération peuvent être actives simultanément ; toute absence supplémentaire dans le cache échoue de manière sûre. Pour les tâches chiffrées, le routage par combinaison conserve son filtre existant limité aux cibles natives et n’utilise pas la récupération. +Ce mécanisme de récupération s’applique aux enfants routés directement et aux `NEW_TASK` chiffrés d’un combo. Au maximum 32 requêtes de récupération peuvent être actives simultanément ; toute absence supplémentaire dans le cache échoue de manière sûre. Un combo disposant d’une cible native canonique disponible continue d’envoyer directement le texte chiffré ; la récupération ne s’exécute que si aucune cible native n’est sélectionnable. Un échec de récupération, l’épuisement des cibles ou leur indisponibilité conserve l’échec fermé sans transmettre le texte chiffré à un fournisseur routé. ## Plafonds d’effort diff --git a/docs-site/src/content/docs/ja/reference/configuration/agents.md b/docs-site/src/content/docs/ja/reference/configuration/agents.md index f0453461a4..2b185b81c4 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/agents.md +++ b/docs-site/src/content/docs/ja/reference/configuration/agents.md @@ -53,7 +53,7 @@ V1 ガイダンスは、`max` または `ultra` でのみプロアクティブ 拒否し、ロールをスキップします（#1190）。TOML 内のレガシー `model_fallback` 行は後方互換性の ために引き続き読み取られますが、`ocx doctor` がそれをフラグ付けします。 -opencodex は、無効、ルーティング不能、異常、冷却期間、またはクォータしきい値の候補をスキップします。可用性スナップショットは `subagentModelFallbackPollMs` に対してキャッシュされます。暗号化された子タスクでは、チェーンを正規のネイティブ ChatGPT ターゲットと、`allowEncryptedV2AgentTasks: true` で明示的に信頼された直接のキー認証 Responses ルートに制限します。暗号化されたペイロードを処理できる対象がない場合、読み取り不可能な暗号文を別の場所へ送らず、リクエストは失敗します。コンボは引き続き正規のネイティブ対象だけを使用します。 +opencodex は、無効、ルーティング不能、異常、冷却期間、またはクォータしきい値の候補をスキップします。可用性スナップショットは `subagentModelFallbackPollMs` に対してキャッシュされます。暗号化された子タスクでは、チェーンを正規のネイティブ ChatGPT ターゲットと、`allowEncryptedV2AgentTasks: true` で明示的に信頼された直接のキー認証 Responses ルートに制限します。暗号化されたペイロードを処理できる対象がない場合、読み取り不可能な暗号文を別の場所へ送らず、リクエストは失敗します。コンボはまず利用可能な正規ネイティブ対象を試し、選択できるネイティブ対象がなく `agentTaskRecovery` が有効な場合、暗号化された `NEW_TASK` をルーティングされたコンボ送信の前に一度だけ復旧します。 ```json { diff --git a/docs-site/src/content/docs/ko/reference/configuration/agents.md b/docs-site/src/content/docs/ko/reference/configuration/agents.md index ec5764bbd2..1c999536f2 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/agents.md +++ b/docs-site/src/content/docs/ko/reference/configuration/agents.md @@ -53,7 +53,7 @@ V1 안내는 `max` 또는 `ultra`에서만 선제 텍스트로 제공됩니다. 거부하고 역할을 건너뜁니다 (#1190). TOML의 기존 `model_fallback` 줄은 하위 호환성을 위해 계속 읽히지만 `ocx doctor`가 이를 표시합니다. -opencodex는 비활성, 라우팅 불가, 비정상, 쿨다운 중, 또는 할당량 임계값에 걸린 후보를 건너뜁니다. 사용 가능성 스냅샷은 `subagentModelFallbackPollMs` 동안 캐시됩니다. 암호화된 하위 작업은 정규 네이티브 ChatGPT 대상과 `allowEncryptedV2AgentTasks: true`로 명시적으로 신뢰한 직접 키 인증 Responses 라우트만 후보로 사용합니다. 암호화된 페이로드를 처리할 수 있는 대상이 없으면 읽을 수 없는 암호문을 다른 곳으로 보내지 않고 요청이 실패합니다. 콤보는 계속 정규 네이티브 대상만 사용합니다. +opencodex는 비활성, 라우팅 불가, 비정상, 쿨다운 중, 또는 할당량 임계값에 걸린 후보를 건너뜁니다. 사용 가능성 스냅샷은 `subagentModelFallbackPollMs` 동안 캐시됩니다. 암호화된 하위 작업은 정규 네이티브 ChatGPT 대상과 `allowEncryptedV2AgentTasks: true`로 명시적으로 신뢰한 직접 키 인증 Responses 라우트만 후보로 사용합니다. 암호화된 페이로드를 처리할 수 있는 대상이 없으면 읽을 수 없는 암호문을 다른 곳으로 보내지 않고 요청이 실패합니다. 콤보는 먼저 사용 가능한 정규 네이티브 대상을 시도하고, 선택 가능한 네이티브 대상이 없으며 `agentTaskRecovery`가 켜져 있으면 암호화된 `NEW_TASK`를 라우팅된 콤보 전송 전에 한 번 복구합니다. ```json { diff --git a/docs-site/src/content/docs/reference/configuration/agents.md b/docs-site/src/content/docs/reference/configuration/agents.md index 8b1c536032..03741e013f 100644 --- a/docs-site/src/content/docs/reference/configuration/agents.md +++ b/docs-site/src/content/docs/reference/configuration/agents.md @@ -117,8 +117,9 @@ opencodex skips disabled, unroutable, unhealthy, cooling-down, or quota-threshol availability snapshot is cached for `subagentModelFallbackPollMs`. Encrypted child tasks restrict the chain to canonical native ChatGPT targets plus direct key-auth Responses routes explicitly trusted with `allowEncryptedV2AgentTasks: true`; if none can consume the encrypted payload, the -request fails instead of routing unreadable ciphertext elsewhere. Combo routing remains -canonical-native-only. +request fails instead of routing unreadable ciphertext elsewhere. Combo routing first tries an +available canonical native target; when none is selectable and `agentTaskRecovery` is enabled, +an encrypted `NEW_TASK` is recovered once before routed combo dispatch. ```json { @@ -203,9 +204,11 @@ Enable this only when the additional authenticated request, quota use, plaintext and private-backend dependency are acceptable. Prefer a native ChatGPT child or v1 heterogeneous delegation when they are not. -This recovery path applies to direct-routed children. At most 32 recovery requests can be active at -once; additional misses fail closed. Combo routing keeps its existing native-only filter for -encrypted tasks and does not invoke recovery. +This recovery path applies to direct-routed children and encrypted combo `NEW_TASK` spawns. At +most 32 recovery requests can be active at once; additional misses fail closed. A combo with an +available canonical native target still sends ciphertext directly; recovery runs only when no +native target is selectable. Failed recovery, exhausted targets, or unavailable targets still fail +closed without forwarding ciphertext to a routed provider. ## Effort caps diff --git a/docs-site/src/content/docs/ru/reference/configuration/agents.md b/docs-site/src/content/docs/ru/reference/configuration/agents.md index 1b8def013e..a3a0bc434b 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/agents.md +++ b/docs-site/src/content/docs/ru/reference/configuration/agents.md @@ -84,7 +84,8 @@ cooldown либо уже достигли порога quota. Availability-сн native ChatGPT-target'ами и прямыми key-auth Responses-маршрутами, явно доверенными через `allowEncryptedV2AgentTasks: true`. Если ни один из них не может обработать encrypted payload, запрос завершается ошибкой вместо отправки нечитаемого ciphertext наружу. Combo по-прежнему -использует только канонические native-цели. +сначала выбирает доступную каноническую native-цель; если её нельзя выбрать и включён +`agentTaskRecovery`, encrypted `NEW_TASK` восстанавливается один раз перед routed combo dispatch. ```json { diff --git a/docs-site/src/content/docs/tr/reference/configuration/agents.md b/docs-site/src/content/docs/tr/reference/configuration/agents.md index 7b07247a73..3bf154f14b 100644 --- a/docs-site/src/content/docs/tr/reference/configuration/agents.md +++ b/docs-site/src/content/docs/tr/reference/configuration/agents.md @@ -122,7 +122,9 @@ görevlerinde zincir, kurallı yerel ChatGPT hedefleriyle ve `allowEncryptedV2AgentTasks: true` kullanılarak açıkça güvenilen doğrudan anahtar kimlik doğrulamalı Responses rotalarıyla sınırlıdır. Hiçbiri şifrelenmiş yükü işleyemezse istek, okunamayan şifreli metni başka bir yere yönlendirmek yerine -başarısız olur. Kombolar yalnızca kurallı yerel hedefleri kullanmaya devam eder. +başarısız olur. Kombo önce kullanılabilir kurallı yerel hedefi dener; seçilebilir +yerel hedef kalmazsa ve `agentTaskRecovery` etkinse, şifrelenmiş `NEW_TASK` yönlendirilen +kombo gönderiminden önce bir kez kurtarılır. ```json { @@ -226,10 +228,13 @@ sınırı ve özel arka uç bağımlılığı kabul edilebilir olduğunda etkinl Olmadıklarında yerel bir ChatGPT çocuğunu veya v1 heterojen yetkilendirmesini tercih edin. -Bu kurtarma yolu doğrudan yönlendirilen çocuklara uygulanır. Aynı anda en fazla -32 kurtarma isteği etkin olabilir; ek ıskalamalar kapalı olarak başarısız olur. -Kombo yönlendirmesi şifrelenmiş görevler için mevcut yalnızca yerel filtresini -korur ve kurtarmayı çağırmaz. +Bu kurtarma yolu doğrudan yönlendirilen çocuklara ve bir kombodaki şifrelenmiş +`NEW_TASK` oluşturma isteklerine uygulanır. Aynı anda en fazla 32 kurtarma isteği +etkin olabilir; ek ıskalamalar kapalı olarak başarısız olur. Kullanılabilir kanonik +yerel hedefi olan bir kombo şifreli metni yine doğrudan gönderir; kurtarma yalnızca +seçilebilir yerel hedef kalmadığında çalışır. Kurtarma hatası, tükenen hedefler veya +kullanılamayan hedefler, şifreli metin yönlendirilen sağlayıcıya gönderilmeden yine +kapalı biçimde başarısız olur. ## Çaba sınırları @@ -248,4 +253,3 @@ ile `xhigh` arasını sunar. v1, varsayılan ve v2 davranışının yeni başlayanlara yönelik açıklaması için [Alt ajan yüzeyleri](/tr/guides/sub-agent-surface/) sayfasına bakın. - diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/agents.md b/docs-site/src/content/docs/zh-cn/reference/configuration/agents.md index bcdb51cf05..238f13ac8a 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/agents.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/agents.md @@ -52,7 +52,7 @@ per-role fallback 链必须放在 opencodex 配置里。把 `model_fallback` 写 `$CODEX_HOME/agents/*.toml` 会让 Codex 0.146+ 把整个角色文件当作未知字段拒绝并跳过该角色 （#1190）。TOML 中的旧版 `model_fallback` 仍会被读取以保持向后兼容，但 `ocx doctor` 会标记它。 -opencodex 会跳过已禁用、不可路由、不健康、处于冷却中，或已达到配额阈值的候选项。可用性快照会在 `subagentModelFallbackPollMs` 期间缓存。对于加密的子任务，候选链只包含规范的原生 ChatGPT 目标，以及通过 `allowEncryptedV2AgentTasks: true` 明确信任的直接密钥认证 Responses 路由。如果没有任何目标能处理加密载荷，请求就会失败，而不是把不可读的密文路由到别处。combo 仍然只使用规范的原生目标。 +opencodex 会跳过已禁用、不可路由、不健康、处于冷却中，或已达到配额阈值的候选项。可用性快照会在 `subagentModelFallbackPollMs` 期间缓存。对于加密的子任务，候选链只包含规范的原生 ChatGPT 目标，以及通过 `allowEncryptedV2AgentTasks: true` 明确信任的直接密钥认证 Responses 路由。如果没有任何目标能处理加密载荷，请求就会失败，而不是把不可读的密文路由到别处。combo 会先尝试可用的规范原生目标；如果没有可选择的原生目标且已启用 `agentTaskRecovery`，会在路由到 combo 目标前对加密的 `NEW_TASK` 恢复一次。 ```json { diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/agents.md b/docs-site/src/content/docs/zh-tw/reference/configuration/agents.md index 87141789a1..15545db7ed 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/agents.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/agents.md @@ -50,7 +50,7 @@ V1 指引僅在 `max` 或 `ultra` 時為主動文字。V2 僅在存在偏好模 Codex 0.146+ 會將角色檔案中的 `model_fallback` 視為未知欄位並略過整個角色；`ocx doctor` 也會對此發出警告。因此新的角色級 fallback 應設定在 opencodex，而不是角色 TOML 中。 -opencodex 會跳過已停用、不可路由、不健康、冷卻中或達到配額閾值的候選項。可用性快取保存 `subagentModelFallbackPollMs`。對於加密的子任務，候選鏈僅包含規範的原生 ChatGPT 目標，以及透過 `allowEncryptedV2AgentTasks: true` 明確信任的直接金鑰驗證 Responses 路由。若無任何目標可處理加密 payload，請求會失敗，而不會將無法讀取的密文路由到別處。組合仍只使用規範的原生目標。 +opencodex 會跳過已停用、不可路由、不健康、冷卻中或達到配額閾值的候選項。可用性快取保存 `subagentModelFallbackPollMs`。對於加密的子任務，候選鏈僅包含規範的原生 ChatGPT 目標，以及透過 `allowEncryptedV2AgentTasks: true` 明確信任的直接金鑰驗證 Responses 路由。若無任何目標可處理加密 payload，請求會失敗，而不會將無法讀取的密文路由到別處。組合會先嘗試可用的規範原生目標；若沒有可選擇的原生目標且已啟用 `agentTaskRecovery`，會在路由到組合目標前對加密的 `NEW_TASK` 恢復一次。 ```json { diff --git a/src/combos/resolve.ts b/src/combos/resolve.ts index ae48650b0d..bd88e82244 100644 --- a/src/combos/resolve.ts +++ b/src/combos/resolve.ts @@ -1,6 +1,7 @@ import type { OcxComboTarget, OcxConfig } from "../types"; import { getCachedProviderQuota } from "../providers/quota-routing-cache"; import type { ProviderQuota } from "../providers/quota-types"; +import { isCanonicalOpenAiForwardProvider } from "../providers/openai-tiers"; import { sleepWithAbort } from "../lib/upstream-retry"; import { coolComboTarget, @@ -60,9 +61,13 @@ export class NoAvailableComboTargetsError extends Error { } } -function targetProviderIsUsable(config: OcxConfig, target: OcxComboTarget): boolean { - return Object.hasOwn(config.providers, target.provider) - && config.providers[target.provider]?.disabled !== true; +function targetProviderIsUsable(config: OcxConfig, target: OcxComboTarget, now: number): boolean { + if (!Object.hasOwn(config.providers, target.provider)) return false; + const provider = config.providers[target.provider]; + if (!provider || provider.disabled === true) return false; + // Native account selection owns model-scoped quota; a provider summary cannot veto it. + return isCanonicalOpenAiForwardProvider(provider) + || !cachedProviderQuotaIsExhausted(getCachedProviderQuota(target.provider, now), now); } function quotaWindowExhausted(percent: number | undefined, resetAt: number | undefined, now: number): boolean { @@ -159,8 +164,7 @@ export function pickComboTarget( const excluded = new Set(options.exclude ?? []); const now = options.now ?? Date.now(); const eligible = (target: Required): boolean => - targetProviderIsUsable(config, target) - && !cachedProviderQuotaIsExhausted(getCachedProviderQuota(target.provider, now), now) + targetProviderIsUsable(config, target, now) && !isComboTargetInCooldown(comboId, target, now) && !excluded.has(targetKey(target)) && (options.eligible?.(target) ?? true); @@ -338,8 +342,7 @@ export async function pickComboTargetWithWait( const combo = getCombo(config, comboId); if (!combo) throw new UnknownComboError(comboId); const waitingTargets = combo.targets.filter(target => - targetProviderIsUsable(config, target) - && !cachedProviderQuotaIsExhausted(getCachedProviderQuota(target.provider, now), now) + targetProviderIsUsable(config, target, now) && !excluded.has(targetKey(target)) && isComboTargetInCooldown(comboId, target, now) && (customEligible?.(target) ?? true), diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 0471617da0..c4c146d02e 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -2404,19 +2404,10 @@ export async function handleComboResponses( let comboPayloadReadable = false; const payloadEligible = (target: (typeof combo.targets)[number]): boolean => comboPayloadReadable || !unreadableEncryptedAgentTask || canDecryptUnreadableAgentTask(target); - const initialNow = Date.now(); - let pick: ReturnType = null; - const pickWithWait = (pickOptions: { - exclude?: Iterable; - eligible?: (target: NonNullable["targets"][number]) => boolean; - now?: number; - }) => pickComboTargetWithWait(config, comboId, { - ...pickOptions, - waitForCooldownMs: combo.waitForCooldownMs, - abortSignal: options.abortSignal, - }); - - if (unreadableEncryptedAgentTask && !combo.targets.some(canDecryptUnreadableAgentTask)) { + let encryptedTaskRecoveryAttempted = false; + const recoverUnreadableEncryptedTask = async (): Promise => { + if (encryptedTaskRecoveryAttempted) return false; + encryptedTaskRecoveryAttempted = true; const recovery = agentTaskRecoveryConfig(config); if ( (options.inboundWire ?? "responses") !== "responses" @@ -2430,19 +2421,7 @@ export async function handleComboResponses( config, { parentThreadId: inboundClientThreadId }, ); - return unreadableEncryptedAgentTaskResponse(); - } - pick = await pickWithWait({ now: initialNow }); - if (!pick) { - discardEncryptedAgentTaskRecovery( - req, - (body as { input?: unknown } | undefined)?.input, - config, - { parentThreadId: inboundClientThreadId }, - ); - return options.abortSignal?.aborted - ? clientCancelledResponse() - : comboUnavailable(comboId); + return false; } let recovered = false; try { @@ -2467,15 +2446,43 @@ export async function handleComboResponses( config, { parentThreadId: inboundClientThreadId }, ); - return unreadableEncryptedAgentTaskResponse(); + return false; } comboPayloadReadable = true; comboReplaySnapshot.recoveredPlaintext = true; - } else { - pick = await pickWithWait({ - eligible: payloadEligible, - now: initialNow, - }); + return true; + }; + const initialNow = Date.now(); + const pickWithWait = (pickOptions: { + exclude?: Iterable; + eligible?: (target: NonNullable["targets"][number]) => boolean; + now?: number; + }) => pickComboTargetWithWait(config, comboId, { + ...pickOptions, + waitForCooldownMs: combo.waitForCooldownMs, + abortSignal: options.abortSignal, + }); + let pick = await pickWithWait({ + eligible: payloadEligible, + now: initialNow, + }); + + if (unreadableEncryptedAgentTask && !pick) { + pick = await pickWithWait({ now: initialNow }); + if (!pick) { + discardEncryptedAgentTaskRecovery( + req, + (body as { input?: unknown } | undefined)?.input, + config, + { parentThreadId: inboundClientThreadId }, + ); + return options.abortSignal?.aborted + ? clientCancelledResponse() + : comboUnavailable(comboId); + } + if (!(await recoverUnreadableEncryptedTask())) { + return unreadableEncryptedAgentTaskResponse(); + } } if (!pick) { @@ -2686,6 +2693,7 @@ export async function handleComboResponses( `[combo] ${comboId}: ${targetKey(pick.target)} failed with ${failure.response.status} after ${Date.now() - started}ms`, ); const failureNow = Date.now(); + const attemptedTargets = pick.attempted; const nextPick = advanceComboAfterFailure(config, pick, { retryAfter: failure.retryAfter, resetAt: failure.resetAt, @@ -2710,6 +2718,16 @@ export async function handleComboResponses( } if (!pick) { if (options.abortSignal?.aborted) return clientCancelledResponse(); + if (unreadableEncryptedAgentTask && !comboPayloadReadable) { + const recoveredTarget = await pickWithWait({ + exclude: attemptedTargets, + now: failureNow, + }); + if (recoveredTarget && await recoverUnreadableEncryptedTask()) { + pick = recoveredTarget; + continue; + } + } adoptFailedChildLog(childLog); } } diff --git a/tests/codex-integration/combos.test.ts b/tests/codex-integration/combos.test.ts index 1c4924d3de..98174c3848 100644 --- a/tests/codex-integration/combos.test.ts +++ b/tests/codex-integration/combos.test.ts @@ -858,6 +858,101 @@ describe("combo failure policy and advancement", () => { expect(pick?.target.provider).toBe("b"); }); + test.each(["pool", "direct"] as const)("defers native %s quota decisions to account and model scoped authentication", mode => { + const now = 50_000; + const config = baseConfig({ + providers: { + a: { + adapter: "openai-responses", + authMode: "forward", + codexAccountMode: mode, + baseUrl: "https://chatgpt.com/backend-api/codex", + }, + b: { adapter: "openai-chat", baseUrl: "https://b.example/v1", apiKey: "kb" }, + }, + }); + setCachedProviderQuotaForTests("a", { weeklyPercent: 100, updatedAt: now }); + + const pick = pickComboTarget(config, "free", { now }); + + expect(pick?.target.provider).toBe("a"); + }); + + test("native provider summary quota does not suppress a bounded cooldown wait", async () => { + const now = 50_000; + const config = baseConfig({ + providers: { + a: { + adapter: "openai-responses", + authMode: "forward", + codexAccountMode: "pool", + baseUrl: "https://chatgpt.com/backend-api/codex", + }, + }, + combos: { + free: { + targets: [{ provider: "a", model: "m1" }], + waitForCooldownMs: 2_000, + }, + }, + }); + setCachedProviderQuotaForTests("a", { weeklyPercent: 100, updatedAt: now }); + coolComboTarget("free", { provider: "a", model: "m1" }, { now, cooldownMs: 1_000 }); + const sleeps: number[] = []; + + const pick = await pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 2_000, + sleep: async ms => { sleeps.push(ms); }, + }); + + expect(pick?.target.provider).toBe("a"); + expect(sleeps).toEqual([1_000]); + }); + + test("still filters exhausted quota on a noncanonical forward destination", () => { + const now = 50_000; + const config = baseConfig({ + providers: { + a: { + adapter: "openai-responses", + authMode: "forward", + codexAccountMode: "pool", + baseUrl: "https://chatgpt.com.example/backend-api/codex", + }, + b: { adapter: "openai-chat", baseUrl: "https://b.example/v1", apiKey: "kb" }, + }, + }); + setCachedProviderQuotaForTests("a", { weeklyPercent: 100, updatedAt: now }); + + const pick = pickComboTarget(config, "free", { now }); + + expect(pick?.target.provider).toBe("b"); + }); + + test("retains caller eligibility restrictions for native targets", () => { + const now = 50_000; + const config = baseConfig({ + providers: { + a: { + adapter: "openai-responses", + authMode: "forward", + codexAccountMode: "pool", + baseUrl: "https://chatgpt.com/backend-api/codex", + }, + b: { adapter: "openai-chat", baseUrl: "https://b.example/v1", apiKey: "kb" }, + }, + }); + setCachedProviderQuotaForTests("a", { weeklyPercent: 100, updatedAt: now }); + + const pick = pickComboTarget(config, "free", { + now, + eligible: target => target.provider !== "a", + }); + + expect(pick?.target.provider).toBe("b"); + }); + test("elapsed quota reset does not permanently blacklist a provider", () => { const now = 50_000; const config = baseConfig(); diff --git a/tests/server/agent-task-recovery-combo.test.ts b/tests/server/agent-task-recovery-combo.test.ts index 0e6e22a590..14d8ffcf69 100644 --- a/tests/server/agent-task-recovery-combo.test.ts +++ b/tests/server/agent-task-recovery-combo.test.ts @@ -10,6 +10,11 @@ import { } from "../../src/responses/state"; import { resetAgentTaskRecoveryState } from "../../src/server/responses/agent-task-recovery"; import { agentTaskRecoveryCacheSnapshotForTests } from "../../src/server/responses/agent-task-recovery-cache"; +import { clearComboTargetCooldowns, coolComboTarget } from "../../src/combos/failover"; +import { + clearCachedProviderQuotas, + setCachedProviderQuotaForTests, +} from "../../src/providers/quota-routing-cache"; import { codexHeaders, encryptedInput, @@ -55,11 +60,15 @@ describe("combo path encrypted agent task recovery", () => { process.env["OPENCODEX_HOME"] = home; clearResponseStateMemoryForTests(); resetAgentTaskRecoveryState(); + clearCachedProviderQuotas(); + clearComboTargetCooldowns(); }); afterEach(() => { globalThis.fetch = originalFetch; resetAgentTaskRecoveryState(); + clearCachedProviderQuotas(); + clearComboTargetCooldowns(); clearResponseStateForTests(); removeTreeWithRetry(home); if (priorHome === undefined) delete process.env["OPENCODEX_HOME"]; @@ -172,6 +181,122 @@ describe("combo path encrypted agent task recovery", () => { expect(providerFetches).toBe(1); }); + test.each(["disabled", "cooldown"] as const)("recovers a mixed combo when the native target is blocked by %s", async (reason) => { + const config = comboConfig([ + { provider: "xai", model: "grok-4.5" }, + { provider: "openai", model: "gpt-5.5" }, + ]); + if (reason === "disabled") { + config.providers.openai!.disabled = true; + } else { + coolComboTarget("routed", { provider: "openai", model: "gpt-5.5" }, { cooldownMs: 60_000 }); + } + const assignment = "MIXED-RECOVERY-PRIVATE-ASSIGNMENT"; + const recoveryBodies: string[] = []; + const forwardedBodies: string[] = []; + globalThis.fetch = (async (input, init) => { + const body = typeof init?.body === "string" ? init.body : ""; + if (String(input).includes("chatgpt.com")) { + recoveryBodies.push(body); + return new Response(recoverySse(assignment), { + status: 200, + headers: { "content-type": "text/event-stream" }, + }); + } + forwardedBodies.push(body); + return providerCompletion(); + }) as typeof fetch; + + const response = await post(config, "combo/routed", encryptedInput(), codexHeaders()); + await response.text(); + + expect(response.status).toBe(200); + expect(recoveryBodies).toHaveLength(1); + expect(forwardedBodies).toHaveLength(1); + expect(forwardedBodies[0]).toContain(assignment); + expect(forwardedBodies[0]).not.toContain(FERNET_TASK); + expect(responseContinuationRetainedStoreSnapshot().count).toBe(0); + }); + + test("fails closed without routed dispatch when mixed-combo recovery fails", async () => { + const config = comboConfig([ + { provider: "xai", model: "grok-4.5" }, + { provider: "openai", model: "gpt-5.5" }, + ]); + coolComboTarget("routed", { provider: "openai", model: "gpt-5.5" }, { cooldownMs: 60_000 }); + const urls: string[] = []; + globalThis.fetch = (async (input) => { + urls.push(String(input)); + return new Response("unavailable", { status: 503 }); + }) as typeof fetch; + + const response = await post(config, "combo/routed", encryptedInput(), codexHeaders()); + + expect(response.status).toBe(400); + expect(await response.json()).toMatchObject({ error: { code: "unreadable_encrypted_agent_task" } }); + expect(urls).toHaveLength(1); + expect(urls[0]).toContain("chatgpt.com/backend-api/codex/responses"); + expect(responseContinuationRetainedStoreSnapshot().count).toBe(0); + }); + + test("recovers once when the selected native target fails model authorization", async () => { + const config = comboConfig([ + { provider: "openai", model: "gpt-5.5" }, + { provider: "xai", model: "grok-4.5" }, + ]); + const assignment = "RECOVERED-AFTER-NATIVE-401"; + const chatgptBodies: string[] = []; + const forwardedBodies: string[] = []; + globalThis.fetch = (async (input, init) => { + const body = typeof init?.body === "string" ? init.body : ""; + if (String(input).includes("chatgpt.com")) { + chatgptBodies.push(body); + if (body.includes("capture_assignment")) { + return new Response(recoverySse(assignment), { + status: 200, + headers: { "content-type": "text/event-stream" }, + }); + } + return Response.json( + { error: { message: "model is not enabled for this account", code: "model_not_found" } }, + { status: 401 }, + ); + } + forwardedBodies.push(body); + return providerCompletion(); + }) as typeof fetch; + + const response = await post(config, "combo/routed", encryptedInput(), codexHeaders()); + await response.text(); + + expect(response.status).toBe(200); + expect(chatgptBodies).toHaveLength(2); + expect(chatgptBodies[0]).not.toContain("capture_assignment"); + expect(chatgptBodies[1]).toContain("capture_assignment"); + expect(forwardedBodies).toHaveLength(1); + expect(forwardedBodies[0]).toContain(assignment); + expect(forwardedBodies[0]).not.toContain(FERNET_TASK); + expect(responseContinuationRetainedStoreSnapshot().count).toBe(0); + }); + + test("does not recover when every mixed-combo target is unavailable", async () => { + const config = comboConfig([ + { provider: "xai", model: "grok-4.5" }, + { provider: "openai", model: "gpt-5.5" }, + ]); + coolComboTarget("routed", { provider: "openai", model: "gpt-5.5" }, { cooldownMs: 60_000 }); + setCachedProviderQuotaForTests("xai", { updatedAt: Date.now(), weeklyPercent: 100 }); + globalThis.fetch = (async () => { + throw new Error("No network call is permitted without an eligible execution target"); + }) as typeof fetch; + + const response = await post(config, "combo/routed", encryptedInput(), codexHeaders()); + + expect(response.status).toBe(503); + expect(await response.json()).toMatchObject({ error: { code: "combo_unavailable" } }); + expect(responseContinuationRetainedStoreSnapshot().count).toBe(0); + }); + test("keeps an opted-in Responses target out of encrypted combo dispatch", async () => { const config = comboConfig([ { provider: "relay", model: "relay-model" },