Skip to content

Org-wide OpenCode model-pool outage: all 3 provider families down since 2026-07-25 — human credential/budget action required #628

Description

@seonghobae

Summary

Every opencode-review model-pool run across the org has ended OPENCODE_MODEL_POOL_OUTCOME: exhausted since ~2026-07-25T00:30Z (~34h). The last real model review completed 2026-07-24T23:42:52Z (run 30134503832, 43.5m). Later "success" runs (e.g. 30193838586 at 08:01Z, 30196130362 for #579) passed only via pre-existing same-head approvals — their pools were also exhausted. Approved-but-unreviewed PRs (e.g. #618 at head c4a9023) cannot merge until this is fixed.

Per-provider signature (human action needed)

Provider family Failure class Evidence Action
github-models (majority of pool) authentication-or-permission (~5s fatal per candidate) runs 30153577309 (att. 21), 30195391225, 30196418659, 30196803994 (naruon#1081: 24× auth class) STRIX_GITHUB_MODELS_TOKEN secret (PAT, models: read) is rejected — likely expired. Rotate/renew the org/repo secret. Fallback github.token is also failing, so check GitHub Models org entitlement/billing too.
openai gpt-5.6-luna quota-or-budget same runs; persists >34h across a daily reset boundary Hard budget cap reached — raise budget or wait for the billing-cycle reset.
openrouter (deepseek-v3.2, qwen3-coder) credit-exhausted (HTTP 402) same runs; root cause documented in merged #621: runaway runs 30120972549 (~3h × 102 pool cycles) and 30122360773 (700+ attempts into 402) burned the entire org credit Top up OpenRouter credits. Spend guards from #621 now bound future burn.

Org-wide confirmation (2026-07-26 09:29–09:44Z batch, all identical signature)

kaefa#70, html4tree#178, pg-erd-cloud#596, clearfolio#161, bandscope#645, newsdom-api#401, naruon#1081, fast-mlsirm#259 — all failure, all provider classes as above.

What was already done autonomously

  • PR Handle stacked PR retarget and merge-queue trigger gaps #618 adversarial review: no PR-caused defects; every other required check green at head c4a9023; stale auto-merge (from 07-25) cleared via scheduler dispatch 30195149878.
  • Same-head OpenCode re-dispatched twice (runs 30195391225, 30196418659) per the "opencode-review failures are infrastructure → re-run only" rule; both exhausted.
  • No timing/gate changes were made (fixed rule: review timing untouched; no required-check weakening).

Self-healing after fix

No code change needed. Scheduled scans (:00/:30 repo, :15/:45 org) re-dispatch stale OpenCode reviews automatically once providers work again; approved heads then auto-merge. For an immediate retry: gh api repos/ContextualWisdomLab/.github/dispatches -f event_type=merge-scheduler -F 'client_payload[pr_number]=618' -F 'client_payload[trigger_reviews]=true' -F 'client_payload[enable_auto_merge]=true' (note: omitting the boolean fields makes GitHub expressions coerce them to false — see gotcha below).

Scheduler payload gotcha (worth a follow-up docs/test PR)

pr-review-merge-scheduler.yml maps client_payload.trigger_reviews != false → GitHub expressions coerce absent (null) to false via numeric coercion, so null != false is false. A bare client_payload[pr_number]=N dispatch runs with TRIGGER_REVIEWS/ENABLE_AUTO_MERGE/UPDATE_BRANCHES: false (observed: run 30195265329 → decision block; run 30195382302 with explicit trigger_reviews=truereview_dispatch).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions