ci: refresh the model catalog daily via scheduled workflow

[eliebak]

Part of ENG-5435.

Context

The committed packages/ai/src/models.generated.ts snapshot only updates when someone manually runs npm run generate-models and opens a PR (#1445, #1481, and now #1632). Release builds regenerate from live catalogs anyway, so the checked-in snapshot silently drifts between manual refreshes. This automates the existing habit without removing human review.

Changes

New scheduled workflow .github/workflows/update-model-catalog.yml:

• Runs daily at 05:17 UTC (plus workflow_dispatch), regenerates the catalog, and opens a PR when the snapshot drifted. Reruns force-push the fixed bot/update-model-catalog branch, so at most one refresh PR is open at a time; a stale open PR just gets updated.
• Safety guard: generate-models exits 0 when a source fetch fails, and every source except Prime Inference falls back to an empty list rather than the committed snapshot. The job fails instead of opening a PR when the regenerated catalog lost more than 10% of its models versus the committed file, so a flaky fetch can't produce a "removed 300 models" PR.
• A human still merges: pricing changes, vendor renames (like the xai/ -> spacexai/ move in chore(ai): regenerate the model catalog from live provider catalogs #1632), and removals deserve eyeballs.

Interactions with existing checks, verified against current workflow configs:

• Contribution gate: bots are allowed automatically per .github/VOUCHED.td, so the gate won't auto-close the bot PR.
• Changelog fragment check: skips bot-authored PRs (user.type != 'Bot'), so no fragment is required on the automated PRs.

Validation

• YAML parse-checked. The regeneration + guard path is the same npm run generate-models flow validated in chore(ai): regenerate the model catalog from live provider catalogs #1632, where the guard condition (1260 vs 1252 committed, well above the 90% floor) passes.
• Known caveat, noted in the workflow: PRs opened with the default GITHUB_TOKEN don't trigger pull_request CI. Push to the branch or close/reopen to run CI, or swap in a PAT/App token if you want CI green automatically.

🤖 Generated with Claude Code

---

Note

Medium Risk
The workflow has contents and pull-requests write access and force-pushes a bot branch. Failures in generate-models can still produce incomplete catalogs unless the 10% drop guard fires.

Overview
Automates the previously manual refresh of packages/ai/src/models.generated.ts so the checked-in snapshot does not drift between human PRs.

A new scheduled workflow runs npm run generate-models daily (and on dispatch). If the catalog changed, it force-pushes bot/update-model-catalog and opens or updates a single PR for review. A guard fails the job if the regenerated file has fewer than 90% of the committed model count, which is meant to catch silent empty-list fallbacks from failed provider fetches.

PRs still need a human merge. Default GITHUB_TOKEN PRs do not run pull_request CI until close/reopen or a follow-up push.

^{Reviewed by Cursor Bugbot for commit 6365b24. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add daily scheduled workflow to regenerate models.generated.ts and open PR

• New GitHub Actions workflow runs daily at 05:17 UTC (and on manual dispatch) to regenerate the AI model catalog via npm run generate-models in packages/ai.
• If packages/ai/src/models.generated.ts changes, the workflow force-pushes branch bot/update-model-catalog and creates/updates a PR against main.
• Guards against provider failures by failing the job if the new satisfies Model count drops below 90% of the previous count.
• Workflow pins actions/checkout@v7.0.0 and actions/setup-node@v7.0.0 by SHA, runs on ubuntu-latest with a 20-minute timeout.
• Risk: PRs created with the default GITHUB_TOKEN will not trigger downstream CI checks; the PR body notes this limitation.

📊 Macroscope summarized 6365b24. 1 file reviewed, 1 issue evaluated, 1 issue filtered, 0 comments posted

🗂️ Filtered Issues

.github/workflows/update-model-catalog.yml — 0 comments posted, 1 evaluated, 1 filtered

• line 61: The aggregate 90% check does not detect a failed source that supplies fewer than 10% of all models. generate-models catches source-fetch errors and returns [] (for example, the Vercel AI Gateway fetch), so such a failure removes that provider's catalog yet still passes this test and opens/force-updates a PR containing the silent provider-wide deletion. Track or validate source/provider counts rather than only the global count. [ Failed validation ]

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 8c7c938. Configure here.}

[sethkarten]

Design and implementation reviewed. A human-reviewed scheduled snapshot refresh is a sound lower-risk fit for the checked-in fallback catalog. Commit 6365b24 fixes both review findings by always checking out the default branch and refreshing an existing PR's body. The default workflow token's CI limitation is explicit: a maintainer must close/reopen each generated PR before merge, avoiding an unprovisioned PAT/App credential. Local root checks and the fresh CI suite passed; both review threads are resolved.

.

[sethkarten]

daily might be too frequently. ideally we have a more dynamic way of doing this that doesn't require human review