Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ARCHITECTURE-MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ flowchart LR

1. **Key resolution**: BYOK `options.configuration.apiKey` β†’ (if BYOK group observed, return `[]` to avoid duplicates, issue #106/#131) β†’ `SecretStorage` fallback.
2. Persist key to SecretStorage (non-agent variants) so agent variants inherit it.
3. `fetchModels()` β€” live GET `modelsUrl` with retry/backoff/timeout (issue #78) β†’ `filterAvailableModels()` (drops `KNOWN_UNAVAILABLE_MODEL_IDS`, deprecated Zen models, `freeOnly` filter).
3. `fetchModels()` β€” live GET `modelsUrl` with retry/backoff/timeout (issue #78) β†’ `filterAvailableModels()` (drops `KNOWN_UNAVAILABLE_MODEL_IDS`, deprecated Zen models cross-checked against gateway response (issue #182), `freeOnly` filter).
4. Per model: `resolveModelMetadata()` β†’ `resolveModelRouting()` β†’ `modelLimits()` β†’ `modelCapabilities()` β†’ `modelConfigurationSchema()` (thinking submenu + context-size tier) β†’ build `OpenCodeModel` (general variant or `::agent-host` variant with `targetChatSessionType: "copilotcli"`).

### 5.3 Chat Request (`provideLanguageModelChatResponse`)
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ All notable changes to the **OpenCode Go BYOK Provider** extension are documente

## [Unreleased]

### Fixed

- **`[Models]` Deprecated filter no longer hides live models (#182).** `models.dev` `status: deprecated` was hiding models still served by the gateway (e.g. `laguna-s-2.1-free`). The filter now cross-checks against the live gateway response β€” only hides when `models.dev` says deprecated AND the gateway confirms the model is absent. Note: `deepseek-v4-flash-free` is listed by the gateway but actually broken upstream ("Model is unavailable") β€” this is an upstream issue, not solvable from the extension side. Documented in `docs/issues/78-20260822-issue182-deprecated-model-gateway-crosscheck.md`.

## [0.7.0] β€” 2026-08-22

### Added
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
**Status:** βœ… Solved (partial β€” `deepseek-v4-flash-free` is upstream-blocked)

# Deprecated Model Gateway Cross-Check (#182)

**Topic:** models / provider / registry / availability
**Updated:** 2026-08-22
**Tags:** #models #provider #zen #deprecated #gateway
**Supersedes:** β€”

---

## Overview

`models.dev` `status: deprecated` was hiding live models from the picker. The fix cross-checks against the gateway response: only hide when both `models.dev` says deprecated AND the gateway confirms the model is absent.

The original reporter's example (`deepseek-v4-flash-free`) turned out to be a deeper upstream problem: the gateway lists the model but it's actually broken (`Upstream request failed: Model is unavailable`). A working example is `laguna-s-2.1-free`, which is live and correctly shown by our fix.

---

## Problem

`deepseek-v4-flash-free` appears in the OpenCode Zen gateway (`https://opencode.ai/zen/v1/models` returns it), but requests to it fail:

```text
(400) model=deepseek-v4-flash-free: Error from provider (Console): Upstream request failed: Model is unavailable
```

Meanwhile, `laguna-s-2.1-free` is also listed by `models.dev` as `deprecated`, but IS live and working. The extension's `shouldHideDeprecatedModel` filter was hiding both unconditionally β€” a stale false positive for working models.

### Root Cause

The original deprecated filter (issue #03, 2026-05-16) was added because the gateway **can** list models that are broken at the provider level (`ring-2.6-1t-free`, `trinity-large-preview-free`). `models.dev deprecated` was the only signal that caught them.

However, `models.dev` is community-maintained and can drift β€” marking working models as deprecated. The filter had no cross-check against the gateway, so stale `deprecated` flags hid live models.

### Two Competing Failure Modes

| | Scenario | Before fix |
| --------------- | --------------------------------------------------------- | ------------------- |
| #03 (May 2026) | Gateway lists broken model, `models.dev` says deprecated | βœ… Correctly hidden |
| #182 (Aug 2026) | Gateway lists working model, `models.dev` says deprecated | ❌ Falsely hidden |

### Why `deepseek-v4-flash-free` is a separate problem

Neither `models.dev` nor the gateway is a reliable source of truth for availability:

- `models.dev` says `deprecated` β†’ but `laguna-s-2.1-free` works fine (false positive)
- Gateway lists the model β†’ but `deepseek-v4-flash-free` returns "Model is unavailable" (false positive)

The extension can't distinguish a working model from a broken one without actually sending a request. The honest behavior is to show what the gateway tells us and surface the error clearly when it fails. The real fix for `deepseek-v4-flash-free` is upstream: either the gateway stops listing it, or `models.dev` removes the `deprecated` flag.

---

## Solution

`shouldHideDeprecatedModel` now takes an optional `liveModelIds` set (the gateway response). It only hides when:

1. `models.dev` says `deprecated` **AND**
2. `liveModelIds` is provided (not offline/fallback) **AND**
3. The model is NOT in `liveModelIds` (gateway confirms absence)

This means:

- Gateway lists it β†’ live β†’ don't hide (fixes #182)
- Gateway absent + `models.dev` deprecated β†’ hide (preserves #03 protection)
- Offline/fallback (no live data) β†’ fail open, don't hide on stale metadata alone

### Files Changed

| File | Change |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `src/provider/settings.ts` | `shouldHideDeprecatedModel` gains `liveModelIds?: ReadonlySet<string>` parameter; early-returns `false` when live set confirms the model is present or absent data |
| `src/provider/modelList.ts` | `filterAvailableModels` signature gains `liveModelIds?`; gateway fetch builds `Set(ids)` and passes it through |
| `src/provider/OpenCodeProvider.ts` | `filterAvailableModels` threads `liveModelIds` to `shouldHideDeprecatedModel`; fetcher wiring updated |

---

## Verification

```bash
npm run compile # passes
npm test # passes (8 new tests for shouldHideDeprecatedModel)
```

Registry check:

| Model | Gateway | `models.dev` | Actually works? | Before fix | After fix |
| ---------------------------- | --------- | ------------ | --------------- | ---------- | --------- |
| `laguna-s-2.1-free` | βœ… listed | deprecated | βœ… Yes | ❌ hidden | βœ… shown |
| `deepseek-v4-flash-free` | βœ… listed | deprecated | ❌ No (400) | ❌ hidden | βœ… shown* |
| `ring-2.6-1t-free` | ❌ absent | deprecated | ❌ No | βœ… hidden | βœ… hidden |
| `trinity-large-preview-free` | ❌ absent | deprecated | ❌ No | βœ… hidden | βœ… hidden |

\* Shown but fails at runtime β€” upstream issue, not solvable from extension side.

---

## Notes

- `KNOWN_UNAVAILABLE_MODEL_IDS` (`ring-2.6-1t`, `ring-2.6-1t-free`, `trinity-large-preview-free`) remains as a manual safety net for models known to fail even if listed.
- `models.dev` is still valuable for enrichment (pricing, context windows, capabilities) β€” just not as the sole source of truth for availability.
- Runtime failure tracking was considered but rejected β€” it creates confusing UX where models silently vanish from the picker with no explanation.
6 changes: 3 additions & 3 deletions src/provider/OpenCodeProvider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -493,7 +493,7 @@ export class OpenCodeProvider implements vscode.LanguageModelChatProvider<OpenCo
replaceLiveModelMetadata: (models) => {
this.replaceLiveModelMetadata(models);
},
filterAvailableModels: (ids) => this.filterAvailableModels(ids),
filterAvailableModels: (ids, liveIds) => this.filterAvailableModels(ids, liveIds),
});
}
return this.modelListFetcher;
Expand All @@ -503,15 +503,15 @@ export class OpenCodeProvider implements vscode.LanguageModelChatProvider<OpenCo
return this.fetcher.fetch(apiKey, token);
}

private async filterAvailableModels(modelIds: string[]): Promise<string[]> {
private async filterAvailableModels(modelIds: string[], liveModelIds?: ReadonlySet<string>): Promise<string[]> {
const uniqueModelIds = [...new Set(modelIds)];

try {
const metadataSnapshot = await this.getMetadataSnapshot();
const filteredModelIds = uniqueModelIds.filter(
(modelId) =>
!KNOWN_UNAVAILABLE_MODEL_IDS.has(modelId) &&
!shouldHideDeprecatedModel(modelId, this.baseVendor, metadataSnapshot) &&
!shouldHideDeprecatedModel(modelId, this.baseVendor, metadataSnapshot, liveModelIds) &&
(this.definition.filterModel?.(modelId) ?? true),
);

Expand Down
7 changes: 5 additions & 2 deletions src/provider/modelList.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ export class ModelListFetcher {
definition: ProviderDefinition;
log(message: string): void;
replaceLiveModelMetadata(models: ModelListEntry[] | undefined): void;
filterAvailableModels(modelIds: string[]): Promise<string[]>;
filterAvailableModels(modelIds: string[], liveModelIds?: ReadonlySet<string>): Promise<string[]>;
},
) {
this.cacheKey = `${MODEL_LIST_CACHE_KEY_PREFIX}::${resolveBaseVendor(this.deps.definition.vendor)}`;
Expand Down Expand Up @@ -72,7 +72,10 @@ export class ModelListFetcher {
.filter((id): id is string => typeof id === "string" && id.length > 0)
.filter((id) => this.deps.definition.filterModel?.(id) ?? true);

const filtered = await this.deps.filterAvailableModels(ids?.length ? ids : this.deps.definition.fallbackModels);
// Gateway is source of truth β€” pass live IDs so stale `deprecated` in
// models.dev doesn't hide models still served (issue #182).
const liveIds = ids?.length ? new Set(ids) : undefined;
const filtered = await this.deps.filterAvailableModels(ids?.length ? ids : this.deps.definition.fallbackModels, liveIds);
// Persist the successful snapshot for future fallback coverage.
this.cached = { ids: filtered, fetchedAt: Date.now() };
void this.deps.context.globalState.update(this.cacheKey, this.cached);
Expand Down
16 changes: 15 additions & 1 deletion src/provider/settings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -225,11 +225,25 @@ export function shouldHideDeprecatedModel(
modelId: string,
vendor: ProviderDefinition["vendor"],
snapshot: CachedModelMetadataSnapshot,
liveModelIds?: ReadonlySet<string>,
): boolean {
if (resolveBaseVendor(vendor) !== ZEN_VENDOR) {
return false;
}
return snapshot.providers[ZEN_VENDOR]?.[modelId]?.status === "deprecated";
if (snapshot.providers[ZEN_VENDOR]?.[modelId]?.status !== "deprecated") {
return false;
}
// Gateway is the source of truth for availability (issue #182). Only hide
// when we have live gateway data confirming the model is absent. If the
// gateway still serves it, models.dev is stale β€” don't hide. If we have
// no live data (offline/fallback), fail open and don't hide either.
if (!liveModelIds) {
return false;
}
if (liveModelIds.has(modelId)) {
return false;
}
return true;
}

export function resolveRawModelId(modelId: string): string {
Expand Down
97 changes: 97 additions & 0 deletions src/test/deprecatedFilter.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
import Module from "node:module";
import path from "node:path";
import fs from "node:fs";
import os from "node:os";
import assert from "node:assert/strict";
import { describe, it, before } from "node:test";

// Install vscode mock before any extension imports (same pattern as goUsageTestUtils)
const vscodeMockPath = path.join(fs.mkdtempSync(path.join(os.tmpdir(), "vscode-mock-deprecated-")), "index.js");
fs.mkdirSync(path.dirname(vscodeMockPath), { recursive: true });
fs.writeFileSync(
vscodeMockPath,
`"use strict";\nmodule.exports = { workspace: { getConfiguration: () => ({ get: () => undefined }) } };`,
"utf-8",
);
const originalResolveFilename = (Module as unknown as { _resolveFilename: (req: string, parent: unknown, ...args: unknown[]) => string })
._resolveFilename;
(Module as unknown as { _resolveFilename: (req: string, parent: unknown, ...args: unknown[]) => string })._resolveFilename = function (
req: string,
parent: unknown,
...args: unknown[]
): string {
return req === "vscode" ? vscodeMockPath : originalResolveFilename.call(this, req, parent, ...args);
};

let shouldHideDeprecatedModel: typeof import("../provider/settings.js").shouldHideDeprecatedModel;
let GO_VENDOR: string;
let ZEN_VENDOR: string;
let AGENT_ZEN_VENDOR: string;
type CachedModelMetadataSnapshot = import("../models/metadata.js").CachedModelMetadataSnapshot;

before(async () => {
const settings = await import("../provider/settings.js");
shouldHideDeprecatedModel = settings.shouldHideDeprecatedModel;
const types = await import("../providerTypes.js");
GO_VENDOR = types.GO_VENDOR;
ZEN_VENDOR = types.ZEN_VENDOR;
AGENT_ZEN_VENDOR = types.AGENT_ZEN_VENDOR;
});

function snapshotWithStatus(modelId: string, status: string | undefined): CachedModelMetadataSnapshot {
return {
fetchedAt: Date.now(),
providers: {
opencodego: undefined,
opencodezen: {
[modelId]: { status },
},
},
};
}

describe("shouldHideDeprecatedModel", () => {
it("returns false for non-Zen vendors", () => {
const snap = snapshotWithStatus("deepseek-v4-flash-free", "deprecated");
assert.equal(shouldHideDeprecatedModel("deepseek-v4-flash-free", GO_VENDOR as never, snap), false);
});

it("returns false when status is not deprecated", () => {
const snap = snapshotWithStatus("deepseek-v4-flash-free", "beta");
assert.equal(shouldHideDeprecatedModel("deepseek-v4-flash-free", ZEN_VENDOR as never, snap), false);
});

it("returns false when status is absent", () => {
const snap = snapshotWithStatus("deepseek-v4-flash-free", undefined);
assert.equal(shouldHideDeprecatedModel("deepseek-v4-flash-free", ZEN_VENDOR as never, snap), false);
});

it("returns false when no live data (offline/fallback) β€” fail open", () => {
const snap = snapshotWithStatus("deepseek-v4-flash-free", "deprecated");
assert.equal(shouldHideDeprecatedModel("deepseek-v4-flash-free", ZEN_VENDOR as never, snap, undefined), false);
});

it("returns false when gateway still serves the model (stale models.dev)", () => {
const snap = snapshotWithStatus("deepseek-v4-flash-free", "deprecated");
const liveIds = new Set(["deepseek-v4-flash-free", "mimo-v2.5-free"]);
assert.equal(shouldHideDeprecatedModel("deepseek-v4-flash-free", ZEN_VENDOR as never, snap, liveIds), false);
});

it("returns true when deprecated and gateway confirms model is absent", () => {
const snap = snapshotWithStatus("ring-2.6-1t-free", "deprecated");
const liveIds = new Set(["deepseek-v4-flash-free", "mimo-v2.5-free"]);
assert.equal(shouldHideDeprecatedModel("ring-2.6-1t-free", ZEN_VENDOR as never, snap, liveIds), true);
});

it("resolves agent-variant vendor to base vendor", () => {
const snap = snapshotWithStatus("deepseek-v4-flash-free", "deprecated");
const liveIds = new Set(["deepseek-v4-flash-free"]);
assert.equal(shouldHideDeprecatedModel("deepseek-v4-flash-free", AGENT_ZEN_VENDOR as never, snap, liveIds), false);
});

it("hides when live set is empty (gateway returned no models)", () => {
const snap = snapshotWithStatus("deepseek-v4-flash-free", "deprecated");
const liveIds = new Set<string>();
assert.equal(shouldHideDeprecatedModel("deepseek-v4-flash-free", ZEN_VENDOR as never, snap, liveIds), true);
});
});
Loading