Tool-selection ergonomics: grouped catalog, tier bulk switches, opt-in servers, KV hints

CLAUDE.md

@@ -15,15 +15,15 @@ Self-hosted MCP manager/gateway: one streamable-HTTP `/mcp` endpoint federating
 - `db/` — `node:sqlite` schema (roles/upstreams/grants/tool_overrides/tool_settings/users/group_mappings, seeded viewer/editor/admin) + typed `Repo`
 - `domain/catalog.ts` — namespacing (`${namespace}_${tool}`, no double-prefix), routing map (no string-splitting), annotation-derived tiers (port of mcp-itglue `tierOf`)
 - `domain/presets.ts` — one-click upstream presets: builtin family configs (itglue/cwpsa/planner/cipp — full specs incl. BYOK headers, per-user mode, userConnect, personalCredentials, the `auth` mint block) + optional `mspstack.presets.json` (file overrides builtin ids); `{{param}}` templating rendered server-side and validated via `parseUpstreamSpec`; recommended grants by role NAME resolved at install (`GET /api/presets`, `POST /api/presets/:id/install` with `dryRun`). Spec's `personalCredentials` metadata drives the /me guided credential forms (`credentialFields` in `/api/me/access`)
-- `domain/policy.ts` — `PolicyService`: toolEnabled ∧ (override(allow) ∨ (tier ≤ maxTier ∧ ¬deny)); maxTier = per-upstream grant ?? role default. Same function gates tools/list AND tools/call. `allowsFor(principal, entry)` = envelope ∧ personal prefs (deny-only rows in `user_prefs`; "enable" deletes the row — narrowing can never widen)
+- `domain/policy.ts` — `PolicyService`: toolEnabled ∧ (override(allow) ∨ (tier ≤ maxTier ∧ ¬deny)); maxTier = per-upstream grant ?? role default. Same function gates tools/list AND tools/call. `allowsFor(principal, entry)` = envelope ∧ personal prefs (deny rows in `user_prefs`; "enable" deletes the row — narrowing can never widen). Spec `userDefault: "off"` inverts the personal layer for that upstream: nothing is live until an explicit opt-in row exists (per tool or server-wide `''`), still capped by the envelope — for servers with hundreds of tools. `/api/me/access` derives its `enabled` flags from `allowsFor` so the page can't disagree with the boundary
 - `auth/` — `static-tokens.ts` (timing-safe bearer match), `oidc.ts` (jose JWKS resource-server verifier for inbound *access* tokens), `login.ts` (interactive login: openid-client cookie+PKCE confidential-client flow consuming an *id-token*; signed identity-only session cookie, HMAC + freshness; `safeReturnTo`), `authz-server.ts` (OAuth AS facade: RFC 8414 metadata, RFC 7591 DCR for public clients, single-use hashed 60s codes + PKCE S256, HS256 gateway JWTs keyed by `GATEWAY_JWT_SECRET` (default derived from `SESSION_SECRET`), rotating refresh tokens — 30d sliding, family-revoked on replay, client-bound consume that can't burn a live token — register rate limit; clients managed via `/api/oauth-clients` + Users tab), `prm.ts` (RFC 9728 doc + WWW-Authenticate; lists the gateway itself as AS when login is configured, else the raw IdP), `directory.ts` (app-only Graph search of Entra users/groups via the login app's own creds — powers the admin UI group-mapping typeahead at `/api/directory/search`; null for non-Entra issuers → UI degrades to paste-an-id), `principal.ts` (session binding key). Four inbound auth paths in `createAuthResolver`: static token, gateway-issued JWT (routed by unverified `iss == PUBLIC_URL`, then fully verified), OIDC bearer, and the cookie session — the cookie/JWT carry only identity and the role is re-resolved every request (persisted at callback via `setUserRole`), so a session id never carries privilege. `loginUpsert()` is shared by the bearer + callback paths so they can't drift. `/oauth/authorize` brokers user auth to Entra by piggybacking the interactive login: the pending request rides in the signed transient cookie and `/auth/callback` mints the code.
 - `secrets/` — `SecretStore` interface (scheme-tagged: `bao` | `kv`), `openbao.ts` (KV v2, AppRole or token, 5-min cache), `keyvault.ts` (Azure Key Vault, `DefaultAzureCredential`, lazy SDK import, same 5-min cache; `put(path, field)` writes `path-field`), `memory.ts` (tests). Refs: `bao:path#field` / `kv:secret-name`; env refs: `${VAR}` — all resolved only at upstream connect time. One store at a time (`BAO_ADDR` xor `KEY_VAULT_URI`)
 - `upstream/connection.ts` — one pooled SDK `Client` per upstream; header/env injection; backoff reconnect (1s→60s) + `onRecovered`; retry-once on dropped transport AND on server-side session expiry (upstream 404 "unknown session" → transparent re-initialize + retry, per MCP spec). Optional spec `auth` block (`oauth2-client-credentials`): the gateway mints the upstream's bearer itself (secret via `${VAR}`/`bao:`/`kv:` ref), caches it, and rebuilds the connection when it nears expiry (60s skew) — for third-party servers that want a finished token, e.g. CIPP behind Easy Auth. Neither secret nor token is ever logged
 - `upstream/manager.ts` also pools **per-principal links** for `sessionMode:"per-user"` upstreams: spec clone with the caller's credential REFS layered over headers/env (still resolved via the secret store at connect — anti-passthrough intact); catalog discovery stays on the shared link; personal pool flushed on upstream upsert/remove. `requirePersonalCredentials` refuses the shared fallback
 - `upstream/manager.ts` — policy-free catalog owner; hot `upsertUpstream`/`removeUpstream`; `summaries()` for the UI
 - `mcp/gateway-server.ts` — low-level SDK `Server` per session, closes over the Principal; unknown and forbidden tools get the same error (no existence oracle)
 - `http/app.ts` — `/mcp` (origin check → resolveAuth → principal-bound sessions), PRM endpoints, per-session fingerprint-diffed `list_changed`, mounts `/api` + `/admin`
-- `http/admin-api.ts` — admin-only JSON API (upstream CRUD, preflight, registry search, catalog toggles, roles/grants/overrides, users, mappings, secret writes)
+- `http/admin-api.ts` — admin-only JSON API (upstream CRUD, preflight, registry search, catalog toggles, roles/grants/overrides, users, mappings, secret writes). Bulk toggles: `PATCH /api/catalog/:upstreamId` `{enabled, tier?, group?}` (one transaction, targets resolved from the live catalog, matching on the EFFECTIVE tier and on `groupLabel ?? derivedGroupOf(tool)`); `PUT /api/me/prefs` takes the same `tier?`/`group?` shape scoped to the caller's envelope. `/api/secrets/health` also reports `scheme` so the UI can show `kv:` vs `bao:` hints
 - `http/me-api.ts` — `/api/me/*` for ANY principal (mounted before `/api`), static-token principals included: effective access (envelope ∧ prefs), narrow-only prefs (404 outside the envelope), personal credential registration → secret store under `gw-user-<principalSlug>-<upstreamId>-<field>`, SQLite keeps only refs. Consumed by per-principal sessions later (`sessionMode`). One-click delegated Connect: upstream spec's `userConnect` block + `auth/user-connect.ts` (Entra auth-code+PKCE against a PUBLIC client) + `/me/connect/:upstreamId|callback` routes in app.ts — the refresh token lands in the secret store as the signed-in user's personal credential, no scripts/copy-paste. `/access` emits `connect: null` unless the whole flow is mounted (login + userConnect + secret store) so the UI never renders dead Connect buttons
 - `public/admin.html` — dependency-free single-file admin UI ("Sign in with Microsoft" cookie flow + token paste as break-glass), served at `/admin`
 - `public/me.html` — dependency-free user page ("My MCP Access") served at `/me`: my servers/tools (narrow-only), my personal upstream credentials (ref only), connect snippet. Same sign-in panel as admin.html: cookie session ("Sign in with Microsoft", shown when `/health` reports `login: true`) + bearer-token paste (sessionStorage `mspstack-me-token`) — so /me works on token-only deployments. With login configured, session-less HTML GETs of `/` and `/me` still redirect to `/auth/login` (silent SSO, unchanged UX); `?signin=token` bypasses the gate (break-glass). Without login, `/` redirects to `/me`, the panel shows only the token box, and the Connect snippets include the `Authorization: Bearer` header (no OAuth facade → URL-alone connect can't work; opt-in checkbox embeds the signed-in token). `/api/*` stays server-side-gated (the real boundary)

README.md

@@ -14,6 +14,7 @@ Point Claude (Code, Desktop, or any MCP client) at a single URL; the gateway con
 - **Minted upstream tokens** — for servers that expect a short-lived bearer instead of a static key (CIPP and friends behind Entra/Easy Auth), an upstream's `auth` block holds only a client id plus a secret reference: the gateway runs the client-credentials exchange itself, caches the token, and refreshes it before expiry
 - **Install from the UI** — one-click **presets** for the MSPStack family (IT Glue, ConnectWise PSA, Planner) and CIPP that fill BYOK headers, per-user session mode, Connect wiring, and apply recommended role grants (extend with your own via `mspstack.presets.json`); or add any MCP server by URL, npm package (npx), or Docker image; search the official MCP registry; preflight-test before saving; crashed stdio servers restart with backoff
 - **Guided user setup** — upstreams declare their personal-credential fields, so `/me` renders labeled forms (not raw header names), plus ready-to-copy connect snippets: Claude Code CLI (user-scope by default) and JSON config for Desktop/Cursor/VS Code
+- **Manageable at scale** — the admin catalog groups tools by server with one-click switches for a whole tier (or category, derived from the server's own description prefixes); big servers can ship `userDefault: "off"` so users opt into the tools they need instead of receiving hundreds
 - **Admin UI** at `/admin` — status, server management, tool toggles, role matrix, users & group mappings (with live Entra group search when the login app holds the directory-read Graph roles), OAuth client management, secret writes
 
 ## Quick start

package.json

@@ -1,7 +1,7 @@
 {
   "name": "mcp-gateway-monorepo",
   "private": true,
-  "version": "0.10.0",
+  "version": "0.11.0",
   "description": "MSPStack Gateway — self-hosted MCP manager: one endpoint federating many MCP servers with OAuth, roles, secret storage, and tool toggles",
   "type": "module",
   "workspaces": [

packages/gateway/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mspstack/mcp-gateway",
-  "version": "0.10.0",
+  "version": "0.11.0",
   "description": "Self-hosted MCP gateway: one streamable-HTTP endpoint federating many MCP servers with namespaced tools, per-tool toggles, roles, and secure upstream credential injection",
   "type": "module",
   "main": "dist/index.js",

packages/gateway/public/admin.html

@@ -381,24 +381,104 @@ <h3>${esc(p.title)}</h3>
 }
 
 // ── Tools ──
+/** Tools tab state: which upstream groups are expanded, and the name filter. */
+const toolsUi = { open: new Set(), filter: "", ROW_CAP: 50 };
+
 async function renderTools() {
   const el = $("#tab-tools");
   const tools = await api("/catalog");
-  el.innerHTML = `<section class="panel"><h2>Tool catalog (${tools.length})</h2>
-    <table>
-      <tr><th>On</th><th>Tool</th><th>Upstream</th><th>Tier</th><th>Override</th><th>Group</th></tr>
-      ${tools.map(t => `<tr>
-        <td><input type="checkbox" data-u="${esc(t.upstreamId)}" data-t="${esc(t.toolName)}" data-k="enabled" ${t.enabled ? "checked" : ""}></td>
-        <td class="mono" title="${esc(t.description)}">${esc(t.exposedName)}</td>
-        <td class="muted">${esc(t.upstreamId)}</td>
-        <td>${tierPill(t.derivedTier)}</td>
-        <td><select data-u="${esc(t.upstreamId)}" data-t="${esc(t.toolName)}" data-k="tierOverride">
-          <option value="">(derived)</option>
-          ${["read","write","destructive"].map(x => `<option ${t.tierOverride===x?"selected":""}>${x}</option>`).join("")}
-        </select></td>
-        <td><input size="10" placeholder="—" value="${esc(t.groupLabel||"")}" data-u="${esc(t.upstreamId)}" data-t="${esc(t.toolName)}" data-k="groupLabel"></td>
-      </tr>`).join("")}
-    </table></section>`;
+  const enabledCount = tools.filter(t => t.enabled).length;
+  const groupOf = (t) => t.groupLabel || t.derivedGroup || "";
+
+  // Group by upstream, then count per effective tier for the bulk switches.
+  const byUpstream = new Map();
+  for (const t of tools) {
+    const g = byUpstream.get(t.upstreamId) ?? { id: t.upstreamId, tools: [], tiers: { read: 0, write: 0, destructive: 0 }, on: 0 };
+    g.tools.push(t);
+    g.tiers[t.effectiveTier] = (g.tiers[t.effectiveTier] || 0) + 1;
+    if (t.enabled) g.on += 1;
+    byUpstream.set(t.upstreamId, g);
+  }
+
+  const tierBtn = (id, tier, n, allOn) => n === 0 ? "" :
+    `<button class="btn secondary" data-bulk-u="${esc(id)}" data-bulk-tier="${tier}" data-bulk-en="${allOn ? "0" : "1"}"
+      title="${allOn ? "Disable" : "Enable"} all ${tier} tools of ${esc(id)} — for every role">
+      ${allOn ? "✓ " : ""}${tier} <span class="muted">${n}</span></button>`;
+
+  el.innerHTML = `<section class="panel">
+    <h2>Tool catalog <span class="muted">${enabledCount} of ${tools.length} enabled · ${byUpstream.size} servers</span></h2>
+    <p class="muted">Bulk switches apply to every tool of that tier in the group — for all roles.
+      Per-role limits live on the Roles tab. Tier shown is the override when set, else the annotation-derived value.</p>
+    <div class="row" style="margin-bottom:10px">
+      <input id="tool-filter" placeholder="filter tools…" value="${esc(toolsUi.filter)}" style="flex:1; max-width:280px">
+    </div>
+    ${[...byUpstream.values()].map(g => {
+      const open = toolsUi.open.has(g.id);
+      const matching = toolsUi.filter
+        ? g.tools.filter(t => t.exposedName.toLowerCase().includes(toolsUi.filter.toLowerCase()))
+        : g.tools;
+      const shown = matching.slice(0, toolsUi.ROW_CAP);
+      return `<div class="panel" style="background:var(--panel2); margin-bottom:8px; padding:0">
+        <div class="row" style="align-items:center; gap:10px; padding:8px 10px; margin:0">
+          <button class="btn secondary" data-open="${esc(g.id)}" title="${open ? "collapse" : "expand"}">${open ? "▾" : "▸"}</button>
+          <button class="btn ${g.on > 0 ? "" : "secondary"}" data-bulk-u="${esc(g.id)}" data-bulk-en="${g.on > 0 ? "0" : "1"}"
+            title="${g.on > 0 ? "Disable" : "Enable"} every tool of ${esc(g.id)}">${g.on > 0 ? "server on" : "server off"}</button>
+          <b>${esc(g.id)}</b>
+          <span class="muted">${g.on} of ${g.tools.length} enabled</span>
+          <span style="flex:1"></span>
+          ${["read", "write", "destructive"].map(tier => {
+            const n = g.tiers[tier] || 0;
+            const allOn = n > 0 && g.tools.filter(t => t.effectiveTier === tier).every(t => t.enabled);
+            return tierBtn(g.id, tier, n, allOn);
+          }).join(" ")}
+        </div>
+        ${open ? `<div style="border-top:1px solid var(--border); padding:6px 10px 10px">
+          <table>
+            <tr><th>On</th><th>Tool</th><th>Tier</th><th>Override</th><th>Group</th></tr>
+            ${shown.map(t => `<tr>
+              <td><input type="checkbox" data-u="${esc(t.upstreamId)}" data-t="${esc(t.toolName)}" data-k="enabled" ${t.enabled ? "checked" : ""}></td>
+              <td class="mono" title="${esc(t.description)}">${esc(t.exposedName)}</td>
+              <td>${tierPill(t.effectiveTier)}${t.tierOverride ? ` <span class="muted" style="font-size:11px">override, was ${esc(t.derivedTier)}</span>` : ""}</td>
+              <td><select data-u="${esc(t.upstreamId)}" data-t="${esc(t.toolName)}" data-k="tierOverride">
+                <option value="">(derived)</option>
+                ${["read","write","destructive"].map(x => `<option ${t.tierOverride===x?"selected":""}>${x}</option>`).join("")}
+              </select></td>
+              <td><input size="10" placeholder="${esc(t.derivedGroup || "—")}" value="${esc(t.groupLabel||"")}" data-u="${esc(t.upstreamId)}" data-t="${esc(t.toolName)}" data-k="groupLabel"></td>
+            </tr>`).join("")}
+          </table>
+          ${matching.length > shown.length ? `<p class="muted">…${matching.length - shown.length} more — narrow the filter, or use the switches above</p>` : ""}
+          ${matching.length === 0 ? '<p class="muted">no tools match the filter</p>' : ""}
+        </div>` : ""}
+      </div>`;
+    }).join("")}
+  </section>`;
+
+  $("#tool-filter").oninput = (ev) => {
+    toolsUi.filter = ev.target.value;
+    const el2 = ev.target;
+    clearTimeout(el2._t);
+    el2._t = setTimeout(() => renderTools().then(() => $("#tool-filter").focus()), 250);
+  };
+
+  el.onclick = async (ev) => {
+    const openBtn = ev.target.closest("[data-open]");
+    if (openBtn) {
+      const id = openBtn.dataset.open;
+      toolsUi.open.has(id) ? toolsUi.open.delete(id) : toolsUi.open.add(id);
+      renderTools();
+      return;
+    }
+    const bulk = ev.target.closest("[data-bulk-u]");
+    if (!bulk) return;
+    const body = { enabled: bulk.dataset.bulkEn === "1" };
+    if (bulk.dataset.bulkTier) body.tier = bulk.dataset.bulkTier;
+    try {
+      const r = await api(`/catalog/${encodeURIComponent(bulk.dataset.bulkU)}`, { method: "PATCH", body: JSON.stringify(body) });
+      toast(`${body.enabled ? "Enabled" : "Disabled"} ${r.changed} tool${r.changed === 1 ? "" : "s"}`);
+      renderTools();
+    } catch (e) { toast(e.message, true); }
+  };
+
   el.onchange = async (ev) => {
     const i = ev.target; if (!i.dataset.k) return;
     const patch = {};
@@ -594,11 +674,24 @@ <h3>${esc(p.title)}</h3>
   el.innerHTML = `
     <section class="panel"><h2>Secret store</h2>
       <p>${dot(health.ok)}${esc(health.detail)}</p>
-      <p class="muted">Values are written straight to OpenBao and never stored or displayed by the gateway.
-      Reference them in upstream headers/env as <code>bao:&lt;path&gt;#&lt;field&gt;</code>.</p>
+      <p class="muted">${!health.scheme
+        ? `No secret store is configured, so nothing can be written here. Set <code>KEY_VAULT_URI</code>
+           (Azure Key Vault, <code>kv:</code> refs) or <code>BAO_ADDR</code> (OpenBao, <code>bao:</code> refs).
+           Without a store, upstream headers/env can still reference this process's environment as
+           <code>\${VAR}</code> — resolved server-side at connect time.`
+        : health.scheme === "kv"
+        ? `Values are written straight to Azure Key Vault and never stored or displayed by the gateway.
+           Key Vault has no nested fields, so the gateway stores one secret named
+           <code>&lt;name&gt;-&lt;field&gt;</code> — letters, digits and dashes only.
+           Reference it in upstream headers/env as <code>kv:&lt;name&gt;-&lt;field&gt;</code>,
+           e.g. <code>kv:cipp-mcp-secret</code>.`
+        : `Values are written straight to OpenBao and never stored or displayed by the gateway.
+           KV v2 keeps several fields under one path, so
+           <code>upstreams/itglue</code> can hold both <code>token</code> and <code>region</code>.
+           Reference them in upstream headers/env as <code>bao:&lt;path&gt;#&lt;field&gt;</code>.`}</p>
       <div class="row">
-        <label>Path <input id="sec-path" placeholder="upstreams/itglue"></label>
-        <label>Field <input id="sec-field" placeholder="token"></label>
+        <label>${health.scheme === "kv" ? "Name" : "Path"} <input id="sec-path" placeholder="${health.scheme === "kv" ? "cipp-mcp" : "upstreams/itglue"}"></label>
+        <label>Field <input id="sec-field" placeholder="${health.scheme === "kv" ? "secret" : "token"}"></label>
         <label style="flex:1">Value <input id="sec-value" type="password" style="width:100%"></label>
         <button class="btn" id="sec-save" ${health.ok ? "" : "disabled"}>Save</button>
       </div>