From 82e5d917eaeae5757d985c68d24618daf9de4322 Mon Sep 17 00:00:00 2001 From: Eugene Samotija Date: Thu, 30 Jul 2026 03:02:43 -0400 Subject: [PATCH 1/3] db: online SQLite backups with retention and optional off-instance copies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #3. The database holds everything that isn't reproducible from config — upstream specs, roles/grants/tool settings, users, DCR clients, refresh-token families, credential refs — and a lost file meant rebuilding it by hand. src/db/backup.ts snapshots via VACUUM INTO (a plain file copy of a live db can capture a torn page mid-WAL), keeps the newest BACKUP_KEEP locally, and can ship each snapshot off-instance to Azure Blob with DefaultAzureCredential (lazily imported, so deployments without it pay nothing). Upload failures are logged but never fail the snapshot: a local copy beats none. The scheduler is unref'd and stops on shutdown; :memory: databases opt out automatically. Admin API: GET /api/backups (config + what's on disk) and POST /api/backups (snapshot now). Restore stays manual and is documented in docs/backups.md, including the -wal/-shm cleanup step that otherwise bites. Co-Authored-By: Claude Fable 5 --- docs/backups.md | 69 ++++++++ package-lock.json | 210 ++++++++++++++++++++++++- packages/gateway/package.json | 1 + packages/gateway/src/config.ts | 31 ++++ packages/gateway/src/db/backup.test.ts | 153 ++++++++++++++++++ packages/gateway/src/db/backup.ts | 164 +++++++++++++++++++ packages/gateway/src/http/admin-api.ts | Bin 19013 -> 19929 bytes packages/gateway/src/http/app.ts | 6 + packages/gateway/src/index.ts | 12 ++ 9 files changed, 643 insertions(+), 3 deletions(-) create mode 100644 docs/backups.md create mode 100644 packages/gateway/src/db/backup.test.ts create mode 100644 packages/gateway/src/db/backup.ts diff --git a/docs/backups.md b/docs/backups.md new file mode 100644 index 0000000..da29d44 --- /dev/null +++ b/docs/backups.md @@ -0,0 +1,69 @@ +# Backups and restore + +The gateway's SQLite database is not reproducible from configuration. It holds +upstream specs, roles, grants and per-tool settings, users and group mappings, +dynamically registered OAuth clients, refresh-token families, and the +references to every user's personal credentials. Losing it means rebuilding all +of that by hand — so back it up. + +## What runs by default + +On every boot the gateway starts a periodic snapshot loop: + +| Env | Default | Meaning | +| --- | --- | --- | +| `BACKUP_INTERVAL_HOURS` | `24` | snapshot cadence; `0` disables the loop (on-demand still works) | +| `BACKUP_DIR` | `/backups` | where snapshots land | +| `BACKUP_KEEP` | `7` | how many local snapshots to keep — oldest pruned first | +| `BACKUP_BLOB_CONTAINER_URL` | — | Azure Blob container for off-instance copies (`DefaultAzureCredential`) | + +Snapshots are taken with SQLite's `VACUUM INTO`, which writes a consistent +copy while the gateway keeps serving. Copying `gateway.db` with `cp` while the +process runs is **not** a backup — it can capture a torn page mid-WAL. + +Files are named `gateway-backup-.db`, so they sort +chronologically and never overwrite each other. + +An in-memory database (`DB_PATH=:memory:`) has nothing durable to snapshot, so +the loop stays off there regardless of the interval. + +## Off-instance copies + +A backup living next to the database is not a backup: the App Service instance +that loses the disk loses both. Set `BACKUP_BLOB_CONTAINER_URL` to a container +URL and each snapshot is uploaded with the gateway's managed identity (needs +**Storage Blob Data Contributor** on the container). Upload failures are logged +loudly but never fail the snapshot itself — a local copy still beats none. + +## On demand + +Admin API (admin role required): + +```bash +curl -s -X POST https:///api/backups -H "Authorization: Bearer " +``` + +```bash +curl -s https:///api/backups -H "Authorization: Bearer " +``` + +`GET` returns the configured directory, retention, interval, whether an +off-instance target is set, and the snapshots currently on disk. + +## Restore + +Deliberately manual — restoring is rare and destructive, so it is not a button. + +1. Stop the gateway (App Service: stop the app, or `docker compose stop`). +2. Put the snapshot where `DB_PATH` points, e.g. `data/gateway.db`. +3. Delete any `gateway.db-wal` / `gateway.db-shm` siblings — they belong to the + old database and will confuse SQLite about the restored one. +4. Start the gateway. Schema migrations are idempotent and run on boot, so a + snapshot from an older version upgrades itself. +5. Verify: `GET /api/status` (upstream count and tool count) and `GET /api/roles` + (grants matrix). Personal credentials keep working because the database + stores only references — the values live in the secret store, untouched by + the restore. + +Test the restore path on a scratch instance before you need it; an untested +backup is a hope, not a plan. diff --git a/package-lock.json b/package-lock.json index 5c77f81..17554b9 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "mcp-gateway-monorepo", - "version": "0.9.1", + "version": "0.12.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "mcp-gateway-monorepo", - "version": "0.9.1", + "version": "0.12.0", "license": "MIT", "workspaces": [ "packages/*" @@ -76,6 +76,22 @@ "node": ">=22.0.0" } }, + "node_modules/@azure/core-http-compat": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@azure/core-http-compat/-/core-http-compat-2.5.0.tgz", + "integrity": "sha512-BoSmXPx2er1Ai+wKlDvj29jIQespCNBwEmKyZVHO2kEFsWbGjAjwMCGzug3DJM5/QYIV3vej0S1zcU5bq9fa8w==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2" + }, + "engines": { + "node": ">=22.0.0" + }, + "peerDependencies": { + "@azure/core-client": "^1.10.0", + "@azure/core-rest-pipeline": "^1.22.0" + } + }, "node_modules/@azure/core-lro": { "version": "2.7.2", "resolved": "https://registry.npmjs.org/@azure/core-lro/-/core-lro-2.7.2.tgz", @@ -147,6 +163,19 @@ "node": ">=22.0.0" } }, + "node_modules/@azure/core-xml": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/@azure/core-xml/-/core-xml-1.6.0.tgz", + "integrity": "sha512-e7lX/dk//F6Qf7BB6PTY4+p2yuOQtyOeHGyapYHNwqSp2OnYpwQt49A/Nin2XmKBQ69pwagR4k/lQBq8lbHQkA==", + "license": "MIT", + "dependencies": { + "fast-xml-parser": "^5.5.9", + "tslib": "^2.8.1" + }, + "engines": { + "node": ">=22.0.0" + } + }, "node_modules/@azure/identity": { "version": "4.13.1", "resolved": "https://registry.npmjs.org/@azure/identity/-/identity-4.13.1.tgz", @@ -257,6 +286,51 @@ "node": ">=20" } }, + "node_modules/@azure/storage-blob": { + "version": "12.33.0", + "resolved": "https://registry.npmjs.org/@azure/storage-blob/-/storage-blob-12.33.0.tgz", + "integrity": "sha512-2SX8oP8PyblUcAFZSg39c8Ls+tFjavM6sBeV+qpw33mRzRhI/5hrFJmJ/x0H9xx5l6ECPvgSP8uPxqTeVbHNIA==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@azure/core-auth": "^1.9.0", + "@azure/core-client": "^1.9.3", + "@azure/core-http-compat": "^2.2.0", + "@azure/core-lro": "^2.2.0", + "@azure/core-paging": "^1.6.2", + "@azure/core-rest-pipeline": "^1.19.1", + "@azure/core-tracing": "^1.2.0", + "@azure/core-util": "^1.11.0", + "@azure/core-xml": "^1.4.5", + "@azure/logger": "^1.1.4", + "@azure/storage-common": "^12.4.1", + "events": "^3.0.0", + "tslib": "^2.8.1" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/storage-common": { + "version": "12.4.1", + "resolved": "https://registry.npmjs.org/@azure/storage-common/-/storage-common-12.4.1.tgz", + "integrity": "sha512-t14unw/WofGDUi7TKJrsyXyPsN+NLgRm7hMaq0llxNmTIzt7f257+6LE6FKIJPh88zLj6M7LPvzve0fEYg/L3A==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@azure/core-auth": "^1.9.0", + "@azure/core-http-compat": "^2.2.0", + "@azure/core-rest-pipeline": "^1.24.0", + "@azure/core-tracing": "^1.2.0", + "@azure/core-util": "^1.11.0", + "@azure/logger": "^1.1.4", + "events": "^3.3.0", + "tslib": "^2.8.1" + }, + "engines": { + "node": ">=20.0.0" + } + }, "node_modules/@emnapi/core": { "version": "1.11.1", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", @@ -815,6 +889,18 @@ "@emnapi/runtime": "^1.7.1" } }, + "node_modules/@nodable/entities": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/@nodable/entities/-/entities-3.0.0.tgz", + "integrity": "sha512-8L9xFeTYKhm49xfIypoe2W5wV1m/3Z58kT+7kR9A8OyFxcPduI4VmxaUMQyKYrRjUoLLSXv6EKKID5Tvj9cUVw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/nodable" + } + ], + "license": "MIT" + }, "node_modules/@oxc-project/types": { "version": "0.138.0", "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.138.0.tgz", @@ -1430,6 +1516,18 @@ } } }, + "node_modules/anynum": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/anynum/-/anynum-1.0.1.tgz", + "integrity": "sha512-N6//FLET/tXYNM/F6ABca1oH6fWB+KlTt909Le28WMDBk8oaT4vY17DCrwg2MvmuqUKt3Ni4N5dGJ/EoBgcO6A==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT" + }, "node_modules/assertion-error": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", @@ -1842,6 +1940,15 @@ "node": ">= 0.6" } }, + "node_modules/events": { + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz", + "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==", + "license": "MIT", + "engines": { + "node": ">=0.8.x" + } + }, "node_modules/eventsource": { "version": "3.0.7", "resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz", @@ -1956,6 +2063,45 @@ ], "license": "BSD-3-Clause" }, + "node_modules/fast-xml-builder": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/fast-xml-builder/-/fast-xml-builder-1.3.0.tgz", + "integrity": "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "path-expression-matcher": "^1.6.2", + "xml-naming": "^0.3.0" + } + }, + "node_modules/fast-xml-parser": { + "version": "5.10.1", + "resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-5.10.1.tgz", + "integrity": "sha512-IEMIf7298kXuZSRFoGfMYrl7is8LpavODgbNz1cwIudv7KwVFnuU+UsMporfq6PD6aXSlawZlARiA3UywCTfMw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "@nodable/entities": "^3.0.0", + "fast-xml-builder": "^1.2.0", + "is-unsafe": "^2.0.0", + "path-expression-matcher": "^1.6.2", + "strnum": "^2.4.1", + "xml-naming": "^0.3.0" + }, + "bin": { + "fxparser": "src/cli/cli.js" + } + }, "node_modules/fdir": { "version": "6.5.0", "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", @@ -2244,6 +2390,18 @@ "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==", "license": "MIT" }, + "node_modules/is-unsafe": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/is-unsafe/-/is-unsafe-2.0.0.tgz", + "integrity": "sha512-2LdV822R+wmI86unXA93WCFpL6g+av8ynWk0nrHyJqGop5VoocYsSLFgN8jrfalT6iGeLNM4KXuVSsULP53kEA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT" + }, "node_modules/is-wsl": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/is-wsl/-/is-wsl-3.1.1.tgz", @@ -2848,6 +3006,21 @@ "node": ">= 0.8" } }, + "node_modules/path-expression-matcher": { + "version": "1.6.2", + "resolved": "https://registry.npmjs.org/path-expression-matcher/-/path-expression-matcher-1.6.2.tgz", + "integrity": "sha512-enSlaiat05iasnzmgNxRj8reFdj3puY2QpNgP1aPIaVfT6nn9ICuPoFlKHk8EN22HcwewshO+mN2DGbkCEOtqQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, "node_modules/path-key": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", @@ -3282,6 +3455,21 @@ "dev": true, "license": "MIT" }, + "node_modules/strnum": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/strnum/-/strnum-2.4.1.tgz", + "integrity": "sha512-M9eUSMT2dCB2cTNPG7UYj6KuK7RJR2SN2+yCV/fTW3xzTCS6EaGZ5pSMgDIjB7r8zSfTGk+dvvn9rTjpVS9Mwg==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "anynum": "^1.0.1" + } + }, "node_modules/tinybench": { "version": "2.9.0", "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", @@ -3651,6 +3839,21 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/xml-naming": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/xml-naming/-/xml-naming-0.3.0.tgz", + "integrity": "sha512-ghig2TBE/H11aOVgmahA3MhimvkBr6JIYknH/Dhdk10nXwdbIqBJsbfMxpvFPG8bAw77gN29aQWvKpmVoPlvPQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "engines": { + "node": ">=16.0.0" + } + }, "node_modules/zod": { "version": "4.4.3", "resolved": "https://registry.npmjs.org/zod/-/zod-4.4.3.tgz", @@ -3671,11 +3874,12 @@ }, "packages/gateway": { "name": "@mspstack/mcp-gateway", - "version": "0.9.1", + "version": "0.12.0", "license": "MIT", "dependencies": { "@azure/identity": "^4.13.1", "@azure/keyvault-secrets": "^4.11.2", + "@azure/storage-blob": "^12.33.0", "@modelcontextprotocol/sdk": "^1.29.0", "express": "^5.2.1", "jose": "^6.0.11", diff --git a/packages/gateway/package.json b/packages/gateway/package.json index 252c364..87a879d 100644 --- a/packages/gateway/package.json +++ b/packages/gateway/package.json @@ -53,6 +53,7 @@ "dependencies": { "@azure/identity": "^4.13.1", "@azure/keyvault-secrets": "^4.11.2", + "@azure/storage-blob": "^12.33.0", "@modelcontextprotocol/sdk": "^1.29.0", "express": "^5.2.1", "jose": "^6.0.11", diff --git a/packages/gateway/src/config.ts b/packages/gateway/src/config.ts index 890470a..f951be2 100644 --- a/packages/gateway/src/config.ts +++ b/packages/gateway/src/config.ts @@ -39,6 +39,7 @@ */ import { readFileSync } from "node:fs"; +import { dirname } from "node:path"; import { createHash } from "node:crypto"; import { z } from "zod"; @@ -211,6 +212,13 @@ export interface GatewayConfig { publicUrl: string; configPath: string; dbPath: string; + /** Online SQLite snapshots (VACUUM INTO) + retention + optional off-instance copy. */ + backup: { + dir: string; + keep: number; + intervalHours: number; + blobContainerUrl?: string; + }; allowedOrigins: string[]; upstreamsFromFile: UpstreamSpec[]; staticTokens: StaticTokenEntry[]; @@ -503,12 +511,35 @@ export function loadConfig( } } + // ── backups ── + const backupIntervalRaw = cleanEnv(env.BACKUP_INTERVAL_HOURS) ?? "24"; + const backupIntervalHours = Number(backupIntervalRaw); + if (!Number.isFinite(backupIntervalHours) || backupIntervalHours < 0) { + throw new ConfigError(`BACKUP_INTERVAL_HOURS must be a non-negative number, got "${backupIntervalRaw}"`); + } + const backupKeepRaw = cleanEnv(env.BACKUP_KEEP) ?? "7"; + const backupKeep = Number(backupKeepRaw); + if (!Number.isInteger(backupKeep) || backupKeep < 1) { + throw new ConfigError(`BACKUP_KEEP must be a positive integer, got "${backupKeepRaw}"`); + } + const blobContainerUrl = cleanEnv(env.BACKUP_BLOB_CONTAINER_URL); + if (blobContainerUrl && !/^https:\/\//i.test(blobContainerUrl)) { + throw new ConfigError(`BACKUP_BLOB_CONTAINER_URL must be an https:// container URL, got "${blobContainerUrl}"`); + } + return { mode, port, publicUrl, configPath, dbPath, + backup: { + dir: cleanEnv(env.BACKUP_DIR) ?? `${dirname(dbPath)}/backups`, + keep: backupKeep, + // In-memory databases have nothing durable to snapshot. + intervalHours: dbPath === ":memory:" ? 0 : backupIntervalHours, + ...(blobContainerUrl ? { blobContainerUrl } : {}), + }, allowedOrigins, upstreamsFromFile: parseConfigFile(raw), staticTokens: parseStaticTokens(env), diff --git a/packages/gateway/src/db/backup.test.ts b/packages/gateway/src/db/backup.test.ts new file mode 100644 index 0000000..09bb295 --- /dev/null +++ b/packages/gateway/src/db/backup.test.ts @@ -0,0 +1,153 @@ +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import { describe, expect, it, vi } from "vitest"; +import { openDatabase } from "./index.js"; +import { Repo } from "./repo.js"; +import { listSnapshots, pruneSnapshots, runBackup, snapshot, startBackupSchedule } from "./backup.js"; + +const freshDir = () => mkdtempSync(join(tmpdir(), "gw-backup-")); + +/** A real on-disk gateway db with something worth backing up. */ +function seededDb(dir: string) { + const db = openDatabase(join(dir, "gateway.db")); + const repo = new Repo(db); + repo.upsertUpstream( + { id: "cipp", namespace: "cipp", transport: "http", url: "https://cipp/mcp", headers: {}, enabled: true } as never, + "api" + ); + repo.setGroupMapping("https://idp", "group-guid", repo.roleByName("admin")!.id); + return { db, repo }; +} + +describe("snapshot", () => { + it("writes a queryable copy of the live database", () => { + const dir = freshDir(); + const { db } = seededDb(dir); + const file = snapshot(db, join(dir, "backups"), new Date("2026-07-29T15:04:05.123Z")); + + expect(file.name).toBe("gateway-backup-2026-07-29T15-04-05-123Z.db"); + expect(file.sizeBytes).toBeGreaterThan(0); + + // The snapshot is a real database, not a torn file: open it and read back. + const restored = new DatabaseSync(file.path); + const rows = restored.prepare("SELECT id FROM upstreams").all() as Array<{ id: string }>; + expect(rows.map((r) => r.id)).toEqual(["cipp"]); + expect((restored.prepare("SELECT COUNT(*) c FROM group_mappings").get() as { c: number }).c).toBe(1); + restored.close(); + db.close(); + }); + + it("keeps serving while the snapshot is taken (writes after it are not lost)", () => { + const dir = freshDir(); + const { db, repo } = seededDb(dir); + snapshot(db, join(dir, "backups")); + repo.createRole("post-backup", "read"); // db still usable + expect(repo.roleByName("post-backup")).not.toBeNull(); + db.close(); + }); +}); + +describe("listSnapshots / pruneSnapshots", () => { + it("lists newest first and ignores foreign files", () => { + const dir = freshDir(); + const { db } = seededDb(dir); + const backups = join(dir, "backups"); + snapshot(db, backups, new Date("2026-07-01T00:00:00Z")); + snapshot(db, backups, new Date("2026-07-03T00:00:00Z")); + snapshot(db, backups, new Date("2026-07-02T00:00:00Z")); + writeFileSync(join(backups, "notes.txt"), "not a backup"); + + const listed = listSnapshots(backups); + expect(listed.map((f) => f.name)).toEqual([ + "gateway-backup-2026-07-03T00-00-00-000Z.db", + "gateway-backup-2026-07-02T00-00-00-000Z.db", + "gateway-backup-2026-07-01T00-00-00-000Z.db", + ]); + db.close(); + }); + + it("prunes the oldest beyond `keep`", () => { + const dir = freshDir(); + const { db } = seededDb(dir); + const backups = join(dir, "backups"); + for (const day of ["01", "02", "03", "04"]) { + snapshot(db, backups, new Date(`2026-07-${day}T00:00:00Z`)); + } + const removed = pruneSnapshots(backups, 2); + expect(removed).toEqual([ + "gateway-backup-2026-07-02T00-00-00-000Z.db", + "gateway-backup-2026-07-01T00-00-00-000Z.db", + ]); + expect(listSnapshots(backups)).toHaveLength(2); + db.close(); + }); + + it("returns [] for a directory that doesn't exist yet", () => { + expect(listSnapshots(join(freshDir(), "nope"))).toEqual([]); + }); +}); + +describe("runBackup", () => { + it("snapshots, prunes, and ships the file off-instance", async () => { + const dir = freshDir(); + const { db } = seededDb(dir); + const backups = join(dir, "backups"); + const uploaded: Array<{ name: string; bytes: number }> = []; + const upload = async (localPath: string, name: string) => { + uploaded.push({ name, bytes: readFileSync(localPath).length }); + }; + + for (let i = 0; i < 3; i += 1) { + await runBackup(db, { dir: backups, keep: 2, intervalHours: 0 }, upload); + } + expect(listSnapshots(backups)).toHaveLength(2); // retention held + expect(uploaded).toHaveLength(3); // every snapshot shipped + expect(uploaded[0]!.bytes).toBeGreaterThan(0); + db.close(); + }); + + it("an upload failure does not fail the backup — a local snapshot still beats none", async () => { + const dir = freshDir(); + const { db } = seededDb(dir); + const backups = join(dir, "backups"); + const failing = async () => { + throw new Error("blob unreachable"); + }; + const file = await runBackup(db, { dir: backups, keep: 3, intervalHours: 0 }, failing); + expect(listSnapshots(backups).map((f) => f.name)).toEqual([file.name]); + db.close(); + }); +}); + +describe("startBackupSchedule", () => { + it("is a no-op when the interval is 0 (e.g. :memory: databases)", () => { + const dir = freshDir(); + const { db } = seededDb(dir); + const stop = startBackupSchedule(db, { dir: join(dir, "backups"), keep: 3, intervalHours: 0 }); + stop(); + expect(listSnapshots(join(dir, "backups"))).toEqual([]); + db.close(); + }); + + it("runs on the configured interval and stops cleanly", async () => { + vi.useFakeTimers(); + const dir = freshDir(); + const { db } = seededDb(dir); + const backups = join(dir, "backups"); + const stop = startBackupSchedule(db, { dir: backups, keep: 5, intervalHours: 1 }); + try { + await vi.advanceTimersByTimeAsync(60 * 60 * 1000 + 10); + expect(listSnapshots(backups)).toHaveLength(1); + await vi.advanceTimersByTimeAsync(60 * 60 * 1000); + expect(listSnapshots(backups)).toHaveLength(2); + stop(); + await vi.advanceTimersByTimeAsync(3 * 60 * 60 * 1000); + expect(listSnapshots(backups)).toHaveLength(2); // stopped + } finally { + vi.useRealTimers(); + db.close(); + } + }); +}); diff --git a/packages/gateway/src/db/backup.ts b/packages/gateway/src/db/backup.ts new file mode 100644 index 0000000..39fc66e --- /dev/null +++ b/packages/gateway/src/db/backup.ts @@ -0,0 +1,164 @@ +/** + * Online backups of the gateway's SQLite database. + * + * The state here is not reproducible from config any more: upstream specs, + * roles/grants/overrides, tool settings, users and group mappings, DCR clients, + * refresh-token families, and the refs to every user's personal credentials. + * Losing the file means reconfiguring by hand. + * + * Mechanics: + * - `VACUUM INTO` writes a consistent snapshot while the gateway keeps + * serving — unlike copying the file, which can capture a torn page mid-WAL. + * - Snapshots are pruned to the newest N locally. + * - Optionally shipped off-instance (a backup next to the database is not a + * backup): any uploader can be injected; Azure Blob is provided lazily so + * the SDK is only loaded when a container URL is configured. + * + * Restore is deliberately manual: stop the app, put the snapshot at DB_PATH + * (drop any -wal/-shm siblings), start. Documented in docs/backups.md. + */ + +import { mkdirSync, readdirSync, statSync, unlinkSync } from "node:fs"; +import { basename, join } from "node:path"; +import type { DatabaseSync } from "node:sqlite"; + +/** File name prefix + shape: gateway-backup-2026-07-29T15-04-05-123Z.db */ +const PREFIX = "gateway-backup-"; +const SUFFIX = ".db"; + +export interface BackupConfig { + /** Directory snapshots are written to. */ + dir: string; + /** How many local snapshots to keep (oldest pruned first). */ + keep: number; + /** 0 disables the scheduler; snapshots can still be taken on demand. */ + intervalHours: number; + /** Container URL for off-instance copies, e.g. https://acct.blob.core.windows.net/gw-backups */ + blobContainerUrl?: string; +} + +export interface BackupFile { + name: string; + path: string; + sizeBytes: number; + createdAt: string; +} + +/** Ships a finished snapshot somewhere durable. Injectable for tests. */ +export type BackupUploader = (localPath: string, name: string) => Promise; + +const stamp = (at: Date): string => at.toISOString().replace(/[:.]/g, "-"); + +/** + * Take a snapshot. `VACUUM INTO` refuses to overwrite, so the timestamped name + * doubles as the uniqueness guarantee. Returns the file that was written. + */ +export function snapshot(db: DatabaseSync, dir: string, at: Date = new Date()): BackupFile { + mkdirSync(dir, { recursive: true }); + const name = `${PREFIX}${stamp(at)}${SUFFIX}`; + const path = join(dir, name); + // SQLite has no parameter binding for VACUUM INTO; the path is ours (config + // + generated name), and single quotes are escaped for good measure. + db.exec(`VACUUM INTO '${path.replace(/'/g, "''")}'`); + const stats = statSync(path); + return { name, path, sizeBytes: stats.size, createdAt: at.toISOString() }; +} + +/** Newest first. Ignores anything that isn't one of our snapshots. */ +export function listSnapshots(dir: string): BackupFile[] { + let names: string[]; + try { + names = readdirSync(dir); + } catch { + return []; + } + return names + .filter((n) => n.startsWith(PREFIX) && n.endsWith(SUFFIX)) + .map((name) => { + const path = join(dir, name); + const stats = statSync(path); + return { name, path, sizeBytes: stats.size, createdAt: stats.mtime.toISOString() }; + }) + .sort((a, b) => b.name.localeCompare(a.name)); +} + +/** Keep the newest `keep` snapshots; returns the names removed. */ +export function pruneSnapshots(dir: string, keep: number): string[] { + if (keep <= 0) return []; + const removed: string[] = []; + for (const file of listSnapshots(dir).slice(keep)) { + try { + unlinkSync(file.path); + removed.push(file.name); + } catch (err) { + console.error(`[backup] could not prune ${file.name}: ${String(err)}`); + } + } + return removed; +} + +/** + * Snapshot + prune (+ upload when configured). Upload failures are logged and + * do NOT fail the backup: a local snapshot that exists beats none at all. + */ +export async function runBackup( + db: DatabaseSync, + config: BackupConfig, + upload?: BackupUploader +): Promise { + const file = snapshot(db, config.dir); + const pruned = pruneSnapshots(config.dir, config.keep); + console.error( + `[backup] wrote ${file.name} (${Math.round(file.sizeBytes / 1024)} KiB)` + + (pruned.length ? `, pruned ${pruned.length}` : "") + ); + if (upload) { + try { + await upload(file.path, file.name); + console.error(`[backup] uploaded ${file.name} off-instance`); + } catch (err) { + console.error(`[backup] off-instance upload FAILED for ${file.name}: ${String(err)}`); + } + } + return file; +} + +/** + * Azure Blob uploader using DefaultAzureCredential (same identity story as the + * Key Vault store). Lazily imported so deployments without it pay nothing. + */ +export async function createBlobUploader(containerUrl: string): Promise { + const [{ DefaultAzureCredential }, { ContainerClient }] = await Promise.all([ + import("@azure/identity"), + import("@azure/storage-blob"), + ]); + const container = new ContainerClient(containerUrl, new DefaultAzureCredential()); + return async (localPath, name) => { + await container.getBlockBlobClient(basename(name)).uploadFile(localPath); + }; +} + +/** + * Start the periodic backup loop. Returns a stop function; the timer is + * unref'd so it never holds the process open on shutdown. + */ +export function startBackupSchedule( + db: DatabaseSync, + config: BackupConfig, + upload?: BackupUploader +): () => void { + if (config.intervalHours <= 0) return () => undefined; + const everyMs = config.intervalHours * 60 * 60 * 1000; + const tick = (): void => { + runBackup(db, config, upload).catch((err) => + console.error(`[backup] scheduled backup failed: ${String(err)}`) + ); + }; + const timer = setInterval(tick, everyMs); + timer.unref(); + console.error( + `[backup] every ${config.intervalHours}h → ${config.dir} (keep ${config.keep})` + + (config.blobContainerUrl ? " + off-instance copy" : " — LOCAL ONLY, no off-instance copy") + ); + return () => clearInterval(timer); +} diff --git a/packages/gateway/src/http/admin-api.ts b/packages/gateway/src/http/admin-api.ts index fdd12e643319762ad5113c85e007ac371a825ac6..9cda0b146c92933c1cfd8e42458e6a04ae65d82e 100644 GIT binary patch delta 677 zcmZvZ-%1-n6vm0rAnEp|H`-dvNvLH*w;QBh1(H}(YP=I9AmYv0%xoQ;%q%mriiRM4 zgfiXR)obUX+tvuXS=4|ByXWASIjP#i_+5;7f zJsh9M5zHV|aso}?Z!ydk>4yHhu^5C4($O9$} zRTs#IGFAya;4&CUVUP)`Mv5mzxbxoyt($zlWmR { ); console.error(`[presets] ${presets.length} installable preset(s) available`); + // ── backups ── + // State here isn't reproducible from config (upstreams, roles, users, DCR + // clients, credential refs), so snapshot it while running. + const backupUploader = config.backup.blobContainerUrl + ? await createBlobUploader(config.backup.blobContainerUrl) + : undefined; + const stopBackups = startBackupSchedule(db, config.backup, backupUploader); + // ── secrets ── const secretStore = config.bao ? new OpenBaoStore(config.bao) @@ -117,6 +126,8 @@ async function main(): Promise { const adminUiDir = fileURLToPath(new URL("../public", import.meta.url)); const app = createApp({ config, + db, + backupUploader, repo, manager, policy, @@ -138,6 +149,7 @@ async function main(): Promise { const shutdown = (signal: string): void => { console.error(`[gateway] ${signal} received — shutting down`); + stopBackups(); httpServer.close(); manager .stop() From 3dce8d91b9ba33e8c2add6b434f2a711c3b5d12e Mon Sep 17 00:00:00 2001 From: Eugene Samotija Date: Thu, 30 Jul 2026 03:09:57 -0400 Subject: [PATCH 2/3] mcp: admin-only self-management toolset (gw_*) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #2. Routine gateway administration — what's connected, which groups are enabled, turn a tier off, raise a tool's tier, close an upstream for a role, install a preset, snapshot the database — is now doable from any MCP client instead of a browser trip. Boundaries, deliberately narrow: the tools are listed only for admins and every call re-checks isAdmin (list filtering is UX, the call check is the boundary), with the same "not available" text an unknown tool gets so the set is no existence oracle. The `gw` namespace is now reserved, so a federated server can neither shadow these names nor be shadowed by them. gw_list_servers redacts header/env VALUES — keys plus ref-vs-env-vs-literal shape only, so a spec with a literal token can't be read back out over MCP. Credential writes, secret reads and user administration stay on the HTTP surface. gw_remove_server requires confirm: true. GATEWAY_SELF_TOOLS=off removes the whole surface. domain/tool-targets.ts extracts the "which tools does this selector touch" resolution now shared by the admin API, the /me prefs API and this toolset, so the three can't drift on what a tier or group selector means. Co-Authored-By: Claude Fable 5 --- docs/self-management.md | 56 +++ packages/gateway/src/config.ts | 18 + packages/gateway/src/domain/tool-targets.ts | 56 +++ packages/gateway/src/http/app.test.ts | 68 ++- packages/gateway/src/http/app.ts | 32 +- packages/gateway/src/http/me-api.test.ts | 2 + packages/gateway/src/http/oauth-flow.test.ts | 2 + packages/gateway/src/mcp/gateway-server.ts | 27 +- packages/gateway/src/mcp/self-tools.test.ts | 228 +++++++++ packages/gateway/src/mcp/self-tools.ts | 443 ++++++++++++++++++ .../src/upstream/personal-sessions.test.ts | 2 + 11 files changed, 921 insertions(+), 13 deletions(-) create mode 100644 docs/self-management.md create mode 100644 packages/gateway/src/domain/tool-targets.ts create mode 100644 packages/gateway/src/mcp/self-tools.test.ts create mode 100644 packages/gateway/src/mcp/self-tools.ts diff --git a/docs/self-management.md b/docs/self-management.md new file mode 100644 index 0000000..016ac08 --- /dev/null +++ b/docs/self-management.md @@ -0,0 +1,56 @@ +# Administering the gateway over MCP + +The gateway exposes its own administration as an **admin-only** MCP toolset +(`gw_*`), so routine changes are a sentence in Claude Code instead of a browser +trip: + +> "which CIPP groups are enabled?" · "turn off everything destructive in cwpsa" +> · "raise ExecGetRecoveryKey to destructive" · "close cipp for techs-ro" + +## Access model + +- Tools are listed **only** for principals whose role has `is_admin` — for + everyone else the names simply do not appear. +- Hiding is UX; every call re-checks `isAdmin` and a non-admin gets the same + "not available" text an unknown tool gets, so the toolset is not an oracle + for what exists. +- The `gw` namespace is reserved: `parseUpstreamSpec` refuses it, so a + federated server can never shadow these tools or be shadowed by them. +- Turn the whole thing off with `GATEWAY_SELF_TOOLS=off`. + +## What it will not do + +- **No secret reads.** `gw_list_servers` redacts header/env values: you see the + keys and whether each value is a `kv:`/`bao:` reference, a `${VAR}` env + reference, or a literal — never the literal itself. +- **No credential writes and no impersonation.** Personal credentials, secret + writes, and user/role administration stay on the HTTP surface where the + browser session gates them. +- **No silent destruction.** `gw_remove_server` requires `confirm: true`. + +## The tools + +| Tool | What it does | +| --- | --- | +| `gw_status` | version/mode, upstreams with tool counts and last errors, catalog size, secret-store scheme, auth mode, backup settings | +| `gw_list_servers` | configured upstreams and their mode flags, credential values redacted | +| `gw_list_tools` | catalog rows with effective tier, group and enabled state; filter by upstream/tier/group/enabled/name, or `groupsOnly` for a per-category summary | +| `gw_set_tools_enabled` | enable/disable for everyone — scope by tier and/or group, one tool, or the whole upstream | +| `gw_set_tool_tier` | set or clear a tier override (how a read-only tool that hands out secrets is kept away from low roles) | +| `gw_set_grant` | a role's ceiling on one upstream, by role name | +| `gw_list_presets` / `gw_install_preset` | the preset catalog and one-shot installs (`dryRun` renders without saving) | +| `gw_set_server_enabled` | enable/disable a whole upstream | +| `gw_remove_server` | remove an upstream and its settings/grants/overrides (`confirm: true`) | +| `gw_refresh_catalog` | re-read every upstream's tool list now | +| `gw_backup_now` | snapshot the database, prune to retention, ship off-instance when configured | + +Changes broadcast `tools/list_changed` to live sessions, exactly like the same +change made from `/admin`. + +## Secrets in preset installs + +`gw_install_preset` takes parameters as strings, and a secret parameter must be +a **reference** — `kv:cipp-mcp-secret`, `bao:upstreams/itglue#token`, or +`${SOME_ENV}`. Pasting a raw secret would store it literally in the upstream +spec; write it with `PUT /api/secrets` (or the Secrets tab) first and pass the +ref you get back. diff --git a/packages/gateway/src/config.ts b/packages/gateway/src/config.ts index f951be2..5df841e 100644 --- a/packages/gateway/src/config.ts +++ b/packages/gateway/src/config.ts @@ -48,6 +48,12 @@ export class ConfigError extends Error {} /** Namespaces exclude "_" so exposed tool names stay unambiguous. */ const NAMESPACE_RE = /^[a-z0-9]+$/; +/** + * Reserved for the gateway's own self-management tools (`gw_*`), so a federated + * server can never shadow them — or be shadowed by them. + */ +export const RESERVED_NAMESPACES = new Set(["gw"]); + const upstreamBase = { id: z.string().min(1), namespace: z @@ -212,6 +218,12 @@ export interface GatewayConfig { publicUrl: string; configPath: string; dbPath: string; + /** + * Expose the admin-only self-management tools (`gw_*`) over MCP. On by + * default; `GATEWAY_SELF_TOOLS=off` keeps conversational administration out + * of a deployment entirely. + */ + selfTools: boolean; /** Online SQLite snapshots (VACUUM INTO) + retention + optional off-instance copy. */ backup: { dir: string; @@ -269,6 +281,11 @@ export function parseUpstreamSpec(json: unknown): UpstreamSpec { const parsed = upstreamSpecSchema.safeParse(json); if (!parsed.success) throw new ConfigError(`Invalid upstream: ${parsed.error.message}`); const spec = parsed.data; + if (RESERVED_NAMESPACES.has(spec.namespace)) { + throw new ConfigError( + `upstream "${spec.id}": namespace "${spec.namespace}" is reserved by the gateway's own tools — pick another` + ); + } // URLs may contain ${VAR}/bao: refs resolved at connect time — only // validate the shape when the value is already concrete. if (spec.transport === "http" && !spec.url.includes("${") && !spec.url.startsWith("bao:")) { @@ -533,6 +550,7 @@ export function loadConfig( publicUrl, configPath, dbPath, + selfTools: (cleanEnv(env.GATEWAY_SELF_TOOLS) ?? "on").toLowerCase() !== "off", backup: { dir: cleanEnv(env.BACKUP_DIR) ?? `${dirname(dbPath)}/backups`, keep: backupKeep, diff --git a/packages/gateway/src/domain/tool-targets.ts b/packages/gateway/src/domain/tool-targets.ts new file mode 100644 index 0000000..aa1074c --- /dev/null +++ b/packages/gateway/src/domain/tool-targets.ts @@ -0,0 +1,56 @@ +/** + * Resolving "which tools does this bulk action touch" — shared by the admin + * API, the /me prefs API, and the self-management MCP toolset so the three can + * never disagree about what a tier or group selector means. + * + * Targets always come from the LIVE catalog (and, for user-facing callers, from + * the caller's own envelope), so a stale UI or a guessing client cannot create + * settings rows for tools that don't exist or were never visible to it. + */ + +import type { Repo } from "../db/repo.js"; +import type { CatalogEntry, Tier } from "./catalog.js"; +import { derivedGroupOf } from "./catalog.js"; + +export interface ToolSelector { + upstreamId: string; + /** Effective tier (override ?? derived) — what the UI displays. */ + tier?: Tier; + /** Explicit group label, else the category derived from the description. */ + group?: string; + /** A single tool; "" means the whole upstream (prefs use that convention). */ + toolName?: string; +} + +/** Effective tier of an entry: an admin override wins over the annotation. */ +export const effectiveTierOf = (repo: Repo, entry: CatalogEntry): Tier => + repo.toolSetting(entry.upstreamId, entry.upstreamToolName)?.tierOverride ?? entry.tier; + +/** Effective group: explicit label wins over the derived category. */ +export const effectiveGroupOf = (repo: Repo, entry: CatalogEntry): string => + repo.toolSetting(entry.upstreamId, entry.upstreamToolName)?.groupLabel ?? + derivedGroupOf(entry.tool) ?? + ""; + +/** + * Filter `entries` down to the selector's targets. `entries` is whatever the + * caller is allowed to act on: the whole catalog for admins, the principal's + * visible envelope for /me. + */ +export function resolveToolTargets( + repo: Repo, + entries: Iterable, + selector: ToolSelector +): CatalogEntry[] { + const targets: CatalogEntry[] = []; + for (const entry of entries) { + if (entry.upstreamId !== selector.upstreamId) continue; + if (selector.toolName !== undefined && selector.toolName !== "" && entry.upstreamToolName !== selector.toolName) { + continue; + } + if (selector.tier && effectiveTierOf(repo, entry) !== selector.tier) continue; + if (selector.group !== undefined && effectiveGroupOf(repo, entry) !== selector.group) continue; + targets.push(entry); + } + return targets; +} diff --git a/packages/gateway/src/http/app.test.ts b/packages/gateway/src/http/app.test.ts index 6e03216..59124ed 100644 --- a/packages/gateway/src/http/app.test.ts +++ b/packages/gateway/src/http/app.test.ts @@ -48,6 +48,8 @@ const config: GatewayConfig = { publicUrl: "http://localhost:0", configPath: "unused", dbPath: ":memory:", + selfTools: true, + backup: { dir: "unused", keep: 3, intervalHours: 0 }, allowedOrigins: [], upstreamsFromFile: [], staticTokens: [ @@ -146,6 +148,10 @@ const listTools = async (token: string, sid: string) => (t) => t.name ); +/** Federated tools only — admins also get the built-in gw_* self-management set. */ +const listFederated = async (token: string, sid: string) => + (await listTools(token, sid)).filter((n) => !n.startsWith("gw_")); + describe("gateway HTTP app", () => { it("serves /health without auth, reporting login availability", async () => { const response = await fetch(`${base}/health`); @@ -175,7 +181,7 @@ describe("gateway HTTP app", () => { expect(await listTools("tok-viewer", viewerSid)).toEqual(["fake_read_thing"]); const adminSid = await initSession("tok-admin"); - expect((await listTools("tok-admin", adminSid)).sort()).toEqual([ + expect((await listFederated("tok-admin", adminSid)).sort()).toEqual([ "fake_read_thing", "fake_write_thing", ]); @@ -208,7 +214,7 @@ describe("gateway HTTP app", () => { repo.upsertToolSetting({ upstreamId: "fake", toolName: "read_thing", enabled: false }); try { const sid = await initSession("tok-admin"); - expect(await listTools("tok-admin", sid)).toEqual(["fake_write_thing"]); + expect(await listFederated("tok-admin", sid)).toEqual(["fake_write_thing"]); } finally { repo.upsertToolSetting({ upstreamId: "fake", toolName: "read_thing", enabled: true }); } @@ -299,6 +305,64 @@ describe("admin directory search endpoint", () => { }); }); +describe("self-management tools over MCP", () => { + it("are offered to admins only, and a viewer's call is refused like an unknown tool", async () => { + const adminSid = await initSession("tok-admin"); + const adminTools = await listTools("tok-admin", adminSid); + expect(adminTools).toContain("gw_status"); + expect(adminTools).toContain("gw_set_tools_enabled"); + + const viewerSid = await initSession("tok-viewer"); + const viewerTools = await listTools("tok-viewer", viewerSid); + expect(viewerTools.some((n) => n.startsWith("gw_"))).toBe(false); + + // Hidden isn't enough — calling it anyway must fail at the boundary. + const denied = await rpc( + { jsonrpc: "2.0", id: 9, method: "tools/call", params: { name: "gw_status", arguments: {} } }, + "tok-viewer", + viewerSid + ); + expect(denied.json?.result?.isError).toBe(true); + expect(denied.json?.result?.content?.[0]?.text).toContain("not available"); + }); + + it("an admin can inspect and change the catalog conversationally", async () => { + const sid = await initSession("tok-admin"); + const status = await rpc( + { jsonrpc: "2.0", id: 10, method: "tools/call", params: { name: "gw_status", arguments: {} } }, + "tok-admin", + sid + ); + const reported = JSON.parse(status.json!.result!.content![0]!.text) as { toolCount: number }; + expect(reported.toolCount).toBe(2); // read_thing + write_thing + + const disabled = await rpc( + { + jsonrpc: "2.0", + id: 11, + method: "tools/call", + params: { name: "gw_set_tools_enabled", arguments: { upstreamId: "fake", tier: "write", enabled: false } }, + }, + "tok-admin", + sid + ); + expect(disabled.json?.result?.content?.[0]?.text).toContain("Disabled 1 tool(s)"); + expect(repo.toolSetting("fake", "write_thing")?.enabled).toBe(false); + + // restore + await rpc( + { + jsonrpc: "2.0", + id: 12, + method: "tools/call", + params: { name: "gw_set_tools_enabled", arguments: { upstreamId: "fake", tier: "write", enabled: true } }, + }, + "tok-admin", + sid + ); + }); +}); + describe("bulk catalog toggle", () => { const bulk = (body: unknown, upstream = "fake") => fetch(`${base}/api/catalog/${upstream}`, { diff --git a/packages/gateway/src/http/app.ts b/packages/gateway/src/http/app.ts index d3d5cee..c09d03e 100644 --- a/packages/gateway/src/http/app.ts +++ b/packages/gateway/src/http/app.ts @@ -446,13 +446,31 @@ export function createApp(deps: AppDeps): express.Express { } const principal = auth.principal; - const server = createGatewayServer(manager, policy, principal, (upstreamId) => - Object.fromEntries( - deps.repo - .listUserCredentials(prefsIdentity(principal)) - .filter((row) => row.upstreamId === upstreamId) - .map((row) => [row.field, row.secretRef]) - ) + const server = createGatewayServer( + manager, + policy, + principal, + (upstreamId) => + Object.fromEntries( + deps.repo + .listUserCredentials(prefsIdentity(principal)) + .filter((row) => row.upstreamId === upstreamId) + .map((row) => [row.field, row.secretRef]) + ), + // Admin-only conversational administration; the toolset itself + // re-checks isAdmin on every call. + config.selfTools + ? { + config, + repo: deps.repo, + manager, + policy, + presets: deps.presets ?? [], + ...(deps.db ? { db: deps.db } : {}), + ...(deps.backupUploader ? { backupUploader: deps.backupUploader } : {}), + onPolicyChanged: broadcastVisibility, + } + : undefined ); const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: () => randomUUID(), diff --git a/packages/gateway/src/http/me-api.test.ts b/packages/gateway/src/http/me-api.test.ts index 9c5c92d..ea23d13 100644 --- a/packages/gateway/src/http/me-api.test.ts +++ b/packages/gateway/src/http/me-api.test.ts @@ -59,6 +59,8 @@ const config: GatewayConfig = { publicUrl: "http://localhost:0", configPath: "unused", dbPath: ":memory:", + selfTools: true, + backup: { dir: "unused", keep: 3, intervalHours: 0 }, allowedOrigins: [], upstreamsFromFile: [], staticTokens: [ diff --git a/packages/gateway/src/http/oauth-flow.test.ts b/packages/gateway/src/http/oauth-flow.test.ts index b95e72a..b3b6993 100644 --- a/packages/gateway/src/http/oauth-flow.test.ts +++ b/packages/gateway/src/http/oauth-flow.test.ts @@ -31,6 +31,8 @@ const config: GatewayConfig = { publicUrl: PUBLIC_URL, configPath: "unused", dbPath: ":memory:", + selfTools: true, + backup: { dir: "unused", keep: 3, intervalHours: 0 }, allowedOrigins: [], upstreamsFromFile: [], staticTokens: [ diff --git a/packages/gateway/src/mcp/gateway-server.ts b/packages/gateway/src/mcp/gateway-server.ts index de50db9..f131d80 100644 --- a/packages/gateway/src/mcp/gateway-server.ts +++ b/packages/gateway/src/mcp/gateway-server.ts @@ -15,6 +15,7 @@ import { import { prefsIdentity, type Principal } from "../auth/principal.js"; import type { PolicyService } from "../domain/policy.js"; import type { UpstreamManager } from "../upstream/manager.js"; +import { callSelfTool, SELF_TOOLS, type SelfToolDeps } from "./self-tools.js"; import { SERVER_NAME, SERVER_VERSION } from "../version.js"; @@ -27,7 +28,9 @@ export function createGatewayServer( manager: UpstreamManager, policy: PolicyService, principal: Principal, - personalCredsFor?: PersonalCredsLookup + personalCredsFor?: PersonalCredsLookup, + /** Admin-only self-management tools; omit to serve federated tools only. */ + selfTools?: SelfToolDeps ): Server { const server = new Server( { name: SERVER_NAME, version: SERVER_VERSION }, @@ -37,12 +40,28 @@ export function createGatewayServer( // Envelope ∧ personal prefs — the same allowsFor gates list AND call, so // a user's own narrowing is enforced at the boundary, not just hidden in UX. server.setRequestHandler(ListToolsRequestSchema, async () => ({ - tools: policy - .visibleEntriesFor(principal, manager.catalogEntries()) - .map((entry) => ({ ...entry.tool, name: entry.exposedName })), + tools: [ + // Self-management is offered to admins only; for everyone else these + // names simply don't exist (the call handler re-checks anyway). + ...(selfTools && principal.isAdmin ? SELF_TOOLS : []), + ...policy + .visibleEntriesFor(principal, manager.catalogEntries()) + .map((entry) => ({ ...entry.tool, name: entry.exposedName })), + ], })); server.setRequestHandler(CallToolRequestSchema, async (request) => { + // The `gw` namespace is reserved (config.ts refuses it for upstreams), so + // this can never shadow a federated tool or be shadowed by one. + if (selfTools) { + const handled = await callSelfTool( + selfTools, + principal, + request.params.name, + request.params.arguments ?? {} + ); + if (handled) return handled; + } const entry = manager.entryFor(request.params.name); if (!entry || !policy.allowsFor(principal, entry)) { // Same response for unknown and forbidden — no tool-existence oracle. diff --git a/packages/gateway/src/mcp/self-tools.test.ts b/packages/gateway/src/mcp/self-tools.test.ts new file mode 100644 index 0000000..9b29240 --- /dev/null +++ b/packages/gateway/src/mcp/self-tools.test.ts @@ -0,0 +1,228 @@ +/** + * Self-management toolset: the admin gate (list AND call), what the tools + * actually change, and that server specs never leak credential values. + */ + +import { describe, expect, it } from "vitest"; +import type { Tool, CallToolResult } from "@modelcontextprotocol/sdk/types.js"; +import { openDatabase } from "../db/index.js"; +import { Repo } from "../db/repo.js"; +import { PolicyService } from "../domain/policy.js"; +import { UpstreamManager, type UpstreamLink } from "../upstream/manager.js"; +import { BUILTIN_PRESETS } from "../domain/presets.js"; +import { loadConfig, parseUpstreamSpec, ConfigError } from "../config.js"; +import type { GatewayConfig, UpstreamSpec } from "../config.js"; +import type { Principal } from "../auth/principal.js"; +import { callSelfTool, isSelfTool, SELF_TOOLS, type SelfToolDeps } from "./self-tools.js"; + +const spec: UpstreamSpec = { + id: "fake", + namespace: "fake", + transport: "http", + url: "http://unused/mcp", + headers: { Authorization: "Bearer literal-secret-value", "x-ref": "kv:some-secret", "x-env": "${SOME_VAR}" }, + enabled: true, +} as UpstreamSpec; + +const tools: Tool[] = [ + { name: "get_thing", inputSchema: { type: "object" }, annotations: { readOnlyHint: true }, description: "[Docs] read it" }, + { name: "set_thing", inputSchema: { type: "object" }, description: "[Docs] write it" }, + { name: "nuke_thing", inputSchema: { type: "object" }, annotations: { destructiveHint: true }, description: "[Admin] remove it" }, +]; + +const link: UpstreamLink = { + spec, + onToolListChanged: null, + onRecovered: null, + async connect() {}, + async listTools() { + return tools; + }, + async callTool() { + return { content: [{ type: "text", text: "ok" }] }; + }, + async close() {}, +}; + +async function setup() { + const repo = new Repo(openDatabase(":memory:")); + repo.upsertUpstream(spec, "api"); + const manager = new UpstreamManager([spec], () => link); + await manager.start(); + const config: GatewayConfig = loadConfig(["--transport", "http"], { + MCP_TOKENS_ADMIN: "root:tok", + DB_PATH: ":memory:", + } as NodeJS.ProcessEnv); + let broadcasts = 0; + const deps: SelfToolDeps = { + config, + repo, + manager, + policy: new PolicyService(repo), + presets: BUILTIN_PRESETS, + onPolicyChanged: () => { + broadcasts += 1; + }, + }; + const admin: Principal = { kind: "static", subject: "root", label: "root", roleId: repo.roleByName("admin")!.id, roleName: "admin", isAdmin: true }; + const viewer: Principal = { kind: "static", subject: "alice", label: "alice", roleId: repo.roleByName("viewer")!.id, roleName: "viewer", isAdmin: false }; + return { repo, manager, deps, admin, viewer, broadcasts: () => broadcasts }; +} + +const body = (result: CallToolResult | null): string => + String((result?.content?.[0] as { text?: string } | undefined)?.text ?? ""); +const parsed = (result: CallToolResult | null): Record => JSON.parse(body(result)); + +describe("self-tool registry", () => { + it("names every tool under the reserved gw_ namespace", () => { + expect(SELF_TOOLS.length).toBeGreaterThan(8); + for (const tool of SELF_TOOLS) { + expect(tool.name.startsWith("gw_")).toBe(true); + expect(isSelfTool(tool.name)).toBe(true); + } + expect(isSelfTool("fake_get_thing")).toBe(false); + }); + + it("reserves the gw namespace so an upstream cannot shadow the tools", () => { + expect(() => + parseUpstreamSpec({ id: "x", namespace: "gw", transport: "http", url: "http://x/mcp" }) + ).toThrow(ConfigError); + }); +}); + +describe("admin gate", () => { + it("refuses non-admins with the same text an unknown tool gets", async () => { + const { deps, viewer } = await setup(); + const result = await callSelfTool(deps, viewer, "gw_status", {}); + expect(result?.isError).toBe(true); + expect(body(result)).toContain("is not available to this session"); + expect(body(result)).toContain("may not exist"); // no existence oracle + }); + + it("returns null for names that aren't ours, so federation still runs", async () => { + const { deps, admin } = await setup(); + expect(await callSelfTool(deps, admin, "fake_get_thing", {})).toBeNull(); + }); +}); + +describe("read-only tools", () => { + it("gw_status reports the catalog and upstreams", async () => { + const { deps, admin } = await setup(); + const status = parsed(await callSelfTool(deps, admin, "gw_status", {})); + expect(status.toolCount).toBe(3); + expect((status.upstreams as Array<{ id: string }>).map((u) => u.id)).toEqual(["fake"]); + }); + + it("gw_list_servers redacts literal credential values but shows refs", async () => { + const { deps, admin } = await setup(); + const servers = parsed(await callSelfTool(deps, admin, "gw_list_servers", {})) as unknown as Array<{ + headers: Record; + }>; + const headers = servers[0]!.headers; + expect(headers.Authorization).toBe("(literal, redacted)"); + expect(headers.Authorization).not.toContain("literal-secret-value"); + expect(headers["x-ref"]).toBe("ref kv:some-secret"); + expect(headers["x-env"]).toBe("env ${SOME_VAR}"); + }); + + it("gw_list_tools filters by tier and summarises by group", async () => { + const { deps, admin } = await setup(); + const reads = parsed(await callSelfTool(deps, admin, "gw_list_tools", { tier: "read" })); + expect(reads.total).toBe(1); + + const groups = parsed(await callSelfTool(deps, admin, "gw_list_tools", { groupsOnly: true })) as unknown as { + groups: Array<{ group: string; total: number }>; + }; + // groups derive from the "[Docs]" / "[Admin]" description prefixes + expect(groups.groups.map((g) => `${g.group}:${g.total}`).sort()).toEqual(["Admin:1", "Docs:2"]); + }); +}); + +describe("mutating tools", () => { + it("gw_set_tools_enabled disables a whole tier and notifies sessions", async () => { + const { deps, admin, repo, broadcasts } = await setup(); + const result = await callSelfTool(deps, admin, "gw_set_tools_enabled", { + upstreamId: "fake", + tier: "write", + enabled: false, + }); + expect(body(result)).toContain("Disabled 1 tool(s)"); + expect(repo.toolSetting("fake", "set_thing")?.enabled).toBe(false); + expect(repo.toolSetting("fake", "get_thing")?.enabled ?? true).toBe(true); + expect(broadcasts()).toBe(1); + }); + + it("gw_set_tools_enabled reports a miss instead of silently doing nothing", async () => { + const { deps, admin } = await setup(); + const result = await callSelfTool(deps, admin, "gw_set_tools_enabled", { + upstreamId: "fake", + group: "Nope", + enabled: false, + }); + expect(result?.isError).toBe(true); + expect(body(result)).toContain("Nothing matched"); + }); + + it("gw_set_tool_tier raises and clears an override, and rejects unknown tools", async () => { + const { deps, admin, repo } = await setup(); + await callSelfTool(deps, admin, "gw_set_tool_tier", { upstreamId: "fake", toolName: "get_thing", tier: "destructive" }); + expect(repo.toolSetting("fake", "get_thing")?.tierOverride).toBe("destructive"); + + await callSelfTool(deps, admin, "gw_set_tool_tier", { upstreamId: "fake", toolName: "get_thing", tier: null }); + expect(repo.toolSetting("fake", "get_thing")?.tierOverride).toBeNull(); + + const bad = await callSelfTool(deps, admin, "gw_set_tool_tier", { upstreamId: "fake", toolName: "ghost", tier: "read" }); + expect(bad?.isError).toBe(true); + }); + + it("gw_set_grant resolves the role by name and rejects unknown names", async () => { + const { deps, admin, repo } = await setup(); + await callSelfTool(deps, admin, "gw_set_grant", { roleName: "viewer", upstreamId: "fake", maxTier: "none" }); + expect(repo.grantFor(repo.roleByName("viewer")!.id, "fake")).toBe("none"); + + const bad = await callSelfTool(deps, admin, "gw_set_grant", { roleName: "ghosts", upstreamId: "fake", maxTier: "read" }); + expect(bad?.isError).toBe(true); + expect(body(bad)).toContain("existing roles"); + }); + + it("gw_remove_server refuses without confirm, then cascades", async () => { + const { deps, admin, repo } = await setup(); + repo.upsertToolSetting({ upstreamId: "fake", toolName: "get_thing", enabled: false }); + + const refused = await callSelfTool(deps, admin, "gw_remove_server", { upstreamId: "fake", confirm: false }); + expect(refused?.isError).toBe(true); + expect(repo.getUpstream("fake")).not.toBeNull(); + + await callSelfTool(deps, admin, "gw_remove_server", { upstreamId: "fake", confirm: true }); + expect(repo.getUpstream("fake")).toBeNull(); + expect(repo.toolSetting("fake", "get_thing")).toBeNull(); + }); + + it("gw_install_preset dry-runs without saving, then installs with grants", async () => { + const { deps, admin, repo } = await setup(); + const params = { url: "https://itglue.example/mcp", token: "kv:itglue-token" }; + + const dry = parsed(await callSelfTool(deps, admin, "gw_install_preset", { presetId: "itglue", params, dryRun: true })); + expect(dry.dryRun).toBe(true); + expect(repo.getUpstream("itglue")).toBeNull(); + + const done = parsed(await callSelfTool(deps, admin, "gw_install_preset", { presetId: "itglue", params })); + expect(done.installed).toBe("itglue"); + expect(done.grants).toEqual(["viewer=read", "editor=write"]); + expect(repo.getUpstream("itglue")).not.toBeNull(); + }); + + it("gw_install_preset surfaces a missing required parameter", async () => { + const { deps, admin } = await setup(); + const result = await callSelfTool(deps, admin, "gw_install_preset", { presetId: "itglue", params: {} }); + expect(result?.isError).toBe(true); + expect(body(result)).toContain("Missing required parameter"); + }); + + it("gw_backup_now explains itself when no database handle was wired", async () => { + const { deps, admin } = await setup(); + const result = await callSelfTool(deps, admin, "gw_backup_now", {}); + expect(result?.isError).toBe(true); + expect(body(result)).toContain("No database handle"); + }); +}); diff --git a/packages/gateway/src/mcp/self-tools.ts b/packages/gateway/src/mcp/self-tools.ts new file mode 100644 index 0000000..4a0d032 --- /dev/null +++ b/packages/gateway/src/mcp/self-tools.ts @@ -0,0 +1,443 @@ +/** + * Self-management toolset: administer the gateway through its own protocol + * (issue #2), so a "turn off CIPP's destructive tools" or "what's connected?" + * is a sentence in Claude Code instead of a browser trip. + * + * Boundaries, deliberately narrow: + * - ADMIN ONLY. The tools are hidden from tools/list for everyone else, and + * every call re-checks `principal.isAdmin` (list filtering is UX, the call + * check is the boundary). A non-admin gets the same "not available" text a + * non-existent tool gets — no oracle for what exists. + * - The `gw` namespace is reserved (config.ts rejects it for upstreams), so a + * federated server can never shadow these names or be shadowed by them. + * - Secrets never come back out: server specs are returned with header/env + * VALUES redacted, keys and ref-vs-literal shape only. + * - No credential writes, no user impersonation, no secret reads — those stay + * on the HTTP admin surface where the browser session gates them. + */ + +import type { Tool, CallToolResult } from "@modelcontextprotocol/sdk/types.js"; +import type { DatabaseSync } from "node:sqlite"; +import type { GatewayConfig, UpstreamSpec } from "../config.js"; +import type { Repo } from "../db/repo.js"; +import type { Preset } from "../domain/presets.js"; +import type { PolicyService, Tier } from "../domain/policy.js"; +import type { UpstreamManager } from "../upstream/manager.js"; +import type { Principal } from "../auth/principal.js"; +import { isMaxTier } from "../domain/policy.js"; +import { renderPreset } from "../domain/presets.js"; +import { effectiveGroupOf, effectiveTierOf, resolveToolTargets } from "../domain/tool-targets.js"; +import { runBackup } from "../db/backup.js"; +import type { BackupUploader } from "../db/backup.js"; + +/** Reserved namespace: `config.ts` refuses it for upstreams. */ +export const SELF_NAMESPACE = "gw"; + +/** Cap on how many tool rows one listing returns — keep responses readable. */ +const LIST_CAP = 150; + +export interface SelfToolDeps { + config: GatewayConfig; + repo: Repo; + manager: UpstreamManager; + policy: PolicyService; + presets: Preset[]; + db?: DatabaseSync; + backupUploader?: BackupUploader; + /** Re-broadcast tools/list_changed after a change (same hook the API uses). */ + onPolicyChanged: () => void; +} + +const text = (body: string): CallToolResult => ({ content: [{ type: "text", text: body }] }); +const failure = (body: string): CallToolResult => ({ isError: true, content: [{ type: "text", text: body }] }); +const json = (value: unknown): CallToolResult => text(JSON.stringify(value, null, 2)); + +/** Header/env values can be literal secrets — never echo them back. */ +function redactInjection(record: Record | undefined): Record { + return Object.fromEntries( + Object.entries(record ?? {}).map(([key, value]) => [ + key, + /^(bao:|kv:)/.test(value) ? `ref ${value}` : value.includes("${") ? `env ${value}` : "(literal, redacted)", + ]) + ); +} + +function describeSpec(spec: UpstreamSpec, source: string) { + const shared = { + id: spec.id, + namespace: spec.namespace, + transport: spec.transport, + enabled: spec.enabled, + source, + sessionMode: spec.sessionMode, + requirePersonalCredentials: spec.requirePersonalCredentials, + userDefault: spec.userDefault, + mintsOwnToken: Boolean(spec.auth), + offersUserConnect: Boolean(spec.userConnect), + }; + return spec.transport === "http" + ? { ...shared, url: spec.url, headers: redactInjection(spec.headers) } + : { ...shared, command: spec.command, args: spec.args, env: redactInjection(spec.env) }; +} + +const TIER_ENUM = ["read", "write", "destructive"] as const; + +/** The tool definitions, in the shape tools/list returns them. */ +export const SELF_TOOLS: Tool[] = [ + { + name: `${SELF_NAMESPACE}_status`, + description: + "Gateway health: version, connected upstreams with tool counts and last errors, catalog size, secret-store scheme, auth mode.", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + annotations: { readOnlyHint: true }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_list_servers`, + description: + "Configured upstream MCP servers with their mode flags. Header/env values are redacted — only keys and whether the value is a secret ref, an env ref, or a literal.", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + annotations: { readOnlyHint: true }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_list_tools`, + description: + "Tools in the catalog with their effective tier, group and enabled state. Filter by upstream, tier, group, enabled state, or a name substring.", + inputSchema: { + type: "object", + properties: { + upstreamId: { type: "string", description: "Only this upstream." }, + tier: { type: "string", enum: [...TIER_ENUM], description: "Only this effective tier." }, + group: { type: "string", description: "Only this group/category." }, + enabled: { type: "boolean", description: "Only enabled (true) or only disabled (false) tools." }, + query: { type: "string", description: "Case-insensitive substring of the tool name." }, + groupsOnly: { + type: "boolean", + description: "Return a per-group summary (counts by tier) instead of individual tools.", + }, + }, + additionalProperties: false, + }, + annotations: { readOnlyHint: true }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_set_tools_enabled`, + description: + "Enable or disable tools for EVERYONE (the admin envelope, not a personal preference). Scope with tier and/or group, a single toolName, or neither for the whole upstream.", + inputSchema: { + type: "object", + properties: { + upstreamId: { type: "string" }, + enabled: { type: "boolean" }, + tier: { type: "string", enum: [...TIER_ENUM] }, + group: { type: "string" }, + toolName: { type: "string" }, + }, + required: ["upstreamId", "enabled"], + additionalProperties: false, + }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_set_tool_tier`, + description: + "Override a tool's tier (read/write/destructive), or clear the override to fall back to the annotation-derived tier. Raising the tier is how a read-only tool that hands out secrets is kept away from low roles.", + inputSchema: { + type: "object", + properties: { + upstreamId: { type: "string" }, + toolName: { type: "string" }, + tier: { type: ["string", "null"], enum: [...TIER_ENUM, null], description: "null clears the override." }, + }, + required: ["upstreamId", "toolName"], + additionalProperties: false, + }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_set_grant`, + description: + "Set a role's ceiling for one upstream by role NAME: none / read / write / destructive. 'none' closes the upstream for that role.", + inputSchema: { + type: "object", + properties: { + roleName: { type: "string" }, + upstreamId: { type: "string" }, + maxTier: { type: "string", enum: ["none", ...TIER_ENUM] }, + }, + required: ["roleName", "upstreamId", "maxTier"], + additionalProperties: false, + }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_list_presets`, + description: "Installable server presets: id, title, description, the parameters each one needs, and its recommended grants.", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + annotations: { readOnlyHint: true }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_install_preset`, + description: + "Install (or re-install) an upstream from a preset. Secret parameters must be REFERENCES (kv:/bao:/${ENV}) — never a raw secret, which would end up stored in the spec.", + inputSchema: { + type: "object", + properties: { + presetId: { type: "string" }, + params: { type: "object", additionalProperties: { type: "string" } }, + dryRun: { type: "boolean", description: "Render and validate the spec without saving." }, + }, + required: ["presetId"], + additionalProperties: false, + }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_set_server_enabled`, + description: "Enable or disable a whole upstream server (its tools disappear from every session while disabled).", + inputSchema: { + type: "object", + properties: { upstreamId: { type: "string" }, enabled: { type: "boolean" } }, + required: ["upstreamId", "enabled"], + additionalProperties: false, + }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_remove_server`, + description: + "Remove an upstream and everything attached to it: tool settings, per-role grants and overrides. Personal credentials for it are orphaned, not deleted.", + inputSchema: { + type: "object", + properties: { upstreamId: { type: "string" }, confirm: { type: "boolean", description: "Must be true." } }, + required: ["upstreamId", "confirm"], + additionalProperties: false, + }, + annotations: { destructiveHint: true }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_refresh_catalog`, + description: "Re-read every upstream's tool list now (also happens automatically on list_changed and after reconnects).", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + _meta: { group: "gateway" }, + }, + { + name: `${SELF_NAMESPACE}_backup_now`, + description: "Take a database snapshot immediately (VACUUM INTO), prune to the retention limit, and ship it off-instance when configured.", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + _meta: { group: "gateway" }, + }, +]; + +const SELF_TOOL_NAMES = new Set(SELF_TOOLS.map((t) => t.name)); +export const isSelfTool = (name: string): boolean => SELF_TOOL_NAMES.has(name); + +type Args = Record; +const str = (args: Args, key: string): string | undefined => + typeof args[key] === "string" ? (args[key] as string) : undefined; +const bool = (args: Args, key: string): boolean | undefined => + typeof args[key] === "boolean" ? (args[key] as boolean) : undefined; + +/** + * Execute a self-management tool. Returns null when `name` isn't one of ours, + * so the caller can fall through to the federated catalog. + */ +export async function callSelfTool( + deps: SelfToolDeps, + principal: Principal, + name: string, + args: Args +): Promise { + if (!isSelfTool(name)) return null; + // The boundary: never trust that the list filter kept a non-admin away. + if (!principal.isAdmin) { + return failure( + `Tool "${name}" is not available to this session — it may not exist, be disabled, or require a higher role than "${principal.roleName}".` + ); + } + + const { config, repo, manager, presets } = deps; + const entries = () => [...manager.catalogEntries()]; + + switch (name) { + case `${SELF_NAMESPACE}_status`: + return json({ + version: config.mode === "integrated" ? "integrated" : "standalone", + toolCount: entries().length, + upstreams: manager.summaries(), + secretStore: config.bao ? "openbao" : config.keyVault ? "keyvault" : null, + auth: { + staticTokenLabels: config.staticTokens.map((t) => t.label), + oidcIssuer: config.oidc?.issuer ?? null, + interactiveLogin: Boolean(config.login), + }, + backups: { dir: config.backup.dir, keep: config.backup.keep, intervalHours: config.backup.intervalHours }, + }); + + case `${SELF_NAMESPACE}_list_servers`: + return json(repo.listUpstreams().map((row) => describeSpec(row.spec, row.source))); + + case `${SELF_NAMESPACE}_list_tools`: { + const upstreamId = str(args, "upstreamId"); + const tier = str(args, "tier") as Tier | undefined; + const group = str(args, "group"); + const wantEnabled = bool(args, "enabled"); + const query = str(args, "query")?.toLowerCase(); + const rows = entries() + .filter((e) => !upstreamId || e.upstreamId === upstreamId) + .map((e) => ({ + upstreamId: e.upstreamId, + toolName: e.upstreamToolName, + exposedName: e.exposedName, + tier: effectiveTierOf(repo, e), + group: effectiveGroupOf(repo, e), + enabled: repo.toolSetting(e.upstreamId, e.upstreamToolName)?.enabled ?? true, + })) + .filter((r) => !tier || r.tier === tier) + .filter((r) => group === undefined || r.group === group) + .filter((r) => wantEnabled === undefined || r.enabled === wantEnabled) + .filter((r) => !query || r.exposedName.toLowerCase().includes(query)); + + if (bool(args, "groupsOnly")) { + const summary = new Map(); + for (const r of rows) { + const key = `${r.upstreamId}/${r.group}`; + const s = summary.get(key) ?? { upstreamId: r.upstreamId, group: r.group || "(ungrouped)", read: 0, write: 0, destructive: 0, enabled: 0, total: 0 }; + s[r.tier] += 1; + s.total += 1; + if (r.enabled) s.enabled += 1; + summary.set(key, s); + } + return json({ groups: [...summary.values()] }); + } + return json({ + total: rows.length, + shown: Math.min(rows.length, LIST_CAP), + tools: rows.slice(0, LIST_CAP), + ...(rows.length > LIST_CAP ? { note: `${rows.length - LIST_CAP} more — narrow with upstreamId/tier/group/query, or use groupsOnly.` } : {}), + }); + } + + case `${SELF_NAMESPACE}_set_tools_enabled`: { + const upstreamId = str(args, "upstreamId")!; + const enabled = bool(args, "enabled")!; + const targets = resolveToolTargets(repo, entries(), { + upstreamId, + ...(str(args, "tier") ? { tier: str(args, "tier") as Tier } : {}), + ...(str(args, "group") !== undefined ? { group: str(args, "group")! } : {}), + ...(str(args, "toolName") ? { toolName: str(args, "toolName")! } : {}), + }); + if (targets.length === 0) { + return failure(`Nothing matched in upstream "${upstreamId}" — check the id, tier and group with ${SELF_NAMESPACE}_list_tools.`); + } + const changed = repo.bulkSetToolEnabled(upstreamId, targets.map((e) => e.upstreamToolName), enabled); + deps.onPolicyChanged(); + return text(`${enabled ? "Enabled" : "Disabled"} ${changed} tool(s) in "${upstreamId}" for every role.`); + } + + case `${SELF_NAMESPACE}_set_tool_tier`: { + const upstreamId = str(args, "upstreamId")!; + const toolName = str(args, "toolName")!; + const tier = args.tier === null ? null : (str(args, "tier") as Tier | undefined); + if (tier === undefined) return failure("Provide tier (read/write/destructive) or null to clear the override."); + if (!entries().some((e) => e.upstreamId === upstreamId && e.upstreamToolName === toolName)) { + return failure(`Tool "${toolName}" is not in upstream "${upstreamId}"'s catalog.`); + } + repo.upsertToolSetting({ upstreamId, toolName, tierOverride: tier }); + deps.onPolicyChanged(); + return text(tier ? `"${toolName}" is now tier ${tier}.` : `"${toolName}" is back to its derived tier.`); + } + + case `${SELF_NAMESPACE}_set_grant`: { + const roleName = str(args, "roleName")!; + const upstreamId = str(args, "upstreamId")!; + const maxTier = str(args, "maxTier")!; + if (!isMaxTier(maxTier)) return failure(`maxTier must be none/read/write/destructive, got "${maxTier}".`); + const role = repo.roleByName(roleName); + if (!role) { + return failure(`No role named "${roleName}" — existing roles: ${repo.listRoles().map((r) => r.name).join(", ")}.`); + } + repo.setGrant(role.id, upstreamId, maxTier); + deps.onPolicyChanged(); + return text(`Role "${roleName}" now has ceiling "${maxTier}" on "${upstreamId}".`); + } + + case `${SELF_NAMESPACE}_list_presets`: + return json( + presets.map((p) => ({ id: p.id, title: p.title, description: p.description, params: p.params, grants: p.grants })) + ); + + case `${SELF_NAMESPACE}_install_preset`: { + const presetId = str(args, "presetId")!; + const preset = presets.find((p) => p.id === presetId); + if (!preset) return failure(`No preset "${presetId}" — see ${SELF_NAMESPACE}_list_presets.`); + const params = (args.params ?? {}) as Record; + let spec: UpstreamSpec; + try { + spec = renderPreset(preset, params); + } catch (err) { + return failure(err instanceof Error ? err.message : String(err)); + } + if (bool(args, "dryRun")) return json({ dryRun: true, spec: describeSpec(spec, "preset") }); + repo.upsertUpstream(spec, "api"); + await manager.upsertUpstream(spec); + const applied: string[] = []; + const warnings: string[] = []; + for (const [roleName, maxTier] of Object.entries(preset.grants)) { + const role = repo.roleByName(roleName); + if (!role) { + warnings.push(`role "${roleName}" does not exist — grant "${maxTier}" skipped`); + continue; + } + repo.setGrant(role.id, spec.id, maxTier); + applied.push(`${roleName}=${maxTier}`); + } + deps.onPolicyChanged(); + return json({ installed: spec.id, grants: applied, warnings }); + } + + case `${SELF_NAMESPACE}_set_server_enabled`: { + const upstreamId = str(args, "upstreamId")!; + const enabled = bool(args, "enabled")!; + if (!repo.setUpstreamEnabled(upstreamId, enabled)) return failure(`No upstream "${upstreamId}".`); + await manager.upsertUpstream(repo.getUpstream(upstreamId)!.spec); + deps.onPolicyChanged(); + return text(`Upstream "${upstreamId}" is now ${enabled ? "enabled" : "disabled"}.`); + } + + case `${SELF_NAMESPACE}_remove_server`: { + const upstreamId = str(args, "upstreamId")!; + if (bool(args, "confirm") !== true) { + return failure(`Refusing to remove "${upstreamId}" without confirm: true.`); + } + const existed = repo.deleteUpstream(upstreamId); + await manager.removeUpstream(upstreamId); + deps.onPolicyChanged(); + return text( + existed + ? `Removed "${upstreamId}" with its tool settings, grants and overrides. Personal credentials for it are now orphaned.` + : `No upstream "${upstreamId}" — nothing to remove.` + ); + } + + case `${SELF_NAMESPACE}_refresh_catalog`: { + await manager.refreshCatalog(); + const perUpstream = Object.fromEntries(manager.summaries().map((s) => [s.id, s.toolCount])); + deps.onPolicyChanged(); + return json({ toolCount: entries().length, perUpstream }); + } + + case `${SELF_NAMESPACE}_backup_now`: { + if (!deps.db) return failure("No database handle available for backups in this process."); + const file = await runBackup(deps.db, config.backup, deps.backupUploader); + return json({ name: file.name, sizeBytes: file.sizeBytes, createdAt: file.createdAt, dir: config.backup.dir }); + } + + default: + return failure(`Tool "${name}" is not implemented.`); + } +} diff --git a/packages/gateway/src/upstream/personal-sessions.test.ts b/packages/gateway/src/upstream/personal-sessions.test.ts index 50c0e31..ecb83ed 100644 --- a/packages/gateway/src/upstream/personal-sessions.test.ts +++ b/packages/gateway/src/upstream/personal-sessions.test.ts @@ -70,6 +70,8 @@ const config: GatewayConfig = { publicUrl: "http://localhost:0", configPath: "unused", dbPath: ":memory:", + selfTools: true, + backup: { dir: "unused", keep: 3, intervalHours: 0 }, allowedOrigins: [], upstreamsFromFile: [], staticTokens: [ From 4de5d22598f3e0241fd76f26add1bf51ad03ad3b Mon Sep 17 00:00:00 2001 From: Eugene Samotija Date: Thu, 30 Jul 2026 03:10:33 -0400 Subject: [PATCH 3/3] docs: self-management toolset and backups in the README/CLAUDE.md Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 2 ++ README.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 6cc7734..263ed21 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,6 +12,8 @@ Self-hosted MCP manager/gateway: one streamable-HTTP `/mcp` endpoint federating - `index.ts` — CLI entry: refuses to start with no auth configured; boots DB → secrets → OIDC → upstreams → HTTP - `config.ts` — flags/env + `mspstack.config.json` (`ConfigError`); parses `MCP_TOKENS_` lists (duplicate labels are a `ConfigError` — labels are /me identities), OIDC (`OIDC_ISSUER`/`ENTRA_TENANT_ID` + required `OIDC_AUDIENCE`), `BAO_*` +- `db/backup.ts` — online snapshots (`VACUUM INTO`, never a file copy of a live db), retention, optional Azure Blob shipping (lazy SDK, DefaultAzureCredential); scheduler unref'd, off for `:memory:`; `GET/POST /api/backups`; restore is manual (docs/backups.md) +- `mcp/self-tools.ts` — admin-only `gw_*` toolset (issue #2): status, list servers/tools (values REDACTED), bulk enable by tier/group, tier overrides, grants by role name, preset install, refresh, backup-now. Hidden from non-admins AND re-checked per call with the unknown-tool wording; `gw` namespace reserved in config.ts; `GATEWAY_SELF_TOOLS=off` disables. Target resolution shared with both HTTP APIs via `domain/tool-targets.ts` - `db/` — `node:sqlite` schema (roles/upstreams/grants/tool_overrides/tool_settings/users/group_mappings, seeded viewer/editor/admin) + typed `Repo` - `domain/catalog.ts` — namespacing (`${namespace}_${tool}`, no double-prefix), routing map (no string-splitting), annotation-derived tiers (port of mcp-itglue `tierOf`), `derivedGroupOf(tool)` — the category for the group switches: `_meta.group`/`_meta.toolset` first (how the family servers tag their toolsets), else a bracketed description prefix (`[Identity > …]`, CIPP); an explicit `group_label` always wins - `domain/presets.ts` — one-click upstream presets: builtin family configs (itglue/cwpsa/planner/cipp — full specs incl. BYOK headers, per-user mode, userConnect, personalCredentials, the `auth` mint block) + optional `mspstack.presets.json` (file overrides builtin ids); `{{param}}` templating rendered server-side and validated via `parseUpstreamSpec`; recommended grants by role NAME resolved at install (`GET /api/presets`, `POST /api/presets/:id/install` with `dryRun`). Spec's `personalCredentials` metadata drives the /me guided credential forms (`credentialFields` in `/api/me/access`) diff --git a/README.md b/README.md index 755aaad..2a25a38 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,8 @@ Point Claude (Code, Desktop, or any MCP client) at a single URL; the gateway con - **Install from the UI** — one-click **presets** for the MSPStack family (IT Glue, ConnectWise PSA, Planner) and CIPP that fill BYOK headers, per-user session mode, Connect wiring, and apply recommended role grants (extend with your own via `mspstack.presets.json`); or add any MCP server by URL, npm package (npx), or Docker image; search the official MCP registry; preflight-test before saving; crashed stdio servers restart with backoff - **Guided user setup** — upstreams declare their personal-credential fields, so `/me` renders labeled forms (not raw header names), plus ready-to-copy connect snippets: Claude Code CLI (user-scope by default) and JSON config for Desktop/Cursor/VS Code - **Manageable at scale** — the admin catalog nests tools under server → category, and every level has one-click switches for the whole set or just its read / write / destructive tools; the same switches appear on `/me`. Categories come from the server itself (`_meta.group` on the tool, or a `[Category > …]` description prefix) and can be relabelled per tool. Long lists page in on demand, tool lists can be re-read from the servers at any time, and any upstream can be set to opt-in (`userDefault: "off"`, also a control in the add form) so users pick what they need instead of receiving hundreds of tools +- **Administer it conversationally** — an admin-only `gw_*` MCP toolset (status, tool/tier/group switches, grants, preset installs, backups) so routine changes happen in your MCP client; credential values are never read back ([docs](docs/self-management.md)) +- **Backed up** — periodic online SQLite snapshots with retention and optional off-instance copies to Azure Blob ([docs](docs/backups.md)) - **Admin UI** at `/admin` — status, server management, tool toggles, role matrix, users & group mappings (with live Entra group search when the login app holds the directory-read Graph roles), OAuth client management, secret writes ## Quick start