diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 820bf86..b61298f 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,25 +1,85 @@ # Architecture -`@quantum-l9/llm-router` is a reusable TypeScript routing library. The root `L9LLMRouter` is the supported execution surface. +`@quantum-l9/llm-router` is a reusable TypeScript routing library. `L9LLMRouter` is the supported production execution surface and the only component that composes routing, budget, resilience, and provider dispatch. ## Runtime flow ```text -validated TaskDescriptor +validated execution task + -> effective image set merged into task -> pure route resolution + -> request identity and timestamp -> atomic process-local budget reservation -> provider-family-safe downgrade -> per-provider circuit permit - -> provider dispatch - -> cost reconciliation and audit log + -> provider dispatch with explicit cancellation and timeout + -> aggregate execution accounting + -> budget reconciliation and audit log ``` -Provider clients live under `src/providers/`. Production modules outside `src/index.ts` and `src/providers/` may not import them. Existing provider subpath exports remain available during the 1.x line for compatibility, but direct use is deprecated because it bypasses budget and circuit controls. +Route resolution is pure. Request IDs and timestamps are added afterward and do not participate in routing equivalence. -## State scope +## Module ownership -The built-in budget tracker and circuit breaker are process-local. They prevent concurrent overspend and provider stampedes inside one process. Distributed enforcement requires an external persistence adapter and is intentionally not claimed here. +```text +src/types.ts public legacy contracts +src/schemas.ts runtime validation for public legacy input +src/matrices/* deterministic model and search resolution +src/pricing.ts canonical OpenRouter price table +src/budget/* process-local admission and spend accounting +src/circuit-breaker.ts process-local provider health control +src/provider-errors.ts typed failure classification and redaction +src/providers/* provider I/O and SDK transport isolation +src/vision/* vision configuration and task planning +src/index.ts composition root and supported execution API +src/control-plane/* internal Phase 1 contract kernel +``` + +## Provider boundary + +Provider clients live under `src/providers/`. Production modules outside `src/index.ts` and `src/providers/` may not import them. ESLint and a programmatic probe enforce the rule. + +Existing provider subpath exports remain available during the 1.x line for compatibility. They are deprecated because direct use bypasses budget and circuit controls. Their removal requires a major version. + +## Budget state + +The built-in budget tracker owns committed spend and active reservations. Admission evaluates both, so concurrent requests inside one JavaScript process cannot all pass against the same unreserved ceiling. + +Critical tasks retain the documented override but still reserve and record cost. Reservation identifiers must be non-empty and unique. Client and direct tracker configuration is runtime validated. + +The tracker is not a distributed ledger. Multiple processes require an external atomic persistence adapter, which is outside this repository's current scope. + +## Circuit state + +The circuit breaker owns independent state per provider. + +- Closed calls receive ordinary permits. +- The failure threshold opens the circuit. +- The cooldown permits exactly one half-open probe. +- Retryable network, timeout, rate-limit, and server failures count. +- Client, cancellation, budget, policy, and local validation failures do not count. +- Late results from calls acquired before a circuit opened cannot overwrite newer state. + +## Provider execution + +The OpenAI SDK is isolated behind `OpenAIChatTransport`. SDK retries are disabled. The router controls fallback order explicitly. + +OpenRouter fallbacks advance only after retryable failures. Non-retryable client failures and cancellation terminate immediately. + +Perplexity consensus executes configured variations in parallel. The selected content remains one successful candidate, while budget-facing cost and token accounting aggregate all successful variations. + +## Image safety + +Image execution accepts public HTTPS URLs and bounded supported image data URIs. Local, private, loopback, link-local, reserved, and local-domain targets are rejected before dispatch. Option-supplied images are validated and included in route selection before budget reservation. + +## Control Plane Phase 1 + +The Control Plane kernel owns strict runtime contracts, canonical JSON, deterministic identity, immutable builders, policy interfaces, and provider-adapter interfaces. It does not own provider clients, network access, Gate ingress, TransportPacket authority, Graphiti, Neo4j, promotion, mutable global state, or legacy cutover. + +The internal barrel `src/control-plane/index.ts` is a deliberate module boundary but is absent from package exports. The one-line pass-through type module was removed because it had no independent responsibility. + +Route identity excludes request-specific values and explanatory prose, including route, budget, and provider-health reasons. Complete content hashing still protects those fields. -## Runtime floor +## Runtime support -The package runtime floor is Node 20.19.0. Type definitions track the Node 20 line so compilation cannot silently authorize APIs that are unavailable to supported consumers. CI also exercises the current active LTS as a secondary runtime. +The package preserves a Node 20.19.0 compatibility floor for the 1.x line. CI validates the floor plus maintained Node 22 and Node 24 LTS lines. Publish and supply-chain jobs run on Node 24 LTS. diff --git a/README.md b/README.md index 062a567..74f2898 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,145 @@ # @quantum-l9/llm-router -Shared deterministic model routing, budget enforcement, provider resilience, and visual QA for L9 bots. +`@quantum-l9/llm-router` is the shared TypeScript routing library for L9 applications. It validates task input, selects a provider and model deterministically, reserves budget before dispatch, applies provider-family-safe downgrades, controls provider failure pressure, executes through typed provider clients, and reconciles actual cost. + +## What is implemented + +- Deterministic routing for search, general, and vision task families +- Per-client and global process-local budget enforcement with pre-dispatch reservations +- Per-provider circuit breaking with one half-open recovery probe +- Explicit timeout, cancellation, retry, fallback, and provider-error classification +- OpenRouter and Perplexity clients through an OpenAI SDK transport boundary +- Bounded image URL and inline-image validation +- Runtime validation with Zod 4 +- Internal Control Plane Phase 1 contracts, canonical hashing, builders, and boundaries +- Package, declaration, lint-boundary, audit, and isolated-consumer validation + +## Installation + +The package is published to GitHub Packages. + +```bash +npm install @quantum-l9/llm-router +``` + +Configure the `@quantum-l9` registry and authentication through the consuming environment. Do not commit package tokens or provider credentials. + +## Supported runtimes + +The 1.x compatibility floor remains Node.js `20.19.0`. CI also validates maintained Node.js 22 and 24 LTS lines. Release and supply-chain jobs run on Node.js 24 LTS. + +## Basic use ```ts -import { L9LLMRouter, TaskComplexity, TaskType } from '@quantum-l9/llm-router'; +import { + L9LLMRouter, + TaskComplexity, + TaskType, +} from '@quantum-l9/llm-router'; const router = new L9LLMRouter({ perplexityApiKey: process.env.PERPLEXITY_API_KEY!, openrouterApiKey: process.env.OPENROUTER_API_KEY!, + providerTimeoutMs: 60_000, + providerMaxRetries: 0, +}); + +router.initClient('tenant-a', { + monthlyBudgetPerClient: 200, + weeklyTarget: 50, + weeklyHardCeiling: 100, }); -router.initClient('tenant-a', { monthlyBudgetPerClient: 200 }); + const result = await router.execute( - { clientId: 'tenant-a', type: TaskType.CONTENT_GENERATION, complexity: TaskComplexity.MEDIUM }, + { + clientId: 'tenant-a', + type: TaskType.CONTENT_GENERATION, + complexity: TaskComplexity.MEDIUM, + expectedOutputTokens: 1_500, + }, 'You are a careful writer.', 'Draft the article.', ); ``` -Direct imports from `./openrouter` or `./perplexity` are retained for 1.x compatibility but are deprecated. They bypass router-level budget and circuit controls. +`execute()` requires a non-empty `clientId`. The router rejects malformed execution input before allocating a request ID, reserving budget, or dispatching a provider call. + +## Vision execution + +Images supplied through execution options are merged into the validated task before routing. This ensures model selection and budget estimation use the same image count that reaches the provider. + +```ts +const result = await router.execute( + { + clientId: 'tenant-a', + type: TaskType.SCREENSHOT_ANALYSIS, + complexity: TaskComplexity.MEDIUM, + }, + 'Inspect the screenshots.', + 'Compare the layouts.', + { + images: [ + 'https://cdn.example.com/current.png', + 'https://cdn.example.com/competitor.png', + ], + }, +); +``` + +Only HTTPS public URLs and bounded `data:image/*;base64` payloads are accepted. Private, loopback, link-local, reserved, local-domain, non-image, and oversized inline targets are rejected before provider dispatch. + +## Search consensus + +For eligible high-complexity Perplexity tasks, `{ consensus: true }` executes the configured variations in parallel. The returned content is selected from the successful responses, while token and cost fields represent the aggregate successful consensus execution so budget reconciliation does not undercount spend. + +## Budget semantics + +The built-in tracker is process-local. + +1. Estimate the route cost. +2. Evaluate committed spend plus active reservations. +3. Reserve estimated cost before provider dispatch. +4. Release the reservation on confirmed unbilled failure. +5. Reconcile the reservation to actual reported cost on success. + +This prevents concurrent overspend inside one process. It does not claim distributed enforcement across processes or machines. + +## Provider failure behavior + +Provider failures are classified as network, timeout, rate limit, server, client, cancellation, local, or unknown. + +- Retryable provider failures may advance through an explicit fallback chain. +- Client errors, cancellation, and local validation failures stop immediately. +- Local validation and budget failures do not poison provider circuit health. +- A circuit permits only one half-open recovery probe. +- Late successes from older calls cannot close a circuit opened by newer failures. + +The OpenAI SDK has hidden retries disabled. Every router-controlled fallback remains visible and bounded. + +## Direct provider imports + +These 1.x compatibility exports remain available: + +```ts +import { OpenRouterClient } from '@quantum-l9/llm-router/openrouter'; +import { PerplexityClient } from '@quantum-l9/llm-router/perplexity'; +``` + +They are deprecated because they bypass router-level budget and circuit controls. Use `L9LLMRouter` for production execution. Removal requires a future major-version migration. + +## Internal Control Plane kernel + +Phase 1 Control Plane files are compiled but are not exposed through `package.json` exports. They define strict contracts, deterministic canonicalization, identity verification, immutable builders, policy interfaces, and provider-adapter interfaces. They perform no network calls and do not replace the legacy router. + +See [`docs/control-plane-architecture.md`](docs/control-plane-architecture.md). + +## Validation + +```bash +npm ci +npm run verify:all +``` + +`verify:all` runs the production build, strict type verification, declaration-consumer compilation, ESLint, the provider-boundary probe, Vitest, production dependency audit, package allowlist inspection, isolated tarball installation, and package export smoke tests. + +Operational procedures and failure recovery are documented in [`RUNBOOK.md`](RUNBOOK.md). Architecture and ownership boundaries are documented in [`ARCHITECTURE.md`](ARCHITECTURE.md). diff --git a/RUNBOOK.md b/RUNBOOK.md new file mode 100644 index 0000000..02b1a2a --- /dev/null +++ b/RUNBOOK.md @@ -0,0 +1,155 @@ +# LLM Router Runbook + +## Purpose + +This runbook covers local setup, deterministic validation, package inspection, common failure diagnosis, release preparation, and rollback for `@quantum-l9/llm-router`. + +## Prerequisites + +- Node.js 20.19.0 or newer +- npm 10.9.2 or a compatible npm 10 release +- Access to the package registry when installing dependencies +- No provider credentials are required for tests + +The 1.x package retains Node 20 compatibility, but release and supply-chain workflows use Node 24 LTS. + +## Clean setup + +```bash +rm -rf node_modules dist +npm ci --ignore-scripts +``` + +Use `npm ci`, not `npm install`, when validating a proposed commit. `npm ci` proves the committed lockfile is reproducible and refuses manifest drift. + +## Full validation + +```bash +npm run verify:all +``` + +Expected stages: + +1. TypeScript build +2. Strict no-emit typecheck +3. Declaration-consumer compilation +4. ESLint +5. Provider-boundary probe +6. Complete Vitest suite +7. Production dependency audit +8. Package allowlist and isolated-consumer smoke test + +A failure in any stage blocks merge or release. + +## Focused commands + +```bash +npm run build +npm run verify:types +npm run verify:declarations +npm run lint +npm run lint:boundary +npm test +npm run test:inventory +npm audit --audit-level=high --omit=dev +npm run verify:package +``` + +## Package inspection + +```bash +npm pack --json --ignore-scripts +``` + +The tarball may contain only: + +- `package.json` +- `README.md` +- `ARCHITECTURE.md` +- `RUNBOOK.md` +- compiled `dist/` files + +`npm run verify:package` enforces this allowlist, installs the tarball into an isolated temporary consumer, and checks the root plus supported subpath exports. + +## Provider-boundary failure + +Symptom: `npm run lint:boundary` or `tests/eslint-boundary.test.ts` fails. + +Recovery: + +1. Find the production file importing `src/providers/*`. +2. Route execution through `L9LLMRouter` or move provider I/O into the approved provider boundary. +3. Do not disable the lint rule or add a broad ignore. +4. Rerun `npm run lint:boundary` and the full suite. + +## Budget-reservation failure + +Symptom: an unknown, duplicate, or already-settled reservation error. + +Recovery: + +1. Confirm every successful request reconciles exactly once. +2. Confirm every unbilled failure releases exactly once. +3. Confirm the reservation ID factory cannot return duplicates. +4. Inspect `getClientBudgetReport()` for active reservations. +5. Do not mutate internal maps or compensate by recording spend manually. + +The tracker is process-local. Cross-process consistency requires an external persistence design and is not implemented in this package. + +## Circuit-breaker failure + +Symptom: provider calls remain blocked after the cooldown or too many recovery calls escape. + +Recovery: + +1. Inspect `getCircuitState(provider)`. +2. Confirm every acquired permit is completed with `recordSuccess`, `recordFailure`, or `release`. +3. Confirm only retryable network, timeout, rate-limit, and server failures are counted. +4. Confirm only one half-open probe is active. +5. Do not close an open circuit based on a stale pre-open success. + +## Provider fallback failure + +Fallbacks advance only after retryable provider failures. A 4xx client error, local validation failure, or cancellation must stop immediately. + +Check: + +- provider error `kind` +- `retryable` +- provider request ID +- retry-after metadata +- abort signal state + +The SDK retry count must remain zero so attempts stay explicit in router behavior. + +## Control Plane contract failure + +Control Plane builders reject unknown fields, invalid combinations, non-finite values, cycles, accessors, sparse arrays, unsupported objects, and hash mismatches. + +Recovery: + +1. Validate against the exported Zod schema. +2. Pass plain JSON-compatible values. +3. Normalize unordered sets through the builder rather than precomputing hashes. +4. Rebuild the artifact instead of editing identity fields manually. +5. Run `tests/control-plane/*`. + +## Release preparation + +1. Rebase onto the intended release base. +2. Run a clean `npm ci --ignore-scripts`. +3. Run `npm run verify:all`. +4. Inspect `npm pack --json --ignore-scripts`. +5. Confirm the version is publishable and has not already been released. +6. Confirm the tag matches the package version. +7. Push the tag only after all release evidence is captured. + +The GitHub publish workflow validates again before `npm publish`. + +## Rollback + +Stop the merge or release train on the first regression. + +For an unmerged branch, revert or amend only the failing change and rerun all gates. For a merged regression, revert the most recent merge, validate `main`, and reopen the affected work. Do not stack compensating changes on a red baseline. + +For an already published package, npm packages are immutable. Publish a corrected patch version and deprecate the defective version with a clear migration note. diff --git a/docs/control-plane-architecture.md b/docs/control-plane-architecture.md new file mode 100644 index 0000000..645077f --- /dev/null +++ b/docs/control-plane-architecture.md @@ -0,0 +1,49 @@ +# L9 LLM Control Plane: Phase 1 Kernel + +## Status + +Phase 1 is an internal, additive contract kernel. It is compiled but not exported from `package.json`, performs no provider calls, and does not replace `L9LLMRouter`. + +## Eight-phase trajectory + +1. Contract kernel +2. Provider adapters +3. Canonical route matrix and loader +4. Evidence and signals +5. Fitness engine +6. Promotion workflow +7. Frontier lab +8. Measured legacy cutover + +Only Phase 1 is present. Promotion and cutover remain blocked. + +## Identity model + +- `route_fingerprint` identifies equivalent routing decisions. It excludes request IDs and explanatory route, budget, and provider-health prose. +- `plan_id` identifies one request-specific plan from the request ID and route fingerprint. +- `content_hash` protects the complete immutable artifact, including explanatory fields. + +Canonical hashing normalizes Unicode to NFC, sorts object keys ordinally, normalizes negative zero, and rejects non-finite numbers, undefined values, cycles, sparse arrays, exotic objects, symbol keys, accessors, non-enumerable properties, and normalization-colliding keys. + +Builders run the same safe canonicalization before schema validation. Invalid runtime objects therefore fail with a controlled `ControlPlaneValidationError` rather than executing accessors or overflowing on cycles. + +## Contract behavior + +Every persistent contract has: + +- an explicit schema version +- a strict Zod 4 schema +- derived TypeScript types +- deterministic builder functions +- content or identity hashes +- verification functions +- contradiction checks +- immutable returned values + +Set-like arrays are normalized, sorted, and deduplicated before hashing. Unknown fields are rejected. + +## Boundaries + +The kernel owns contracts, validation, deterministic builders, canonical identity, policy interfaces, and provider-adapter interfaces. It does not own Gate ingress, TransportPacket authority, Graphiti or Neo4j writes, provider SDKs, live network calls, tenant-specific behavior, promotion, or mutable process state. + +The canonical internal entry point is `src/control-plane/index.ts`. It exists to define the Phase 1 module boundary and is intentionally absent from public package exports. diff --git a/fixtures/control-plane/canonical-golden-v1.json b/fixtures/control-plane/canonical-golden-v1.json new file mode 100644 index 0000000..f6770bf --- /dev/null +++ b/fixtures/control-plane/canonical-golden-v1.json @@ -0,0 +1,10 @@ +{ + "input": { + "z": 1, + "a": "é", + "nested": {"beta": true, "alpha": null}, + "array": [3, 2, 1] + }, + "canonical": "{\"a\":\"é\",\"array\":[3,2,1],\"nested\":{\"alpha\":null,\"beta\":true},\"z\":1}", + "sha256": "ad73f9077cce273c06520be01118fe9c2124a22eb54ab046eb469d00d6425c83" +} diff --git a/package.json b/package.json index 8ce774e..938bbb7 100644 --- a/package.json +++ b/package.json @@ -69,7 +69,8 @@ "files": [ "dist", "README.md", - "ARCHITECTURE.md" + "ARCHITECTURE.md", + "RUNBOOK.md" ], "packageManager": "npm@10.9.2" } diff --git a/scripts/verify-package.mjs b/scripts/verify-package.mjs index 6b989f0..dbdeff2 100644 --- a/scripts/verify-package.mjs +++ b/scripts/verify-package.mjs @@ -27,7 +27,7 @@ try { const artifact = packed[0]; tarball = join(root, artifact.filename); const unexpected = artifact.files.map(entry => entry.path).filter(path => !( - path === 'package.json' || path === 'README.md' || path === 'ARCHITECTURE.md' || path.startsWith('dist/') + path === 'package.json' || path === 'README.md' || path === 'ARCHITECTURE.md' || path === 'RUNBOOK.md' || path.startsWith('dist/') )); if (unexpected.length > 0) throw new Error(`Unexpected package files: ${unexpected.join(', ')}`); diff --git a/src/control-plane/builders.ts b/src/control-plane/builders.ts new file mode 100644 index 0000000..672a0b2 --- /dev/null +++ b/src/control-plane/builders.ts @@ -0,0 +1,164 @@ +import type { ZodError, ZodType } from 'zod'; +import { canonicalize, CanonicalJsonError, canonicalJson, sha256Hex } from './canonical-json.js'; +import { + LLM_EXECUTION_RECORD_SCHEMA_VERSION, + LLM_FEEDBACK_SIGNAL_SCHEMA_VERSION, + LLM_ROUTE_PLAN_SCHEMA_VERSION, + TASK_PROFILE_SCHEMA_VERSION, + LLMExecutionRecordSchema, + LLMFeedbackSignalSchema, + LLMRoutePlanSchema, + TaskProfileSchema, + type LLMExecutionRecord, + type LLMExecutionRecordInput, + type LLMFeedbackSignal, + type LLMFeedbackSignalInput, + type LLMRoutePlan, + type LLMRoutePlanInput, + type TaskProfile, + type TaskProfileInput, +} from './contracts.js'; + +export class ControlPlaneValidationError extends Error { + constructor(public readonly artifact: string, public readonly issues: Array<{ path: string; message: string; code: string }>) { + super(`Invalid ${artifact}: ${issues.map(issue => `${issue.path || '(root)'}: ${issue.message}`).join('; ')}`); + this.name = 'ControlPlaneValidationError'; + } + toJSON(): Record { return { name: this.name, message: this.message, artifact: this.artifact, issues: this.issues }; } +} + +function issuesOf(error: ZodError): Array<{ path: string; message: string; code: string }> { + return error.issues.map(issue => ({ path: issue.path.map(String).join('.'), message: issue.message, code: issue.code })); +} + +function parse(schema: ZodType, value: unknown, artifact: string): T { + const result = schema.safeParse(value); + if (!result.success) throw new ControlPlaneValidationError(artifact, issuesOf(result.error)); + return result.data; +} + +function normalizeInput(value: T, artifact: string): T { + try { + return canonicalize(value) as T; + } catch (error) { + if (error instanceof CanonicalJsonError) { + throw new ControlPlaneValidationError(artifact, [{ path: '(root)', message: error.message, code: 'canonical_json' }]); + } + throw error; + } +} + +function normalizeString(value: string): string { return value.normalize('NFC'); } +function sortedUniqueStrings(values: string[]): string[] { return [...new Set(values.map(normalizeString))].sort((left, right) => left < right ? -1 : left > right ? 1 : 0); } +function sortedUniqueTargets(values: T[]): T[] { + const normalized = values.map(value => ({ ...value, provider: normalizeString(value.provider), model: normalizeString(value.model) })); + return [...new Map(normalized.map(value => [`${value.provider}\u0000${value.model}`, value])).values()].sort((left, right) => left.provider === right.provider ? (left.model < right.model ? -1 : left.model > right.model ? 1 : 0) : (left.provider < right.provider ? -1 : 1)) as T[]; +} + +function deepFreeze(value: T): T { + if (value && typeof value === 'object' && !Object.isFrozen(value)) { + Object.freeze(value); + for (const entry of Object.values(value as Record)) deepFreeze(entry); + } + return value; +} + +export function buildTaskProfile(input: TaskProfileInput): TaskProfile { + const normalized = normalizeInput(input, 'TaskProfile'); + const withoutHash = { schema_version: TASK_PROFILE_SCHEMA_VERSION, ...normalized }; + return deepFreeze(parse(TaskProfileSchema, { ...withoutHash, task_profile_hash: sha256Hex(withoutHash) }, 'TaskProfile')); +} + +export function verifyTaskProfile(profile: unknown): TaskProfile { + const parsed = parse(TaskProfileSchema, profile, 'TaskProfile'); + const { task_profile_hash, ...withoutHash } = parsed; + if (sha256Hex(withoutHash) !== task_profile_hash) throw new ControlPlaneValidationError('TaskProfile', [{ path: 'task_profile_hash', message: 'hash mismatch', code: 'custom' }]); + return deepFreeze(parsed); +} + +function routeFingerprintPayload(input: LLMRoutePlanInput): Record { + return { + task_profile_hash: input.task_profile_hash, + selected: input.selected, + search: input.search, + fallback_chain: input.fallback_chain, + policy_decision: input.policy_decision, + budget_decision: { status: input.budget_decision.status }, + provider_health_decision: { status: input.provider_health_decision.status }, + learned_candidate_refs: input.learned_candidate_refs, + dispatch_allowed: input.dispatch_allowed, + }; +} + +export function buildRoutePlan(input: LLMRoutePlanInput): LLMRoutePlan { + const normalized = normalizeInput({ + ...input, + fallback_chain: sortedUniqueTargets(input.fallback_chain), + policy_decision: { ...input.policy_decision, applied_rules: sortedUniqueStrings(input.policy_decision.applied_rules), blockers: sortedUniqueStrings(input.policy_decision.blockers) }, + learned_candidate_refs: sortedUniqueStrings(input.learned_candidate_refs), + }, 'LLMRoutePlan'); + const route_fingerprint = `route_${sha256Hex(routeFingerprintPayload(normalized)).slice(0, 32)}`; + const plan_id = `plan_${sha256Hex({ request_id: normalized.request_id, route_fingerprint }).slice(0, 32)}`; + const withoutHash = { schema_version: LLM_ROUTE_PLAN_SCHEMA_VERSION, route_fingerprint, plan_id, ...normalized }; + return deepFreeze(parse(LLMRoutePlanSchema, { ...withoutHash, content_hash: sha256Hex(withoutHash) }, 'LLMRoutePlan')); +} + +export function verifyRoutePlan(plan: unknown): LLMRoutePlan { + const parsed = parse(LLMRoutePlanSchema, plan, 'LLMRoutePlan'); + const { content_hash, route_fingerprint, plan_id } = parsed; + const input: LLMRoutePlanInput = { + request_id: parsed.request_id, + task_profile_hash: parsed.task_profile_hash, + selected: parsed.selected, + search: parsed.search, + fallback_chain: parsed.fallback_chain, + policy_decision: parsed.policy_decision, + budget_decision: parsed.budget_decision, + provider_health_decision: parsed.provider_health_decision, + learned_candidate_refs: parsed.learned_candidate_refs, + route_reason: parsed.route_reason, + dispatch_allowed: parsed.dispatch_allowed, + }; + const expectedFingerprint = `route_${sha256Hex(routeFingerprintPayload(input)).slice(0, 32)}`; + const expectedPlanId = `plan_${sha256Hex({ request_id: parsed.request_id, route_fingerprint: expectedFingerprint }).slice(0, 32)}`; + const withoutHash = { + schema_version: parsed.schema_version, + route_fingerprint: parsed.route_fingerprint, + plan_id: parsed.plan_id, + ...input, + }; + const issues: Array<{ path: string; message: string; code: string }> = []; + if (route_fingerprint !== expectedFingerprint) issues.push({ path: 'route_fingerprint', message: 'fingerprint mismatch', code: 'custom' }); + if (plan_id !== expectedPlanId) issues.push({ path: 'plan_id', message: 'plan identity mismatch', code: 'custom' }); + if (content_hash !== sha256Hex(withoutHash)) issues.push({ path: 'content_hash', message: 'hash mismatch', code: 'custom' }); + if (issues.length > 0) throw new ControlPlaneValidationError('LLMRoutePlan', issues); + return deepFreeze(parsed); +} + +export function buildExecutionRecord(input: LLMExecutionRecordInput): LLMExecutionRecord { + const normalized = normalizeInput(input, 'LLMExecutionRecord'); + const withoutHash = { schema_version: LLM_EXECUTION_RECORD_SCHEMA_VERSION, ...normalized }; + return deepFreeze(parse(LLMExecutionRecordSchema, { ...withoutHash, content_hash: sha256Hex(withoutHash) }, 'LLMExecutionRecord')); +} + +export function verifyExecutionRecord(record: unknown): LLMExecutionRecord { + const parsed = parse(LLMExecutionRecordSchema, record, 'LLMExecutionRecord'); + const { content_hash, ...withoutHash } = parsed; + if (content_hash !== sha256Hex(withoutHash)) throw new ControlPlaneValidationError('LLMExecutionRecord', [{ path: 'content_hash', message: 'hash mismatch', code: 'custom' }]); + return deepFreeze(parsed); +} + +export function buildFeedbackSignal(input: LLMFeedbackSignalInput): LLMFeedbackSignal { + const normalized = normalizeInput({ ...input, evidence_refs: sortedUniqueStrings(input.evidence_refs) }, 'LLMFeedbackSignal'); + const withoutHash = { schema_version: LLM_FEEDBACK_SIGNAL_SCHEMA_VERSION, signal_family: 'llm_routing' as const, ...normalized }; + return deepFreeze(parse(LLMFeedbackSignalSchema, { ...withoutHash, content_hash: sha256Hex(withoutHash) }, 'LLMFeedbackSignal')); +} + +export function verifyFeedbackSignal(signal: unknown): LLMFeedbackSignal { + const parsed = parse(LLMFeedbackSignalSchema, signal, 'LLMFeedbackSignal'); + const { content_hash, ...withoutHash } = parsed; + if (content_hash !== sha256Hex(withoutHash)) throw new ControlPlaneValidationError('LLMFeedbackSignal', [{ path: 'content_hash', message: 'hash mismatch', code: 'custom' }]); + return deepFreeze(parsed); +} + +export function canonicalBytes(value: unknown): Uint8Array { return new TextEncoder().encode(canonicalJson(value)); } diff --git a/src/control-plane/canonical-json.ts b/src/control-plane/canonical-json.ts new file mode 100644 index 0000000..7962064 --- /dev/null +++ b/src/control-plane/canonical-json.ts @@ -0,0 +1,53 @@ +import { createHash } from 'node:crypto'; + +export class CanonicalJsonError extends TypeError { + constructor(message: string) { super(message); this.name = 'CanonicalJsonError'; } +} + +function isPlainObject(value: object): value is Record { + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +function normalize(value: unknown, stack: Set, path: string): unknown { + if (value === null || typeof value === 'boolean' || typeof value === 'string') { + return typeof value === 'string' ? value.normalize('NFC') : value; + } + if (typeof value === 'number') { + if (!Number.isFinite(value)) throw new CanonicalJsonError(`${path}: non-finite numbers are forbidden`); + return Object.is(value, -0) ? 0 : value; + } + if (['undefined', 'bigint', 'symbol', 'function'].includes(typeof value)) throw new CanonicalJsonError(`${path}: unsupported ${typeof value} value`); + if (typeof value !== 'object') throw new CanonicalJsonError(`${path}: unsupported value`); + if (stack.has(value)) throw new CanonicalJsonError(`${path}: cyclic value`); + stack.add(value); + try { + if (Array.isArray(value)) { + for (let index = 0; index < value.length; index += 1) if (!(index in value)) throw new CanonicalJsonError(`${path}[${index}]: sparse arrays are forbidden`); + return value.map((entry, index) => normalize(entry, stack, `${path}[${index}]`)); + } + if (!isPlainObject(value)) throw new CanonicalJsonError(`${path}: only plain objects are canonicalizable`); + if (Object.getOwnPropertySymbols(value).length > 0) throw new CanonicalJsonError(`${path}: symbol properties are forbidden`); + const descriptors = Object.getOwnPropertyDescriptors(value); + const normalizedEntries = Object.keys(descriptors).map(key => { + const descriptor = descriptors[key]; + if (!descriptor.enumerable || descriptor.get || descriptor.set) throw new CanonicalJsonError(`${path}.${key}: accessors and non-enumerable properties are forbidden`); + return { originalKey: key, key: key.normalize('NFC'), value: descriptor.value }; + }); + const seen = new Set(); + for (const entry of normalizedEntries) { + if (seen.has(entry.key)) throw new CanonicalJsonError(`${path}: Unicode-normalized duplicate key "${entry.key}"`); + seen.add(entry.key); + } + normalizedEntries.sort((left, right) => left.key < right.key ? -1 : left.key > right.key ? 1 : 0); + const result: Record = {}; + for (const entry of normalizedEntries) result[entry.key] = normalize(entry.value, stack, `${path}.${entry.originalKey}`); + return result; + } finally { + stack.delete(value); + } +} + +export function canonicalize(value: unknown): unknown { return normalize(value, new Set(), '$'); } +export function canonicalJson(value: unknown): string { return JSON.stringify(canonicalize(value)); } +export function sha256Hex(value: unknown): string { return createHash('sha256').update(canonicalJson(value), 'utf8').digest('hex'); } diff --git a/src/control-plane/contracts.ts b/src/control-plane/contracts.ts new file mode 100644 index 0000000..fe22c3c --- /dev/null +++ b/src/control-plane/contracts.ts @@ -0,0 +1,184 @@ +import { z } from 'zod'; + +const trimmed = (label: string, max = 512) => z.string().min(1, `${label} is required`).max(max).refine(value => value === value.trim(), `${label} must not contain surrounding whitespace`); +const sha256 = z.string().regex(/^[0-9a-f]{64}$/, 'expected lowercase SHA-256 hex'); +const routeIdentity = z.string().regex(/^route_[0-9a-f]{32}$/); +const planIdentity = z.string().regex(/^plan_[0-9a-f]{32}$/); +const sortedUnique = (item: T) => z.array(item).superRefine((values, context) => { + const serialized = values.map(value => typeof value === 'string' ? value : JSON.stringify(value)); + const expected = [...new Set(serialized)].sort((left, right) => left < right ? -1 : left > right ? 1 : 0); + if (serialized.length !== expected.length || serialized.some((value, index) => value !== expected[index])) context.addIssue({ code: 'custom', message: 'array must be sorted and unique' }); +}); + +export const ProviderIdSchema = z.enum(['openai', 'anthropic', 'mistral', 'gemini', 'deepseek', 'perplexity', 'openrouter']); +export const TaskFamilySchema = z.enum(['classification', 'extraction', 'scoring', 'proposal_generation', 'friction_analysis', 'architecture_review', 'contract_generation', 'contract_hardening', 'evidence_synthesis', 'signal_generation', 'adr_generation', 'risk_assessment', 'validation_interpretation', 'memory_episode_summarization', 'graph_fact_extraction', 'promotion_recommendation', 'fact_verification', 'deep_research', 'code_generation', 'vision_analysis']); +export const TaskComplexityLevelSchema = z.enum(['trivial', 'low', 'medium', 'high', 'critical']); +export const DataSensitivitySchema = z.enum(['public', 'internal', 'confidential', 'restricted']); +export const FreshnessRequirementSchema = z.enum(['none', 'hour', 'day', 'week', 'month', 'year']); +export const ModalitySchema = z.enum(['text', 'vision', 'multimodal']); +export const LatencyClassSchema = z.enum(['realtime', 'fast', 'normal', 'slow']); +export const ValidationProfileSchema = z.enum(['strict_json', 'cited_answer', 'freeform', 'code', 'visual']); +export const ReasoningDepthSchema = z.enum(['none', 'low', 'medium', 'high']); +export const ResponseFormatSchema = z.enum(['text', 'json']); +export const SearchModeSchema = z.enum(['web', 'academic', 'sec']); +export const SearchContextSizeSchema = z.enum(['low', 'medium', 'high']); +export const RecencyFilterSchema = z.enum(['hour', 'day', 'week', 'month', 'year', 'none']); +export const ReasoningEffortSchema = z.enum(['low', 'medium', 'high']); + +export const TASK_PROFILE_SCHEMA_VERSION = 'l9-llm-task-profile/v1' as const; +export const TaskProfileSchema = z.object({ + schema_version: z.literal(TASK_PROFILE_SCHEMA_VERSION), + action: trimmed('action'), + node_id: trimmed('node_id', 128), + tenant_id: trimmed('tenant_id', 128), + task_family: TaskFamilySchema, + complexity: TaskComplexityLevelSchema, + data_sensitivity: DataSensitivitySchema, + requires_search: z.boolean(), + requires_citations: z.boolean(), + requires_json: z.boolean(), + required_output_schema: trimmed('required_output_schema', 2048).nullable(), + freshness_requirement: FreshnessRequirementSchema, + modality: ModalitySchema, + expected_output_tokens: z.number().int().positive().nullable(), + max_latency_class: LatencyClassSchema, + evidence_required: z.boolean(), + prompt_contract_ref: trimmed('prompt_contract_ref', 512).nullable(), + validation_profile: ValidationProfileSchema, + task_profile_hash: sha256, +}).strict().superRefine((profile, context) => { + if (profile.requires_json && !profile.required_output_schema) context.addIssue({ code: 'custom', path: ['required_output_schema'], message: 'JSON tasks require an output schema reference' }); + if (profile.requires_citations && !profile.requires_search && !profile.evidence_required) context.addIssue({ code: 'custom', path: ['requires_citations'], message: 'citations require search or explicit evidence' }); + if (profile.modality === 'vision' && profile.validation_profile !== 'visual') context.addIssue({ code: 'custom', path: ['validation_profile'], message: 'vision tasks require the visual validation profile' }); +}); +export type TaskProfile = z.infer; +export type TaskProfileInput = Omit; + +export const SelectedRouteSchema = z.object({ + provider: ProviderIdSchema, + model: trimmed('model', 256), + temperature: z.number().min(0).max(2), + max_tokens: z.number().int().positive(), + reasoning_depth: ReasoningDepthSchema, + response_format: ResponseFormatSchema, +}).strict(); + +export const SearchDecisionSchema = z.object({ + enabled: z.boolean(), + provider: z.literal('perplexity').nullable(), + search_mode: SearchModeSchema.nullable(), + search_context_size: SearchContextSizeSchema.nullable(), + recency_filter: RecencyFilterSchema.nullable(), + variations: z.number().int().positive().nullable(), + reasoning_effort: ReasoningEffortSchema.nullable(), +}).strict().superRefine((search, context) => { + const dependent = ['provider', 'search_mode', 'search_context_size', 'recency_filter', 'variations'] as const; + if (search.enabled) { + for (const key of dependent) if (search[key] === null) context.addIssue({ code: 'custom', path: [key], message: `${key} is required when search is enabled` }); + } else if ([search.provider, search.search_mode, search.search_context_size, search.recency_filter, search.variations, search.reasoning_effort].some(value => value !== null)) { + context.addIssue({ code: 'custom', message: 'disabled search must not carry search configuration' }); + } +}); + +export const RouteTargetSchema = z.object({ provider: ProviderIdSchema, model: trimmed('fallback model', 256) }).strict(); +export const PolicyDecisionSchema = z.object({ status: z.enum(['allowed', 'modified', 'blocked']), applied_rules: sortedUnique(trimmed('rule', 256)), blockers: sortedUnique(trimmed('blocker', 512)) }).strict().superRefine((decision, context) => { + if (decision.status === 'blocked' && decision.blockers.length === 0) context.addIssue({ code: 'custom', path: ['blockers'], message: 'blocked policy decisions require at least one blocker' }); + if (decision.status !== 'blocked' && decision.blockers.length > 0) context.addIssue({ code: 'custom', path: ['blockers'], message: 'non-blocked decisions cannot carry blockers' }); +}); +export type PolicyDecision = z.infer; + +export const BudgetDecisionSchema = z.object({ status: z.enum(['allowed', 'downgraded', 'blocked']), reason: trimmed('budget reason', 1024) }).strict(); +export type BudgetDecision = z.infer; + +export const ProviderHealthDecisionSchema = z.object({ status: z.enum(['healthy', 'degraded', 'unavailable', 'unknown']), reason: trimmed('health reason', 1024) }).strict(); + +export type ProviderHealthDecision = z.infer; + +export const LLM_ROUTE_PLAN_SCHEMA_VERSION = 'l9-llm-route-plan/v1' as const; +export const LLMRoutePlanSchema = z.object({ + schema_version: z.literal(LLM_ROUTE_PLAN_SCHEMA_VERSION), + route_fingerprint: routeIdentity, + plan_id: planIdentity, + request_id: trimmed('request_id', 256), + task_profile_hash: sha256, + selected: SelectedRouteSchema, + search: SearchDecisionSchema, + fallback_chain: sortedUnique(RouteTargetSchema), + policy_decision: PolicyDecisionSchema, + budget_decision: BudgetDecisionSchema, + provider_health_decision: ProviderHealthDecisionSchema, + learned_candidate_refs: sortedUnique(trimmed('candidate reference', 512)), + route_reason: trimmed('route_reason', 2048), + dispatch_allowed: z.boolean(), + content_hash: sha256, +}).strict().superRefine((plan, context) => { + const blocked = plan.policy_decision.status === 'blocked' || plan.budget_decision.status === 'blocked' || ['unavailable', 'unknown'].includes(plan.provider_health_decision.status); + if (plan.dispatch_allowed === blocked) context.addIssue({ code: 'custom', path: ['dispatch_allowed'], message: 'dispatch_allowed contradicts policy, budget, or health decisions' }); + if (plan.search.enabled && plan.selected.provider !== 'perplexity' && plan.search.provider !== 'perplexity') context.addIssue({ code: 'custom', path: ['search'], message: 'enabled search requires a Perplexity search decision' }); + if (plan.fallback_chain.some(target => target.provider === plan.selected.provider && target.model === plan.selected.model)) context.addIssue({ code: 'custom', path: ['fallback_chain'], message: 'fallback chain cannot repeat the selected route' }); +}); +export type LLMRoutePlan = z.infer; +export type LLMRoutePlanInput = Omit; + +export const LLM_EXECUTION_RECORD_SCHEMA_VERSION = 'l9-llm-execution-record/v1' as const; +export const LLMExecutionRecordSchema = z.object({ + schema_version: z.literal(LLM_EXECUTION_RECORD_SCHEMA_VERSION), + request_id: trimmed('request_id', 256), + plan_id: planIdentity, + route_fingerprint: routeIdentity, + tenant_id: trimmed('tenant_id', 128), + node_id: trimmed('node_id', 128), + action: trimmed('action'), + task_profile_hash: sha256, + provider: ProviderIdSchema, + model: trimmed('model', 256), + config_hash: sha256, + prompt_hash: sha256, + input_hash: sha256, + output_hash: sha256, + input_tokens: z.number().int().nonnegative(), + output_tokens: z.number().int().nonnegative(), + total_tokens: z.number().int().nonnegative(), + cost: z.number().nonnegative().finite(), + latency_ms: z.number().nonnegative().finite(), + citations_count: z.number().int().nonnegative(), + fallback_used: z.boolean(), + fallback_from: RouteTargetSchema.nullable(), + validation_status: z.enum(['passed', 'failed', 'not_run', 'blocked']), + schema_valid: z.boolean().nullable(), + downstream_accepted: z.boolean().nullable(), + quality_score: z.number().min(0).max(1).nullable(), + failure_reason: trimmed('failure_reason', 2048).nullable(), + provider_request_id: trimmed('provider_request_id', 256).nullable(), + pricing_version: trimmed('pricing_version', 256), + finish_reason: trimmed('finish_reason', 256).nullable(), + generated_at: z.string().datetime({ offset: true }), + content_hash: sha256, +}).strict().superRefine((record, context) => { + if (record.total_tokens !== record.input_tokens + record.output_tokens) context.addIssue({ code: 'custom', path: ['total_tokens'], message: 'total_tokens must equal input_tokens + output_tokens' }); + if (record.fallback_used !== (record.fallback_from !== null)) context.addIssue({ code: 'custom', path: ['fallback_from'], message: 'fallback metadata is inconsistent' }); + if (record.fallback_from && record.fallback_from.provider === record.provider && record.fallback_from.model === record.model) context.addIssue({ code: 'custom', path: ['fallback_from'], message: 'fallback origin cannot equal the final route' }); + if (record.validation_status === 'failed' && !record.failure_reason) context.addIssue({ code: 'custom', path: ['failure_reason'], message: 'failed validation requires a failure reason' }); + if (record.validation_status === 'blocked' && !record.failure_reason) context.addIssue({ code: 'custom', path: ['failure_reason'], message: 'blocked validation requires a reason' }); + if (record.validation_status === 'passed' && record.schema_valid !== true) context.addIssue({ code: 'custom', path: ['schema_valid'], message: 'passed validation requires schema_valid=true' }); + if (record.validation_status === 'passed' && record.failure_reason !== null) context.addIssue({ code: 'custom', path: ['failure_reason'], message: 'passed validation cannot carry a failure reason' }); + if (record.validation_status === 'not_run' && (record.schema_valid !== null || record.downstream_accepted !== null || record.failure_reason !== null)) context.addIssue({ code: 'custom', path: ['validation_status'], message: 'not_run validation cannot carry validation outcomes' }); + if (record.downstream_accepted === true && record.validation_status !== 'passed') context.addIssue({ code: 'custom', path: ['downstream_accepted'], message: 'accepted output requires passed validation' }); +}); +export type LLMExecutionRecord = z.infer; +export type LLMExecutionRecordInput = Omit; + +export const LLM_FEEDBACK_SIGNAL_SCHEMA_VERSION = 'l9-llm-router-signal/v1' as const; +export const LLMFeedbackSignalSchema = z.object({ + schema_version: z.literal(LLM_FEEDBACK_SIGNAL_SCHEMA_VERSION), + signal_family: z.literal('llm_routing'), + signal_type: z.enum(['route_success', 'route_failure', 'fallback_used', 'provider_degraded', 'high_cost', 'low_cost_high_quality', 'citation_missing', 'citation_strong', 'json_parse_failed', 'schema_validation_failed', 'hallucination_suspected', 'output_rejected', 'output_accepted', 'latency_high', 'retry_required', 'best_config_candidate']), + severity: z.enum(['critical', 'high', 'medium', 'low', 'info']), + task_profile_hash: sha256, + route_fingerprint: routeIdentity, + plan_id: planIdentity, + evidence_refs: sortedUnique(trimmed('evidence reference', 512)), + content_hash: sha256, +}).strict(); +export type LLMFeedbackSignal = z.infer; +export type LLMFeedbackSignalInput = Omit; diff --git a/src/control-plane/index.ts b/src/control-plane/index.ts new file mode 100644 index 0000000..f9c1c6a --- /dev/null +++ b/src/control-plane/index.ts @@ -0,0 +1,6 @@ +/** Internal Phase 1 barrel. Deliberately absent from package.json exports. */ +export * from './contracts.js'; +export * from './canonical-json.js'; +export * from './builders.js'; +export * from './policy-engine.js'; +export * from './provider-adapter.js'; diff --git a/src/control-plane/policy-engine.ts b/src/control-plane/policy-engine.ts new file mode 100644 index 0000000..7dba9f4 --- /dev/null +++ b/src/control-plane/policy-engine.ts @@ -0,0 +1,21 @@ +import type { PolicyDecision, TaskProfile } from './contracts.js'; + +export interface BudgetSnapshot { + snapshot_id: string; + tenant_id: string; + period: 'daily' | 'weekly' | 'monthly'; + remaining: number; + ceiling: number; + reserved: number; +} + +export interface CapabilitySnapshot { + snapshot_id: string; + provider: string; + models: readonly string[]; +} + +/** Policy evaluation must be pure over explicit immutable snapshots. */ +export interface PolicyEngine { + evaluate(profile: TaskProfile, budget: Readonly, capabilities: readonly Readonly[]): PolicyDecision; +} diff --git a/src/control-plane/provider-adapter.ts b/src/control-plane/provider-adapter.ts new file mode 100644 index 0000000..6954e13 --- /dev/null +++ b/src/control-plane/provider-adapter.ts @@ -0,0 +1,58 @@ +import type { LLMRoutePlan } from './contracts.js'; + +export interface ModelCapability { + provider: string; + model: string; + modality: 'text' | 'vision' | 'multimodal'; + max_context_tokens: number; + max_output_tokens: number; + supports_json_mode: boolean; + supports_search: boolean; + supports_citations: boolean; + supports_tools: boolean; + supports_streaming: boolean; + pricing_version: string; +} + +export interface LLMRequest { + system_prompt: string; + user_prompt: string; + images?: readonly string[]; + signal?: AbortSignal; +} + +export interface ExecutionContext { + trace_id: string; + attempt_id: string; + deadline_at: string; +} + +export interface LLMExecutionResult { + output: string; + provider_request_id: string | null; + input_tokens: number; + output_tokens: number; + total_tokens: number; + cost: number; + latency_ms: number; + citations: readonly string[]; + finish_reason: string | null; + pricing_version: string; +} + +export type AdapterFailureKind = 'network' | 'timeout' | 'rate_limit' | 'server' | 'client' | 'cancelled'; +export interface LLMProviderFailure { + provider: string; + kind: AdapterFailureKind; + retryable: boolean; + message: string; + status: number | null; + provider_request_id: string | null; + retry_after_ms: number | null; +} + +export interface LLMProviderAdapter { + readonly provider: string; + capabilities(): readonly ModelCapability[]; + execute(plan: LLMRoutePlan, request: LLMRequest, context: Readonly): Promise; +} diff --git a/tests/control-plane/boundary.test.ts b/tests/control-plane/boundary.test.ts new file mode 100644 index 0000000..18ef5c7 --- /dev/null +++ b/tests/control-plane/boundary.test.ts @@ -0,0 +1,42 @@ +import { readFileSync, readdirSync, statSync } from 'node:fs'; +import { join } from 'node:path'; +import ts from 'typescript'; +import { describe, expect, it } from 'vitest'; + +const ROOT = join(process.cwd(), 'src', 'control-plane'); +function files(directory: string): string[] { + return readdirSync(directory).flatMap(entry => { + const path = join(directory, entry); + return statSync(path).isDirectory() ? files(path) : path.endsWith('.ts') ? [path] : []; + }); +} + +describe('Control Plane fixed boundaries', () => { + it('contains no forbidden implementation imports', () => { + const forbidden = /(^|\/)(openai|axios|undici|graphiti|neo4j|providers?|gate|transportpacket)(\/|$)|node:https?|node:net|node:tls/i; + const offenders: string[] = []; + for (const file of files(ROOT)) { + const source = ts.createSourceFile(file, readFileSync(file, 'utf8'), ts.ScriptTarget.Latest, true, ts.ScriptKind.TS); + source.forEachChild(node => { + if (ts.isImportDeclaration(node) && ts.isStringLiteral(node.moduleSpecifier) && forbidden.test(node.moduleSpecifier.text)) offenders.push(`${file}:${node.moduleSpecifier.text}`); + }); + } + expect(offenders).toEqual([]); + }); + + it('contains no implicit time, randomness, network URLs, or tenant literals', () => { + const offenders: string[] = []; + for (const file of files(ROOT)) { + const text = readFileSync(file, 'utf8'); + if (/\bDate\.now\s*\(|\bnew\s+Date\s*\(|\bMath\.random\s*\(|\brandomUUID\s*\(/.test(text)) offenders.push(`${file}:volatile`); + if (/https?:\/\//.test(text)) offenders.push(`${file}:url`); + if (/safehavenrr|tenant-[0-9]/i.test(text)) offenders.push(`${file}:tenant`); + } + expect(offenders).toEqual([]); + }); + + it('is not exposed as a stable package export', () => { + const pkg = JSON.parse(readFileSync('package.json', 'utf8')); + expect(pkg.exports).not.toHaveProperty('./control-plane'); + }); +}); diff --git a/tests/control-plane/builders.test.ts b/tests/control-plane/builders.test.ts new file mode 100644 index 0000000..9ca49ff --- /dev/null +++ b/tests/control-plane/builders.test.ts @@ -0,0 +1,88 @@ +import { describe, expect, it } from 'vitest'; +import { buildExecutionRecord, buildFeedbackSignal, buildRoutePlan, buildTaskProfile, verifyExecutionRecord, verifyFeedbackSignal, verifyRoutePlan, verifyTaskProfile } from '../../src/control-plane/builders.js'; + +const profileInput = { + action: 'review', node_id: 'node-a', tenant_id: 'tenant-a', task_family: 'architecture_review' as const, + complexity: 'high' as const, data_sensitivity: 'internal' as const, requires_search: false, + requires_citations: false, requires_json: true, required_output_schema: 'schemas/review/v1', + freshness_requirement: 'none' as const, modality: 'text' as const, expected_output_tokens: 2000, + max_latency_class: 'normal' as const, evidence_required: true, prompt_contract_ref: 'prompts/review/v1', + validation_profile: 'strict_json' as const, +}; + +function routeInput(taskProfileHash: string, requestId = 'request-a', reason = 'Selected by policy') { + return { + request_id: requestId, + task_profile_hash: taskProfileHash, + selected: { provider: 'openrouter' as const, model: 'anthropic/claude-sonnet-4', temperature: 0.2, max_tokens: 2000, reasoning_depth: 'high' as const, response_format: 'json' as const }, + search: { enabled: false, provider: null, search_mode: null, search_context_size: null, recency_filter: null, variations: null, reasoning_effort: null }, + fallback_chain: [{ provider: 'openrouter' as const, model: 'openai/gpt-4o' }, { provider: 'openrouter' as const, model: 'google/gemini-2.5-pro' }], + policy_decision: { status: 'allowed' as const, applied_rules: ['tenant-default', 'internal-data'], blockers: [] }, + budget_decision: { status: 'allowed' as const, reason: 'within budget' }, + provider_health_decision: { status: 'healthy' as const, reason: 'recent success' }, + learned_candidate_refs: ['candidate-b', 'candidate-a'], + route_reason: reason, + dispatch_allowed: true, + }; +} + +describe('Control Plane builders', () => { + it('builds, freezes, and verifies a task profile', () => { + const profile = buildTaskProfile(profileInput); + expect(Object.isFrozen(profile)).toBe(true); + expect(verifyTaskProfile(profile)).toEqual(profile); + expect(() => verifyTaskProfile({ ...profile, action: 'tampered' })).toThrow(/hash mismatch/); + }); + + it('separates route equivalence, plan identity, and complete content', () => { + const profile = buildTaskProfile(profileInput); + const first = buildRoutePlan(routeInput(profile.task_profile_hash)); + const second = buildRoutePlan({ + ...routeInput(profile.task_profile_hash, 'request-b', 'Different prose'), + budget_decision: { status: 'allowed', reason: 'different budget explanation' }, + provider_health_decision: { status: 'healthy', reason: 'different health explanation' }, + }); + expect(first.route_fingerprint).toBe(second.route_fingerprint); + expect(first.plan_id).not.toBe(second.plan_id); + expect(first.content_hash).not.toBe(second.content_hash); + expect(first.fallback_chain.map(target => target.model)).toEqual(['google/gemini-2.5-pro', 'openai/gpt-4o']); + expect(first.learned_candidate_refs).toEqual(['candidate-a', 'candidate-b']); + expect(verifyRoutePlan(first)).toEqual(first); + }); + + it('rejects unsafe runtime input before normalization can execute it', () => { + const cyclic: Record = { ...profileInput }; + cyclic.self = cyclic; + expect(() => buildTaskProfile(cyclic as never)).toThrow(/cyclic/); + + let getterExecuted = false; + const accessor = { ...profileInput } as Record; + Object.defineProperty(accessor, 'action', { + enumerable: true, + get: () => { + getterExecuted = true; + return 'review'; + }, + }); + expect(() => buildTaskProfile(accessor as never)).toThrow(/accessors/); + expect(getterExecuted).toBe(false); + }); + + it('detects identity and content tampering', () => { + const profile = buildTaskProfile(profileInput); + const plan = buildRoutePlan(routeInput(profile.task_profile_hash)); + expect(() => verifyRoutePlan({ ...plan, route_fingerprint: `route_${'0'.repeat(32)}` })).toThrow(/fingerprint mismatch/); + expect(() => verifyRoutePlan({ ...plan, route_reason: 'tampered' })).toThrow(/hash mismatch/); + }); + + it('builds and verifies execution records and feedback signals', () => { + const profile = buildTaskProfile(profileInput); + const plan = buildRoutePlan(routeInput(profile.task_profile_hash)); + const hash = 'a'.repeat(64); + const record = buildExecutionRecord({ request_id: 'request-a', plan_id: plan.plan_id, route_fingerprint: plan.route_fingerprint, tenant_id: 'tenant-a', node_id: 'node-a', action: 'review', task_profile_hash: profile.task_profile_hash, provider: 'openrouter', model: 'anthropic/claude-sonnet-4', config_hash: hash, prompt_hash: hash, input_hash: hash, output_hash: hash, input_tokens: 10, output_tokens: 20, total_tokens: 30, cost: 0.02, latency_ms: 120, citations_count: 0, fallback_used: false, fallback_from: null, validation_status: 'passed', schema_valid: true, downstream_accepted: true, quality_score: 0.9, failure_reason: null, provider_request_id: 'provider-request', pricing_version: 'pricing/2026-07', finish_reason: 'stop', generated_at: '2026-07-20T12:00:00.000Z' }); + expect(verifyExecutionRecord(record)).toEqual(record); + const signal = buildFeedbackSignal({ signal_type: 'route_success', severity: 'info', task_profile_hash: profile.task_profile_hash, route_fingerprint: plan.route_fingerprint, plan_id: plan.plan_id, evidence_refs: ['evidence-b', 'evidence-a'] }); + expect(signal.evidence_refs).toEqual(['evidence-a', 'evidence-b']); + expect(verifyFeedbackSignal(signal)).toEqual(signal); + }); +}); diff --git a/tests/control-plane/canonical-json.test.ts b/tests/control-plane/canonical-json.test.ts new file mode 100644 index 0000000..72c4b11 --- /dev/null +++ b/tests/control-plane/canonical-json.test.ts @@ -0,0 +1,29 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; +import { canonicalJson, CanonicalJsonError, sha256Hex } from '../../src/control-plane/canonical-json.js'; + +describe('canonical JSON', () => { + it('matches the cross-language golden vector', () => { + const vector = JSON.parse(readFileSync('fixtures/control-plane/canonical-golden-v1.json', 'utf8')); + expect(canonicalJson(vector.input)).toBe(vector.canonical); + expect(sha256Hex(vector.input)).toBe(vector.sha256); + }); + + it('is independent of insertion order and Unicode composition', () => { + expect(canonicalJson({ z: 1, a: 'e\u0301' })).toBe(canonicalJson({ a: 'é', z: 1 })); + }); + + it.each([NaN, Infinity, -Infinity, undefined, 1n])('rejects unsupported scalar %s', value => { + expect(() => canonicalJson(value)).toThrow(CanonicalJsonError); + }); + + it('rejects cycles, sparse arrays, accessors, and normalization collisions', () => { + const cycle: Record = {}; cycle.self = cycle; + expect(() => canonicalJson(cycle)).toThrow(/cyclic/); + const sparse = Array(2); sparse[1] = 1; + expect(() => canonicalJson(sparse)).toThrow(/sparse/); + const accessor = Object.defineProperty({}, 'x', { enumerable: true, get: () => 1 }); + expect(() => canonicalJson(accessor)).toThrow(/accessors/); + expect(() => canonicalJson({ 'e\u0301': 1, 'é': 2 })).toThrow(/duplicate key/); + }); +}); diff --git a/tests/control-plane/contracts.test.ts b/tests/control-plane/contracts.test.ts new file mode 100644 index 0000000..a327ff5 --- /dev/null +++ b/tests/control-plane/contracts.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, it } from 'vitest'; +import { buildExecutionRecord, buildRoutePlan, buildTaskProfile } from '../../src/control-plane/builders.js'; + +const baseProfile = { action: 'vision', node_id: 'n', tenant_id: 't', task_family: 'vision_analysis' as const, complexity: 'medium' as const, data_sensitivity: 'public' as const, requires_search: false, requires_citations: false, requires_json: false, required_output_schema: null, freshness_requirement: 'none' as const, modality: 'vision' as const, expected_output_tokens: 100, max_latency_class: 'normal' as const, evidence_required: false, prompt_contract_ref: null, validation_profile: 'visual' as const }; + +describe('Control Plane contract refinements', () => { + it('rejects contradictory TaskProfile fields', () => { + expect(() => buildTaskProfile({ ...baseProfile, requires_json: true, required_output_schema: null })).toThrow(/output schema/); + expect(() => buildTaskProfile({ ...baseProfile, modality: 'vision', validation_profile: 'freeform' })).toThrow(/visual/); + }); + + it('rejects disabled search carrying configuration', () => { + const profile = buildTaskProfile(baseProfile); + expect(() => buildRoutePlan({ request_id: 'r', task_profile_hash: profile.task_profile_hash, selected: { provider: 'openrouter', model: 'm', temperature: 0, max_tokens: 1, reasoning_depth: 'none', response_format: 'text' }, search: { enabled: false, provider: 'perplexity', search_mode: null, search_context_size: null, recency_filter: null, variations: null, reasoning_effort: null }, fallback_chain: [], policy_decision: { status: 'allowed', applied_rules: [], blockers: [] }, budget_decision: { status: 'allowed', reason: 'ok' }, provider_health_decision: { status: 'healthy', reason: 'ok' }, learned_candidate_refs: [], route_reason: 'ok', dispatch_allowed: true })).toThrow(/disabled search/); + }); + + it('rejects blocked decisions marked dispatchable', () => { + const profile = buildTaskProfile(baseProfile); + expect(() => buildRoutePlan({ request_id: 'r', task_profile_hash: profile.task_profile_hash, selected: { provider: 'openrouter', model: 'm', temperature: 0, max_tokens: 1, reasoning_depth: 'none', response_format: 'text' }, search: { enabled: false, provider: null, search_mode: null, search_context_size: null, recency_filter: null, variations: null, reasoning_effort: null }, fallback_chain: [], policy_decision: { status: 'blocked', applied_rules: [], blockers: ['policy'] }, budget_decision: { status: 'allowed', reason: 'ok' }, provider_health_decision: { status: 'healthy', reason: 'ok' }, learned_candidate_refs: [], route_reason: 'blocked', dispatch_allowed: true })).toThrow(/dispatch_allowed/); + expect(() => buildRoutePlan({ request_id: 'r', task_profile_hash: profile.task_profile_hash, selected: { provider: 'openrouter', model: 'm', temperature: 0, max_tokens: 1, reasoning_depth: 'none', response_format: 'text' }, search: { enabled: false, provider: null, search_mode: null, search_context_size: null, recency_filter: null, variations: null, reasoning_effort: null }, fallback_chain: [], policy_decision: { status: 'allowed', applied_rules: [], blockers: [] }, budget_decision: { status: 'allowed', reason: 'ok' }, provider_health_decision: { status: 'unknown', reason: 'no evidence' }, learned_candidate_refs: [], route_reason: 'unknown health', dispatch_allowed: true })).toThrow(/dispatch_allowed/); + }); + + it('rejects impossible execution accounting', () => { + const hash = 'a'.repeat(64); + expect(() => buildExecutionRecord({ request_id: 'r', plan_id: `plan_${'a'.repeat(32)}`, route_fingerprint: `route_${'b'.repeat(32)}`, tenant_id: 't', node_id: 'n', action: 'a', task_profile_hash: hash, provider: 'openrouter', model: 'm', config_hash: hash, prompt_hash: hash, input_hash: hash, output_hash: hash, input_tokens: 1, output_tokens: 2, total_tokens: 99, cost: 0, latency_ms: 0, citations_count: 0, fallback_used: false, fallback_from: null, validation_status: 'not_run', schema_valid: null, downstream_accepted: null, quality_score: null, failure_reason: null, provider_request_id: null, pricing_version: 'v', finish_reason: null, generated_at: '2026-07-20T12:00:00.000Z' })).toThrow(/total_tokens/); + expect(() => buildExecutionRecord({ request_id: 'r', plan_id: `plan_${'a'.repeat(32)}`, route_fingerprint: `route_${'b'.repeat(32)}`, tenant_id: 't', node_id: 'n', action: 'a', task_profile_hash: hash, provider: 'openrouter', model: 'm', config_hash: hash, prompt_hash: hash, input_hash: hash, output_hash: hash, input_tokens: 1, output_tokens: 2, total_tokens: 3, cost: 0, latency_ms: 0, citations_count: 0, fallback_used: false, fallback_from: null, validation_status: 'passed', schema_valid: null, downstream_accepted: true, quality_score: null, failure_reason: null, provider_request_id: null, pricing_version: 'v', finish_reason: null, generated_at: '2026-07-20T12:00:00.000Z' })).toThrow(/schema_valid/); + expect(() => buildExecutionRecord({ request_id: 'r', plan_id: `plan_${'a'.repeat(32)}`, route_fingerprint: `route_${'b'.repeat(32)}`, tenant_id: 't', node_id: 'n', action: 'a', task_profile_hash: hash, provider: 'openrouter', model: 'm', config_hash: hash, prompt_hash: hash, input_hash: hash, output_hash: hash, input_tokens: 1, output_tokens: 2, total_tokens: 3, cost: 0, latency_ms: 0, citations_count: 0, fallback_used: false, fallback_from: null, validation_status: 'not_run', schema_valid: true, downstream_accepted: null, quality_score: null, failure_reason: null, provider_request_id: null, pricing_version: 'v', finish_reason: null, generated_at: '2026-07-20T12:00:00.000Z' })).toThrow(/not_run/); + }); +}); diff --git a/tests/control-plane/provider-adapter.test.ts b/tests/control-plane/provider-adapter.test.ts new file mode 100644 index 0000000..b89048a --- /dev/null +++ b/tests/control-plane/provider-adapter.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, it } from 'vitest'; +import type { LLMProviderAdapter, LLMExecutionResult, ModelCapability } from '../../src/control-plane/provider-adapter.js'; +import type { LLMRoutePlan } from '../../src/control-plane/contracts.js'; + +class FakeAdapter implements LLMProviderAdapter { + readonly provider = 'fake'; + capabilities(): readonly ModelCapability[] { + return [{ provider: 'fake', model: 'model-a', modality: 'multimodal', max_context_tokens: 1000, max_output_tokens: 100, supports_json_mode: true, supports_search: false, supports_citations: false, supports_tools: true, supports_streaming: true, pricing_version: 'pricing/v1' }]; + } + async execute(_plan: LLMRoutePlan, _request: { system_prompt: string; user_prompt: string; signal?: AbortSignal }, context: { trace_id: string; attempt_id: string; deadline_at: string }): Promise { + return { output: `${context.trace_id}:${context.attempt_id}`, provider_request_id: 'provider-request', input_tokens: 1, output_tokens: 1, total_tokens: 2, cost: 0.01, latency_ms: 1, citations: [], finish_reason: 'stop', pricing_version: 'pricing/v1' }; + } +} + +describe('provider adapter contract', () => { + it('supports capability, cancellation, attempt, usage, and pricing metadata', async () => { + const adapter: LLMProviderAdapter = new FakeAdapter(); + expect(adapter.capabilities()[0]).toMatchObject({ supports_tools: true, pricing_version: 'pricing/v1' }); + const result = await adapter.execute({} as LLMRoutePlan, { system_prompt: 's', user_prompt: 'u', signal: new AbortController().signal }, { trace_id: 'trace', attempt_id: 'attempt', deadline_at: '2026-07-20T12:00:00.000Z' }); + expect(result).toMatchObject({ output: 'trace:attempt', total_tokens: 2, provider_request_id: 'provider-request' }); + }); +});