From bace947c9538f984df71f9e96110ff28d41f9cba Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 11:34:38 +0000 Subject: [PATCH] feat: v1 declares memory; the audit argues with the spec's own claim (0.3.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two holes named in #14's out-of-scope, closed. Neither needed a decision: the memory shape already existed in this repo, and comparing a claim against a computation is a tool feature, not vocabulary. ## agent-spec v1 declares memory A required block, per scope: name, purpose, retention, user_visible, user_editable, lawful_basis, plus a forget mechanism. The shape follows memory-policy.schema.yaml, so nothing was invented — it was ported. Required, not optional. An agent that keeps nothing writes `memory: { persistent: false }`, because silence and "none" are different claims and only one of them is checkable. Consequences: - aux.H08 is scoreable under v1, so the three memory gaps in the taxonomy (memory_amnesia, preference_ignored, context_leak) are reachable from a v1 spec again. A quarter of the taxonomy was out of reach. - aux.T02 Contextual Trust is assessable, so the ladder can pass it. - v1 says more than v0 could: whether each scope is bounded, and whether the user can have it forgotten. A retention period with no forget mechanism is a promise about the calendar, not a control the user holds. - Memory the user cannot see costs aux.H02; memory they cannot correct costs aux.H03. Same split v0 made, where user_editable fed H03. Caught by a test that expected H08 to punish a hidden ten-year scope and found the rule ignoring visibility entirely. ## The audit argues with the claim v1 specs may declare a trust_stage. It is never an input to the score. It is compared against it: The spec claims advocacy; the evidence supports no stage at all. A stage is earned by what the spec can show, not by what it says about itself. Claiming more than the evidence supports is a `high` finding naming what blocks it. Claiming less is surfaced next to the computed stage and nothing more — modesty is not a defect, and the reference example under-claims, which is now pinned by a test. 55 tests, up from 48. v0 scoring unchanged: 100/A/advocacy and 4/F, same evidence strings. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MhpqWEvwKp7NrmJfdXYq4M --- CHANGELOG.md | 13 +++ packages/aux-audit/README.md | 8 +- .../aux-audit/fixtures/v1-strong-spec.yaml | 18 +++ packages/aux-audit/fixtures/v1-weak-spec.yaml | 7 ++ packages/aux-audit/package.json | 2 +- .../aux-audit/schemas/agent-spec.schema.yaml | 59 ++++++++++ packages/aux-audit/src/cli.ts | 2 +- packages/aux-audit/src/evidence.ts | 25 ++++- packages/aux-audit/src/report/markdown.ts | 7 ++ packages/aux-audit/src/rules.ts | 35 +++++- packages/aux-audit/src/score.ts | 30 ++++- packages/aux-audit/src/spec-v1.ts | 16 +++ packages/aux-audit/src/types.ts | 5 + packages/aux-audit/test/audit.test.ts | 105 +++++++++++++++--- packages/aux-audit/test/report.test.ts | 2 +- packages/aux-audit/test/spec.test.ts | 4 +- schemas/MIGRATION.md | 15 +++ schemas/agent-spec.schema.yaml | 59 ++++++++++ 18 files changed, 388 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2670a27..930b8ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,19 @@ Publishing the GitHub release and tagging `v0.2` is a manual step after merge. - `CITATION.cff` for GitHub’s *Cite this repository* button. - Project-home links to [auxfirst.com](https://auxfirst.com). +## aux-audit 0.3.0 + +- **agent-spec v1 declares memory.** A required block, per scope: name, retention, + visibility, editability, lawful basis, plus a forget mechanism. Shape follows + `memory-policy.schema.yaml`, so nothing new was invented. `aux.H08` is scoreable + under v1 and `aux.T02` Contextual Trust is assessable again — the three memory + gaps in the taxonomy are reachable from a v1 spec. +- Memory the user cannot see costs `aux.H02`; memory they cannot correct costs + `aux.H03`. Same split v0 made, where `user_editable` fed H03. +- **The audit argues with the spec's claim.** `trust_stage` is compared against the + computed stage, never used as input. Claiming more than the evidence supports is + a `high` finding naming what blocks it. Claiming less is reported and nothing more. + ## aux-audit 0.2.0 - Scores **agent-spec v1.0**. Both formats are read, detected from the document diff --git a/packages/aux-audit/README.md b/packages/aux-audit/README.md index 8f3cd41..cea61a7 100644 --- a/packages/aux-audit/README.md +++ b/packages/aux-audit/README.md @@ -38,7 +38,13 @@ Under **v1.0** the audit asks a better question. Instead of *"what did you call Under **v0.1.0** nothing changed: the same label and the same prose guarantees produce the same scores as before. Scores are not comparable across the two, so `meta.spec_version` records which one produced them. -One thing v1 cannot express: **memory**. It has no field for persistence, scoping, or retention, so `aux.H08` is reported as not scoreable rather than scored zero — and `aux.T02` Contextual Trust, whose only backing heuristic that is, comes back *not assessable* rather than earned. See [trustkit#10](https://github.com/auxfirst/trustkit/issues/10). +**v1 declares memory**, per scope: what is kept, for how long, whether the user can see it, correct it, and have it forgotten. `aux.H08` scores whether the memory is bounded and forgettable; visibility feeds `aux.H02` and editability `aux.H03`, the same split v0 made. + +**The audit argues with the spec's own claim.** A v1 spec may declare a `trust_stage`. That is never an input to the score — it is compared against it, and a spec claiming more than it shows gets told so: + +> **The spec claims *advocacy*; the evidence supports *no stage at all*.** A stage is earned by what the spec can show, not by what it says about itself. + +Under-claiming is surfaced next to the computed stage and nothing more. Modesty is not a defect. Two consequences, both deliberate: diff --git a/packages/aux-audit/fixtures/v1-strong-spec.yaml b/packages/aux-audit/fixtures/v1-strong-spec.yaml index d3d23a3..49fa62c 100644 --- a/packages/aux-audit/fixtures/v1-strong-spec.yaml +++ b/packages/aux-audit/fixtures/v1-strong-spec.yaml @@ -210,3 +210,21 @@ status: live # The approver sees the draft but not the payment-history evidence behind the # ranking. Tracked as tg.judgment.confident_nonsense risk; fix is the # confidence-cues pattern on the approval queue. +memory: + persistent: true + scopes: + - name: invoice_history + purpose: Recognise repeat late payers without re-reading the ledger each run. + retention: P365D + user_visible: true + user_editable: true + lawful_basis: legitimate_interest + - name: contact_preferences + purpose: Respect a customer's stated channel and cadence. + retention: P90D + user_visible: true + user_editable: true + lawful_basis: consent + forget: + available: true + mechanism: Deletion request routed to the data team; purges both scopes within 30 days. diff --git a/packages/aux-audit/fixtures/v1-weak-spec.yaml b/packages/aux-audit/fixtures/v1-weak-spec.yaml index 0440a25..a9bac37 100644 --- a/packages/aux-audit/fixtures/v1-weak-spec.yaml +++ b/packages/aux-audit/fixtures/v1-weak-spec.yaml @@ -34,3 +34,10 @@ exceptions: shutdown: procedure: "Revoke the OAuth token in the admin console." tested: false +memory: + persistent: true + scopes: + - name: everything_seen + retention: P3650D + user_visible: false + user_editable: false diff --git a/packages/aux-audit/package.json b/packages/aux-audit/package.json index 49e6382..6b46812 100644 --- a/packages/aux-audit/package.json +++ b/packages/aux-audit/package.json @@ -1,6 +1,6 @@ { "name": "aux-audit", - "version": "0.2.0", + "version": "0.3.0", "description": "Score an agent product against the 10 AUX Heuristics and the Trust Architecture. Reproducible, CI-friendly, SARIF-capable.", "keywords": [ "aux", diff --git a/packages/aux-audit/schemas/agent-spec.schema.yaml b/packages/aux-audit/schemas/agent-spec.schema.yaml index 38333be..b46eefd 100644 --- a/packages/aux-audit/schemas/agent-spec.schema.yaml +++ b/packages/aux-audit/schemas/agent-spec.schema.yaml @@ -30,6 +30,7 @@ required: - trigger - systems - capability + - memory - mandate - human_control - exceptions @@ -152,6 +153,64 @@ properties: type: array items: { type: string } + memory: + type: object + description: >- + What the agent retains between runs. Required: an agent with no memory + declares `persistent: false` rather than staying silent, because silence + and "none" are different claims and only one of them is checkable. + Shape follows memory-policy.schema.yaml. + additionalProperties: false + required: + - persistent + properties: + persistent: + type: boolean + description: Does anything survive the end of a run? + scopes: + type: array + description: >- + One entry per category of retained content. Required when persistent + is true — "we remember things" is not a memory policy. + items: + type: object + additionalProperties: false + required: + - name + - retention + - user_visible + - user_editable + properties: + name: + type: string + description: The category, e.g. preferences, conversation_history. + purpose: + type: string + retention: + type: string + pattern: "^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?(T(?!$)(\\d+H)?(\\d+M)?(\\d+S)?)?$" + description: ISO-8601 duration. P90D, P1Y. "Indefinite" is a decision, so write it as one. + user_visible: + type: boolean + user_editable: + type: boolean + lawful_basis: + type: string + enum: [consent, contract, legitimate_interest, legal_obligation, not_applicable] + forget: + type: object + description: >- + Deletion on request. A retention period with no forget mechanism is a + promise about the calendar, not a control the user holds. + additionalProperties: false + required: + - available + properties: + available: + type: boolean + mechanism: + type: string + mandate: type: array description: >- diff --git a/packages/aux-audit/src/cli.ts b/packages/aux-audit/src/cli.ts index 0ffa42b..5129b1b 100644 --- a/packages/aux-audit/src/cli.ts +++ b/packages/aux-audit/src/cli.ts @@ -20,7 +20,7 @@ import { toMarkdown } from "./report/markdown.js"; import { toSarif } from "./report/sarif.js"; import { SEVERITY_ORDER, type Severity } from "./types.js"; -const VERSION = "0.2.0"; +const VERSION = "0.3.0"; const FORMATS = new Set(["md", "markdown", "json", "sarif"]); const SEVERITIES = new Set(["low", "medium", "high", "critical"]); diff --git a/packages/aux-audit/src/evidence.ts b/packages/aux-audit/src/evidence.ts index 7c31841..01087d2 100644 --- a/packages/aux-audit/src/evidence.ts +++ b/packages/aux-audit/src/evidence.ts @@ -17,6 +17,10 @@ export interface MemoryEvidence { scopeCount: number; userVisible: boolean; userEditable: boolean; + /** v1 only: every scope names a retention period. v0 had one for all of them. */ + everyScopeRetained?: boolean; + /** v1 only: deletion on request is a control the user holds, not a calendar. */ + forgettable?: boolean; } export interface MandateEvidence { @@ -62,6 +66,12 @@ export interface Evidence { /** True once the spec carries evidence of testing, which gates level 3. */ evidenced: boolean; + + /** + * The trust stage the spec claims for itself. v1 only, and the audit's job is + * to agree or disagree with it — not to take it as input to the score. + */ + claimedStage?: string; } /** @@ -137,8 +147,18 @@ export function fromV1(spec: AgentSpecV1): Evidence { escalationDefault: spec.escalation_default !== undefined, shutdownTested: spec.shutdown.tested, modelPinned: spec.model?.version_pinned ?? false, - // v1 declares no memory. Not false — absent. See trustkit#10. - memory: undefined, + memory: { + persistent: spec.memory.persistent, + scopeCount: (spec.memory.scopes ?? []).length, + // v1 declares visibility per scope. A memory is user-visible only if + // every retained category is — one hidden scope is a hidden scope. + userVisible: (spec.memory.scopes ?? []).every((s) => s.user_visible), + userEditable: (spec.memory.scopes ?? []).every((s) => s.user_editable), + everyScopeRetained: (spec.memory.scopes ?? []).every( + (s) => typeof s.retention === "string" && s.retention.length > 1, + ), + forgettable: spec.memory.forget?.available === true, + }, guarantees: [], tools: spec.capability.can_change, flows: [], @@ -148,5 +168,6 @@ export function fromV1(spec: AgentSpecV1): Evidence { // A tested shutdown and a pinned model are v1's evidence of exercise, in // the way transcripts are v0's. evidenced: spec.shutdown.tested === true, + claimedStage: spec.trust_stage, }; } diff --git a/packages/aux-audit/src/report/markdown.ts b/packages/aux-audit/src/report/markdown.ts index 7c66340..4cbdc53 100644 --- a/packages/aux-audit/src/report/markdown.ts +++ b/packages/aux-audit/src/report/markdown.ts @@ -36,6 +36,13 @@ export function toMarkdown(report: AuditReport): string { lines.push( `**Score ${report.score}/100 · Grade ${report.grade} · Trust stage: ${report.trust_stage ?? "none earned"}**`, ); + if (report.trust_stage_claimed && report.trust_stage_claimed !== report.trust_stage) { + lines.push(""); + lines.push( + `> **The spec claims *${report.trust_stage_claimed}*; the evidence supports *${report.trust_stage ?? "no stage at all"}*.** ` + + "A stage is earned by what the spec can show, not by what it says about itself.", + ); + } lines.push(""); lines.push("### Trust Architecture"); diff --git a/packages/aux-audit/src/rules.ts b/packages/aux-audit/src/rules.ts index 392fc9e..c674456 100644 --- a/packages/aux-audit/src/rules.ts +++ b/packages/aux-audit/src/rules.ts @@ -101,10 +101,12 @@ export const RULES: Rule[] = [ evaluate(e) { if (isV1(e)) { const observe = control(e, "observe"); + const memoryHidden = e.memory !== undefined && e.memory.persistent && !e.memory.userVisible; const level = clamp( (has(e, "observe") ? 1 : 0) + (supervises(e, "provenance_at_decision") ? 1 : 0) + - (observe?.audience === "affected_user" ? 1 : 0), + (observe?.audience === "affected_user" ? 1 : 0) - + (memoryHidden ? 1 : 0), ); return { level, @@ -119,6 +121,7 @@ export const RULES: Rule[] = [ observe?.audience === "affected_user" ? "visible to the affected user, not only operators" : `observability reaches ${observe?.audience ?? "no one"}`, + memoryHidden && "at least one retained scope is hidden from the user", ]), }; } @@ -143,10 +146,12 @@ export const RULES: Rule[] = [ id: "aux.H03", evaluate(e) { if (isV1(e)) { + const memoryStuck = e.memory !== undefined && e.memory.persistent && !e.memory.userEditable; const level = clamp( (has(e, "interrupt") ? 1 : 0) + (has(e, "override") ? 1 : 0) + - (supervises(e, "reversal") ? 1 : 0), + (supervises(e, "reversal") ? 1 : 0) - + (memoryStuck ? 1 : 0), ); return { level, @@ -155,6 +160,7 @@ export const RULES: Rule[] = [ has(e, "interrupt") ? "a run can be interrupted mid-flight" : "no interrupt control", has(e, "override") ? "a decision can be overridden" : "no override control", supervises(e, "reversal") ? "actions are reversible" : "actions are not reversible", + memoryStuck && "a retained scope cannot be corrected by the user", ]), }; } @@ -367,6 +373,31 @@ export const RULES: Rule[] = [ }; } const scopes = e.memory.scopeCount; + if (isV1(e)) { + // v1 declares retention and visibility per scope, so it can say more + // than "memory exists": whether each category is bounded, and whether + // the user can make it go away. + const level = clamp( + (scopes >= 1 ? 1 : 0) + + (e.memory.everyScopeRetained ? 1 : 0) + + (e.memory.forgettable ? 1 : 0), + ); + return { + level, + applicable: true, + evidence: join([ + scopes >= 1 + ? `memory is persistent across ${scopes} scope(s)` + : "memory is persistent but no scope is declared — what is kept is unstated", + e.memory.everyScopeRetained + ? "every scope names a retention period" + : "at least one scope is retained indefinitely, or does not say", + e.memory.forgettable + ? "the user can have it forgotten on request" + : "no forget mechanism — retention is a promise about the calendar, not a control", + ]), + }; + } return { level: clamp(2 + (scopes >= 2 ? 1 : 0)), applicable: true, diff --git a/packages/aux-audit/src/score.ts b/packages/aux-audit/src/score.ts index 6a5e3de..8d881f4 100644 --- a/packages/aux-audit/src/score.ts +++ b/packages/aux-audit/src/score.ts @@ -168,6 +168,16 @@ export function audit(evidence: Evidence, options: AuditOptions = {}): AuditRepo const highest = [...trust_stages].reverse().find((s) => s.earned); const trust_stage = highest ? highest.name.toLowerCase().replace(/ trust$/, "") : null; + // A spec may claim a stage. The audit's job is to agree or disagree with the + // claim, never to take it as input — so it is compared, not scored. + const claimed = evidence.claimedStage + ? trust_stages.find((s) => s.id === evidence.claimedStage) + : undefined; + const trust_stage_claimed = claimed + ? claimed.name.toLowerCase().replace(/ trust$/, "") + : null; + const earnedOrder = highest?.order ?? 0; + const issues: Issue[] = []; const recommendations: string[] = []; @@ -195,6 +205,23 @@ export function audit(evidence: Evidence, options: AuditOptions = {}): AuditRepo }); } + if (claimed && claimed.order > earnedOrder) { + const blocker = trust_stages + .filter((s) => s.order <= claimed.order && !s.earned) + .flatMap((s) => + s.assessable ? s.shortfall : [`${s.depends_on.join(", ")} (not assessable)`], + ); + issues.push({ + id: claimed.id, + type: "overclaimed_trust_stage", + severity: "high", + evidence: + `the spec claims ${claimed.id} ${claimed.name}; the evidence supports ` + + `${highest ? `${highest.id} ${highest.name}` : "no stage at all"}` + + (blocker.length > 0 ? ` — blocked by ${[...new Set(blocker)].join(", ")}` : ""), + }); + } + issues.sort( (a, b) => SEVERITY_ORDER[b.severity] - SEVERITY_ORDER[a.severity] || @@ -206,6 +233,7 @@ export function audit(evidence: Evidence, options: AuditOptions = {}): AuditRepo score, grade: grade(score), trust_stage, + trust_stage_claimed, evolution_stage: null, evolution_stage_status: "human-assessed", heuristics, @@ -215,7 +243,7 @@ export function audit(evidence: Evidence, options: AuditOptions = {}): AuditRepo meta: { tool: "aux-audit", spec_version: evidence.specVersion, - tool_version: options.toolVersion ?? "0.2.0", + tool_version: options.toolVersion ?? "0.3.0", heuristics_version: canon.heuristicsVersion, trust_architecture_version: canon.trustArchitectureVersion, taxonomy_version: canon.taxonomyVersion, diff --git a/packages/aux-audit/src/spec-v1.ts b/packages/aux-audit/src/spec-v1.ts index 53c7ef4..297cffa 100644 --- a/packages/aux-audit/src/spec-v1.ts +++ b/packages/aux-audit/src/spec-v1.ts @@ -52,6 +52,21 @@ export interface ExceptionRow { escalation?: Escalation; } +export interface MemoryScope { + name: string; + purpose?: string; + retention: string; + user_visible: boolean; + user_editable: boolean; + lawful_basis?: string; +} + +export interface MemorySpecV1 { + persistent: boolean; + scopes?: MemoryScope[]; + forget?: { available: boolean; mechanism?: string }; +} + export interface AgentSpecV1 { spec_version: string; id: string; @@ -64,6 +79,7 @@ export interface AgentSpecV1 { users?: unknown[]; systems: { data_sources: string[]; connected: { name: string; auth: unknown }[] }; capability: { can_read: string[]; can_change: string[] }; + memory: MemorySpecV1; mandate: MandateRow[]; human_control: Record<"observe" | "interrupt" | "approve" | "override" | "disable", Control>; exceptions: ExceptionRow[]; diff --git a/packages/aux-audit/src/types.ts b/packages/aux-audit/src/types.ts index 0effe44..9210911 100644 --- a/packages/aux-audit/src/types.ts +++ b/packages/aux-audit/src/types.ts @@ -75,6 +75,11 @@ export interface AuditReport { score: number; grade: Grade; trust_stage: string | null; + /** + * The stage the spec claimed, when it claimed one. Reported next to the + * computed stage so the two can disagree in public. + */ + trust_stage_claimed: string | null; /** * Always null, by design — not pending. * diff --git a/packages/aux-audit/test/audit.test.ts b/packages/aux-audit/test/audit.test.ts index 487058b..5e21435 100644 --- a/packages/aux-audit/test/audit.test.ts +++ b/packages/aux-audit/test/audit.test.ts @@ -202,6 +202,7 @@ test("v1: a prompt is not an enforcing mechanism", () => { trigger: { kind: "event" }, systems: { data_sources: ["s"], connected: [{ name: "n", auth: "service_account" }] }, capability: { can_read: [], can_change: ["refund"] }, + memory: { persistent: false }, mandate: [{ action: "refund", authority: "human_approval", enforced_by: "the system prompt" }], human_control: { observe: { available: true }, @@ -219,32 +220,110 @@ test("v1: a prompt is not an enforcing mechanism", () => { assert.match(h05.evidence, /the system prompt/); }); -test("v1: aux.H08 is not scoreable, and says why rather than scoring zero", () => { - const h08 = v1strong().heuristics.find((h) => h.id === "aux.H08")!; +test("a heuristic with nothing to read is not scoreable, and says why", () => { + // Guards the shape of the escape hatch, not a current gap: v1 gained a + // memory field, so this is exercised on evidence that lacks one entirely. + const report = audit({ ...loadEvidence(fixture("v1-strong-spec.yaml")), memory: undefined }); + const h08 = report.heuristics.find((h) => h.id === "aux.H08")!; assert.equal(h08.applicable, false); - assert.match(h08.evidence, /no memory field/); - assert.ok(!v1strong().issues.some((i) => i.id === "aux.H08"), "must not be reported as a failing"); + assert.match(h08.evidence, /not scoreable/); + assert.ok(!report.issues.some((i) => i.id === "aux.H08"), "must not be reported as a failing"); }); test("a stage with no scoreable evidence is never reported as earned", () => { - // The bug this guards: aux.H08 is the only heuristic backing aux.T02. Under - // v1 it cannot be scored, and an empty shortfall was reading as "earned" — - // a claim dressed as a finding. - const contextual = v1strong().trust_stages.find((s) => s.id === "aux.T02")!; + // The bug this guards: aux.T02 is backed only by aux.H08. When H08 could not + // be scored, an empty shortfall read as "earned" — a claim dressed as a + // finding, on zero evidence. + const report = audit({ ...loadEvidence(fixture("v1-strong-spec.yaml")), memory: undefined }); + const contextual = report.trust_stages.find((s) => s.id === "aux.T02")!; assert.equal(contextual.assessable, false); assert.equal(contextual.earned, false); - assert.ok(!v1strong().issues.some((i) => i.id === "aux.T02"), "not the product's failure to fix"); + // No *stage gap* is raised: an unassessable stage is the format's limit, not + // the agent's failing. (An overclaim issue may still carry this id, and does + // here, because the fixture declares aux.T02 — that is a different finding.) + assert.ok( + !report.issues.some((i) => i.id === "aux.T02" && i.type.endsWith("_trust_gap")), + "an unassessable stage must not be reported as a gap the product should fix", + ); }); test("an unassessable stage stops the ladder, as a broken one would", () => { - const stages = v1strong().trust_stages; - assert.equal(stages.find((s) => s.id === "aux.T01")!.earned, true); + const report = audit({ ...loadEvidence(fixture("v1-strong-spec.yaml")), memory: undefined }); + assert.equal(report.trust_stages.find((s) => s.id === "aux.T01")!.earned, true); for (const id of ["aux.T02", "aux.T03", "aux.T04"]) { - assert.equal(stages.find((s) => s.id === id)!.earned, false, `${id} must not be earned`); + assert.equal(report.trust_stages.find((s) => s.id === id)!.earned, false, `${id} earned`); } - assert.equal(v1strong().trust_stage, "functional"); + assert.equal(report.trust_stage, "functional"); }); test("v0 stages stay assessable — the change is version-specific", () => { assert.ok(strong().trust_stages.every((s) => s.assessable)); }); + +test("v1: memory is scoreable now that the schema has a field for it", () => { + const h08 = v1strong().heuristics.find((h) => h.id === "aux.H08")!; + assert.equal(h08.applicable, true); + assert.ok(h08.level >= 2, "declared scopes, retention and a forget mechanism should score"); + assert.match(h08.evidence, /forgotten on request/); +}); + +test("v1: contextual trust is assessable again, so the ladder can pass it", () => { + const contextual = v1strong().trust_stages.find((s) => s.id === "aux.T02")!; + assert.equal(contextual.assessable, true); + assert.equal(contextual.earned, true); +}); + +test("v1: memory the user cannot see or correct costs transparency and steering", () => { + // Context efficiency is about whether the agent remembers. Whether the user + // can see and correct what it remembers is H02 and H03 — the same split v0 + // made, where user_editable fed H03. + const report = v1weak(); + const h08 = report.heuristics.find((h) => h.id === "aux.H08")!; + assert.equal(h08.applicable, true); + assert.match(h08.evidence, /no forget mechanism/); + + assert.match( + report.heuristics.find((h) => h.id === "aux.H02")!.evidence, + /hidden from the user/, + ); + assert.match( + report.heuristics.find((h) => h.id === "aux.H03")!.evidence, + /cannot be corrected by the user/, + ); +}); + +test("v1: a spec claiming more than it shows is contradicted by name", () => { + const spec = loadEvidence(fixture("v1-weak-spec.yaml")); + const report = audit({ ...spec, claimedStage: "aux.T04" }); + assert.equal(report.trust_stage_claimed, "advocacy"); + assert.equal(report.trust_stage, null); + const issue = report.issues.find((i) => i.type === "overclaimed_trust_stage")!; + assert.ok(issue, "overclaiming must be reported"); + assert.equal(issue.severity, "high"); + assert.match(issue.evidence, /claims aux\.T04/); + assert.match(issue.evidence, /supports no stage at all/); + assert.match(issue.evidence, /blocked by/); +}); + +test("under-claiming is reported, not punished", () => { + // The reference example declares aux.T02 and earns advocacy. Modesty is not + // a defect, so it is surfaced next to the computed stage and nothing more. + const report = v1strong(); + assert.equal(report.trust_stage_claimed, "contextual"); + assert.equal(report.trust_stage, "advocacy"); + assert.ok(!report.issues.some((i) => i.type === "overclaimed_trust_stage")); +}); + +test("a claim the evidence exactly supports raises nothing", () => { + const report = audit({ ...loadEvidence(fixture("v1-strong-spec.yaml")), claimedStage: "aux.T04" }); + assert.equal(report.trust_stage_claimed, "advocacy"); + assert.equal(report.trust_stage, "advocacy"); + assert.ok(!report.issues.some((i) => i.type === "overclaimed_trust_stage")); +}); + +test("claiming nothing is not a finding — v0 has no field for it", () => { + assert.equal(strong().trust_stage_claimed, null); + assert.ok(!strong().issues.some((i) => i.type === "overclaimed_trust_stage")); +}); + + diff --git a/packages/aux-audit/test/report.test.ts b/packages/aux-audit/test/report.test.ts index 374ee4f..46a9314 100644 --- a/packages/aux-audit/test/report.test.ts +++ b/packages/aux-audit/test/report.test.ts @@ -128,7 +128,7 @@ test("CLI: --config is honoured", () => { test("CLI: --version and --help exit 0", () => { assert.match( execFileSync(process.execPath, [cliPath(), "--version"], { encoding: "utf8" }), - /^0\.2\.0/, + /^0\.3\.0/, ); assert.match( execFileSync(process.execPath, [cliPath(), "--help"], { encoding: "utf8" }), diff --git a/packages/aux-audit/test/spec.test.ts b/packages/aux-audit/test/spec.test.ts index f46b870..8e66d4b 100644 --- a/packages/aux-audit/test/spec.test.ts +++ b/packages/aux-audit/test/spec.test.ts @@ -118,6 +118,7 @@ test("a v1 document is detected and scored, not rejected", () => { trigger: { kind: "schedule" }, systems: { data_sources: ["ledger"], connected: [{ name: "erp", auth: "service_account" }] }, capability: { can_read: ["invoices"], can_change: ["invoice_note"] }, + memory: { persistent: false }, mandate: [ { action: "invoice_note", authority: "autonomous", enforced_by: "scoped API token" }, ], @@ -136,8 +137,7 @@ test("a v1 document is detected and scored, not rejected", () => { assert.equal(evidence.name, "Collections Agent"); assert.equal(evidence.mandate?.length, 1); assert.equal(evidence.mandate?.[0]?.enforcementIsMechanism, true); - // v1 declares no memory, and the absence must not read as "no memory". - assert.equal(evidence.memory, undefined); + assert.equal(evidence.memory?.persistent, false); }); test("a v1 document that breaks the schema reports the real field", () => { diff --git a/schemas/MIGRATION.md b/schemas/MIGRATION.md index 5396618..89baec4 100644 --- a/schemas/MIGRATION.md +++ b/schemas/MIGRATION.md @@ -12,6 +12,21 @@ python3 schemas/validate.py your-agent.agent-spec.yaml --- +## What v1.0 requires that v0.1.0 did not + +`memory` is a required block. v0 carried one flat memory declaration; v1 asks +per scope — what is kept, for how long, whether the user can see it, correct +it, and have it forgotten. + +An agent that keeps nothing writes `memory: { persistent: false }`. Saying so +is the point: silence and "none" are different claims, and only one of them is +checkable. The shape follows +[`memory-policy.schema.yaml`](memory-policy.schema.yaml), so nothing here is +new vocabulary. + +`aux-audit` scores it. Without a memory block a spec cannot reach Contextual +Trust, because the only heuristic backing that stage has nothing to read. + ## Why v1.0 exists v0.1.0 was written as an input format for `aux-audit`, before the Agent Owner's diff --git a/schemas/agent-spec.schema.yaml b/schemas/agent-spec.schema.yaml index 38333be..b46eefd 100644 --- a/schemas/agent-spec.schema.yaml +++ b/schemas/agent-spec.schema.yaml @@ -30,6 +30,7 @@ required: - trigger - systems - capability + - memory - mandate - human_control - exceptions @@ -152,6 +153,64 @@ properties: type: array items: { type: string } + memory: + type: object + description: >- + What the agent retains between runs. Required: an agent with no memory + declares `persistent: false` rather than staying silent, because silence + and "none" are different claims and only one of them is checkable. + Shape follows memory-policy.schema.yaml. + additionalProperties: false + required: + - persistent + properties: + persistent: + type: boolean + description: Does anything survive the end of a run? + scopes: + type: array + description: >- + One entry per category of retained content. Required when persistent + is true — "we remember things" is not a memory policy. + items: + type: object + additionalProperties: false + required: + - name + - retention + - user_visible + - user_editable + properties: + name: + type: string + description: The category, e.g. preferences, conversation_history. + purpose: + type: string + retention: + type: string + pattern: "^P(?!$)(\\d+Y)?(\\d+M)?(\\d+W)?(\\d+D)?(T(?!$)(\\d+H)?(\\d+M)?(\\d+S)?)?$" + description: ISO-8601 duration. P90D, P1Y. "Indefinite" is a decision, so write it as one. + user_visible: + type: boolean + user_editable: + type: boolean + lawful_basis: + type: string + enum: [consent, contract, legitimate_interest, legal_obligation, not_applicable] + forget: + type: object + description: >- + Deletion on request. A retention period with no forget mechanism is a + promise about the calendar, not a control the user holds. + additionalProperties: false + required: + - available + properties: + available: + type: boolean + mechanism: + type: string + mandate: type: array description: >-