diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 69bef75..c9fc4ae 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -48,7 +48,7 @@ Every other artifact — audit rules, teardown scores, memory policies, Trust Co │ aux-audit │ │ 1. validate spec against agent-spec.schema.yaml │ │ 2. run heuristic rules (aux-heuristics.yaml) │ - │ 3. place on trust + evolution axes │ + │ 3. place on the trust axis (capability: see #5) │ │ 4. classify gaps (trust-gap-taxonomy.yaml) │ │ 5. emit Markdown + JSON + SARIF │ └────────────────────────────────────────────────────────┘ diff --git a/CHANGELOG.md b/CHANGELOG.md index 0db3f8f..7138dac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,7 @@ Publishing the GitHub release and tagging `v0.2` is a manual step after merge. ### Notes -- `evolution_stage` is reported as `schema-undefined`. The Evolution Curve is - referenced in docs but has no schema; the CLI will not mint `aux.` IDs. +- `evolution_stage` is reported as `human-assessed` and is always `null`. The + capability axis is scored by a human in a teardown, not derived from a spec + ([#5](https://github.com/auxfirst/trustkit/issues/5)). - A spec with no evaluation transcripts is capped at heuristic level 2. diff --git a/README.md b/README.md index 15ab77c..1d5af90 100644 --- a/README.md +++ b/README.md @@ -163,7 +163,7 @@ See **[LICENSE](LICENSE)** for the CC BY 4.0 legal code and **[LICENSING.md](LIC ## Status -**v0.2 · 2026-04-19** — First public release. Onboarding layer live: audience-routed docs (R1–R4) + four forwardable artifacts + two canonical patterns + six first-wave repo READMEs + canonical schemas. `aux-audit` v0.1 is implemented, tested, and wired into CI as a GitHub Action. Next: `npm publish` so `npx aux-audit` resolves without a clone; list the Action on the Marketplace; publish a schema for the Evolution Curve, which `aux-audit` currently reports as `schema-undefined`. +**v0.2 · 2026-04-19** — First public release. Onboarding layer live: audience-routed docs (R1–R4) + four forwardable artifacts + two canonical patterns + six first-wave repo READMEs + canonical schemas. `aux-audit` v0.1 is implemented, tested, and wired into CI as a GitHub Action. Seven patterns close all 12 named trust gaps, enforced in CI. Next: `npm publish` so `npx aux-audit` resolves without a clone; list the Action on the Marketplace; name the Evolution Curve stages for teardowns ([#5](https://github.com/auxfirst/trustkit/issues/5) — the axis is human-assessed, not something `aux-audit` computes). --- diff --git a/docs/audit.md b/docs/audit.md index 471b8f5..001f6a2 100644 --- a/docs/audit.md +++ b/docs/audit.md @@ -22,8 +22,9 @@ npx aux-audit run ./agent-spec.yaml - **Exit codes:** `0` clean · `1` findings at or above `--fail-on` · `2` invalid spec (nothing graded). The audit grades the **spec**, not the running product. Level 3 (“robust”) is -unreachable without evaluation transcripts. `evolution_stage` stays `null` until -an Evolution Curve schema is published. +unreachable without evaluation transcripts. `evolution_stage` is always `null`: +capability is what a product *does*, which a spec cannot state, so it is +assessed by a human in a teardown ([#5](https://github.com/auxfirst/trustkit/issues/5)). ## GitHub Action diff --git a/docs/releases/v0.2.md b/docs/releases/v0.2.md index 1404db3..bbbca5e 100644 --- a/docs/releases/v0.2.md +++ b/docs/releases/v0.2.md @@ -19,7 +19,7 @@ TrustKit is now a **runnable standard**, not only a set of YAML files. ## What it is not yet -- The Evolution Curve has no schema. `evolution_stage` is `null`. +- The Evolution Curve stage names are still open ([#5](https://github.com/auxfirst/trustkit/issues/5)). `evolution_stage` is `null` in `aux-audit` by design — it belongs to teardowns, not to the CLI. - The Action is not listed on the Marketplace until you submit it. - `npx aux-audit` without a local checkout needs `npm publish` from `packages/aux-audit`. diff --git a/onboarding/appendix-a-vocabulary.md b/onboarding/appendix-a-vocabulary.md index 51be9ae..c7abe00 100644 --- a/onboarding/appendix-a-vocabulary.md +++ b/onboarding/appendix-a-vocabulary.md @@ -33,5 +33,5 @@ Two tables. One for GitHub words (useful to R1 and R4 if they venture there). On | **Memory policy** | Declared scopes, retention, and user controls for persistent memory. | [schemas/memory-policy.schema.yaml](../schemas/memory-policy.schema.yaml) | | **Pattern** | A named, reusable design move that closes a specific gap. | [repos/agentic-ux-patterns](../repos/agentic-ux-patterns/README.md) | | **Teardown** | A structured review of a shipped product: `analysis.md` + `score.json`. | [repos/agent-ux-teardowns](../repos/agent-ux-teardowns/README.md) | -| **Evolution curve** | The capability axis, independent of trust. **No schema yet** — `aux-audit` reports `schema-undefined` rather than inventing stages. | referenced from [schemas/trust-architecture.yaml](../schemas/trust-architecture.yaml); tracked as a schema Issue | +| **Evolution curve** | The capability axis. Assessed by a human in a teardown, never computed from a spec — a spec says what a product claims, capability is what it does. Stage names are still open. | [trustkit#5](https://github.com/auxfirst/trustkit/issues/5); home is [repos/agent-ux-teardowns](../repos/agent-ux-teardowns) | | **Definition / Executable / Reference** | The three kinds of repo. Every repo is exactly one. | [REPO_STRUCTURE.md](../REPO_STRUCTURE.md) | diff --git a/packages/aux-audit/README.md b/packages/aux-audit/README.md index fd0af62..663c1d7 100644 --- a/packages/aux-audit/README.md +++ b/packages/aux-audit/README.md @@ -35,7 +35,7 @@ Change the YAML in `schemas/`, and the audit changes with it. No heuristic text, Two consequences, both deliberate: - **Level 3 requires evidence.** "Robust" means *works under novel, adversarial, or edge cases* — prose cannot demonstrate that. A spec with no `evaluation.golden_transcripts` or `evaluation.failure_transcripts` is capped at level 2 on every heuristic, however well written. This is what stops the score from being gamed by rewriting guarantees. -- **`evolution_stage` is `null`.** The AUX docs reference an Evolution Curve as the capability axis, but no schema publishes its stages. Rather than invent IDs in the `aux.` namespace, the field reports `evolution_stage_status: "schema-undefined"` and stays empty until the curve ships. See [issue tracker](https://github.com/auxfirst/trustkit/issues). +- **`evolution_stage` is always `null` — by design, not pending.** A spec states what a product *claims about itself*; capability is a claim about what it *does*. No schema turns a declaration into an observation, so the Evolution Curve is assessed by a human in [`agent-ux-teardowns`](../../repos/agent-ux-teardowns) and this tool reports `evolution_stage_status: "human-assessed"`. Decided in [trustkit#5](https://github.com/auxfirst/trustkit/issues/5). ## Install diff --git a/packages/aux-audit/src/report/markdown.ts b/packages/aux-audit/src/report/markdown.ts index dbead8a..5c6a592 100644 --- a/packages/aux-audit/src/report/markdown.ts +++ b/packages/aux-audit/src/report/markdown.ts @@ -86,7 +86,7 @@ export function toMarkdown(report: AuditReport): string { lines.push( `aux-audit ${report.meta.tool_version} · heuristics ${report.meta.heuristics_version} · trust architecture ${report.meta.trust_architecture_version} · taxonomy ${report.meta.taxonomy_version}. ` + `Scores describe the **spec**, not the running product: aux-audit can prove a mechanism was never declared, never that a declared one works. ` + - `Capability axis (\`evolution_stage\`) is omitted — the Evolution Curve has no published schema. ` + + `Capability axis (\`evolution_stage\`) is not scored here by design: a spec states what a product claims, not what it does. ` + `[The standard](https://github.com/auxfirst/trustkit) · [auxfirst.com](https://auxfirst.com)`, ); lines.push(""); diff --git a/packages/aux-audit/src/score.ts b/packages/aux-audit/src/score.ts index 6e786e9..6fb44a1 100644 --- a/packages/aux-audit/src/score.ts +++ b/packages/aux-audit/src/score.ts @@ -202,7 +202,7 @@ export function audit(spec: AgentSpec, options: AuditOptions = {}): AuditReport grade: grade(score), trust_stage, evolution_stage: null, - evolution_stage_status: "schema-undefined", + evolution_stage_status: "human-assessed", heuristics, trust_stages, issues, diff --git a/packages/aux-audit/src/types.ts b/packages/aux-audit/src/types.ts index 54b468a..bd03cee 100644 --- a/packages/aux-audit/src/types.ts +++ b/packages/aux-audit/src/types.ts @@ -70,12 +70,17 @@ export interface AuditReport { grade: Grade; trust_stage: string | null; /** - * Null until the Evolution Curve is published as a schema. The capability - * axis is referenced by the AUX docs but has no canonical stage list, and - * aux-audit will not invent IDs in the `aux.` namespace. + * Always null, by design — not pending. + * + * A spec describes what a product *declares about itself*; capability is a + * claim about what it actually does. No schema turns a declaration into an + * observation, so the Evolution Curve is assessed by a human in + * `agent-ux-teardowns` and is out of scope for this tool permanently. + * + * Decided in auxfirst/trustkit#5 (option B). */ evolution_stage: null; - evolution_stage_status: "schema-undefined"; + evolution_stage_status: "human-assessed"; heuristics: HeuristicResult[]; trust_stages: TrustStageResult[]; issues: Issue[]; diff --git a/packages/aux-audit/test/audit.test.ts b/packages/aux-audit/test/audit.test.ts index d1864ae..acac779 100644 --- a/packages/aux-audit/test/audit.test.ts +++ b/packages/aux-audit/test/audit.test.ts @@ -121,10 +121,13 @@ test("the report is deterministic for the same spec", () => { assert.deepEqual(audit(spec, { now }), audit(spec, { now })); }); -test("evolution_stage stays null while the Evolution Curve has no schema", () => { - const report = strong(); - assert.equal(report.evolution_stage, null); - assert.equal(report.evolution_stage_status, "schema-undefined"); +test("evolution_stage is permanently out of scope, not pending", () => { + // trustkit#5 option B: capability is assessed by a human in teardowns, + // never computed from a spec. The status must not read as future work. + for (const report of [strong(), weak()]) { + assert.equal(report.evolution_stage, null); + assert.equal(report.evolution_stage_status, "human-assessed"); + } }); test("level 3 requires evidence — a spec with no transcripts is capped at present", () => { diff --git a/repos/agent-ux-teardowns/README.md b/repos/agent-ux-teardowns/README.md index 1a64878..4ec6470 100644 --- a/repos/agent-ux-teardowns/README.md +++ b/repos/agent-ux-teardowns/README.md @@ -31,7 +31,8 @@ Every `analysis.md`: 3. **Where the contract breaks** — named failures from [`trust-gap-classifier`](../trust-gap-classifier). 4. **Heuristic scorecard** — 10 heuristics, scored 0–3, with evidence. 5. **Trust stage** — placed on the Trust Architecture ladder. -6. **The fix** — one concrete pattern that would move it up a stage. +6. **Evolution stage** — placed on the capability axis. This is the *only* place the capability axis is assessed: `aux-audit` reads a spec, and a spec states what a product claims rather than what it does. Stage names are still open — see [trustkit#5](https://github.com/auxfirst/trustkit/issues/5) — so `evolution_stage` stays `null` in teardowns until they land. +7. **The fix** — one concrete pattern that would move it up a stage. Every `score.json`: @@ -40,7 +41,7 @@ Every `score.json`: "product": "ChatGPT", "version": "4o, 2026-04", "trust_stage": "contextual", - "evolution_stage": "personally-intelligent", + "evolution_stage": null, "heuristics": { "aux.H01": 2, "aux.H02": 1, "aux.H03": 3, "aux.H04": 2, "aux.H05": 1, "aux.H06": 2, diff --git a/repos/aux-audit/README.md b/repos/aux-audit/README.md index f17d31b..ac4f62c 100644 --- a/repos/aux-audit/README.md +++ b/repos/aux-audit/README.md @@ -18,7 +18,7 @@ npx aux-audit run ./agent-spec.yaml score: 72 grade: B trust_stage: contextual -evolution_stage: null # schema-undefined until the Evolution Curve ships +evolution_stage: null # human-assessed in teardowns, never computed here issues: - id: aux.H01 type: visibility_of_intent @@ -41,7 +41,7 @@ recommendations: 1. **Parses** an agent spec (JSON/YAML describing the product: surface, memory model, tool use, autonomy model, failure handling). 2. **Runs** the rule set from [`aux-frameworks`](../aux-frameworks) against it. -3. **Scores** trust maturity from the Trust Architecture. Capability (`evolution_stage`) is `null` until an Evolution Curve schema is published. +3. **Scores** trust maturity from the Trust Architecture. Capability (`evolution_stage`) is always `null` here — it is assessed by a human in [`agent-ux-teardowns`](../agent-ux-teardowns), per [trustkit#5](https://github.com/auxfirst/trustkit/issues/5). 4. **Emits** a scored report as Markdown, JSON, or SARIF (for CI integration). ## Why CLI-first diff --git a/repos/aux-frameworks/README.md b/repos/aux-frameworks/README.md index 88793d7..1b0a82f 100644 --- a/repos/aux-frameworks/README.md +++ b/repos/aux-frameworks/README.md @@ -14,9 +14,6 @@ /trust-architecture # 4-stage trust maturity ladder schema.yaml stages.yaml - /aux-evolution-curve # capability maturity axis - schema.yaml - stages.yaml /aux-audit # audit rules derived from the above schema.yaml rules.yaml