Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Every other artifact — audit rules, teardown scores, memory policies, Trust Co
│ aux-audit │
│ 1. validate spec against agent-spec.schema.yaml │
│ 2. run heuristic rules (aux-heuristics.yaml) │
│ 3. place on trust + evolution axes
│ 3. place on the trust axis (capability: see #5)
│ 4. classify gaps (trust-gap-taxonomy.yaml) │
│ 5. emit Markdown + JSON + SARIF │
└────────────────────────────────────────────────────────┘
Expand Down
5 changes: 3 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Publishing the GitHub release and tagging `v0.2` is a manual step after merge.

### Notes

- `evolution_stage` is reported as `schema-undefined`. The Evolution Curve is
referenced in docs but has no schema; the CLI will not mint `aux.` IDs.
- `evolution_stage` is reported as `human-assessed` and is always `null`. The
capability axis is scored by a human in a teardown, not derived from a spec
([#5](https://github.com/auxfirst/trustkit/issues/5)).
- A spec with no evaluation transcripts is capped at heuristic level 2.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ See **[LICENSE](LICENSE)** for the CC BY 4.0 legal code and **[LICENSING.md](LIC

## Status

**v0.2 · 2026-04-19** — First public release. Onboarding layer live: audience-routed docs (R1–R4) + four forwardable artifacts + two canonical patterns + six first-wave repo READMEs + canonical schemas. `aux-audit` v0.1 is implemented, tested, and wired into CI as a GitHub Action. Next: `npm publish` so `npx aux-audit` resolves without a clone; list the Action on the Marketplace; publish a schema for the Evolution Curve, which `aux-audit` currently reports as `schema-undefined`.
**v0.2 · 2026-04-19** — First public release. Onboarding layer live: audience-routed docs (R1–R4) + four forwardable artifacts + two canonical patterns + six first-wave repo READMEs + canonical schemas. `aux-audit` v0.1 is implemented, tested, and wired into CI as a GitHub Action. Seven patterns close all 12 named trust gaps, enforced in CI. Next: `npm publish` so `npx aux-audit` resolves without a clone; list the Action on the Marketplace; name the Evolution Curve stages for teardowns ([#5](https://github.com/auxfirst/trustkit/issues/5) — the axis is human-assessed, not something `aux-audit` computes).

---

Expand Down
5 changes: 3 additions & 2 deletions docs/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ npx aux-audit run ./agent-spec.yaml
- **Exit codes:** `0` clean · `1` findings at or above `--fail-on` · `2` invalid spec (nothing graded).

The audit grades the **spec**, not the running product. Level 3 (“robust”) is
unreachable without evaluation transcripts. `evolution_stage` stays `null` until
an Evolution Curve schema is published.
unreachable without evaluation transcripts. `evolution_stage` is always `null`:
capability is what a product *does*, which a spec cannot state, so it is
assessed by a human in a teardown ([#5](https://github.com/auxfirst/trustkit/issues/5)).

## GitHub Action

Expand Down
2 changes: 1 addition & 1 deletion docs/releases/v0.2.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ TrustKit is now a **runnable standard**, not only a set of YAML files.

## What it is not yet

- The Evolution Curve has no schema. `evolution_stage` is `null`.
- The Evolution Curve stage names are still open ([#5](https://github.com/auxfirst/trustkit/issues/5)). `evolution_stage` is `null` in `aux-audit` by design — it belongs to teardowns, not to the CLI.
- The Action is not listed on the Marketplace until you submit it.
- `npx aux-audit` without a local checkout needs `npm publish` from
`packages/aux-audit`.
Expand Down
2 changes: 1 addition & 1 deletion onboarding/appendix-a-vocabulary.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,5 @@ Two tables. One for GitHub words (useful to R1 and R4 if they venture there). On
| **Memory policy** | Declared scopes, retention, and user controls for persistent memory. | [schemas/memory-policy.schema.yaml](../schemas/memory-policy.schema.yaml) |
| **Pattern** | A named, reusable design move that closes a specific gap. | [repos/agentic-ux-patterns](../repos/agentic-ux-patterns/README.md) |
| **Teardown** | A structured review of a shipped product: `analysis.md` + `score.json`. | [repos/agent-ux-teardowns](../repos/agent-ux-teardowns/README.md) |
| **Evolution curve** | The capability axis, independent of trust. **No schema yet** — `aux-audit` reports `schema-undefined` rather than inventing stages. | referenced from [schemas/trust-architecture.yaml](../schemas/trust-architecture.yaml); tracked as a schema Issue |
| **Evolution curve** | The capability axis. Assessed by a human in a teardown, never computed from a spec — a spec says what a product claims, capability is what it does. Stage names are still open. | [trustkit#5](https://github.com/auxfirst/trustkit/issues/5); home is [repos/agent-ux-teardowns](../repos/agent-ux-teardowns) |
| **Definition / Executable / Reference** | The three kinds of repo. Every repo is exactly one. | [REPO_STRUCTURE.md](../REPO_STRUCTURE.md) |
2 changes: 1 addition & 1 deletion packages/aux-audit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Change the YAML in `schemas/`, and the audit changes with it. No heuristic text,
Two consequences, both deliberate:

- **Level 3 requires evidence.** "Robust" means *works under novel, adversarial, or edge cases* — prose cannot demonstrate that. A spec with no `evaluation.golden_transcripts` or `evaluation.failure_transcripts` is capped at level 2 on every heuristic, however well written. This is what stops the score from being gamed by rewriting guarantees.
- **`evolution_stage` is `null`.** The AUX docs reference an Evolution Curve as the capability axis, but no schema publishes its stages. Rather than invent IDs in the `aux.` namespace, the field reports `evolution_stage_status: "schema-undefined"` and stays empty until the curve ships. See [issue tracker](https://github.com/auxfirst/trustkit/issues).
- **`evolution_stage` is always `null` — by design, not pending.** A spec states what a product *claims about itself*; capability is a claim about what it *does*. No schema turns a declaration into an observation, so the Evolution Curve is assessed by a human in [`agent-ux-teardowns`](../../repos/agent-ux-teardowns) and this tool reports `evolution_stage_status: "human-assessed"`. Decided in [trustkit#5](https://github.com/auxfirst/trustkit/issues/5).

## Install

Expand Down
2 changes: 1 addition & 1 deletion packages/aux-audit/src/report/markdown.ts
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ export function toMarkdown(report: AuditReport): string {
lines.push(
`<sub>aux-audit ${report.meta.tool_version} · heuristics ${report.meta.heuristics_version} · trust architecture ${report.meta.trust_architecture_version} · taxonomy ${report.meta.taxonomy_version}. ` +
`Scores describe the **spec**, not the running product: aux-audit can prove a mechanism was never declared, never that a declared one works. ` +
`Capability axis (\`evolution_stage\`) is omitted — the Evolution Curve has no published schema. ` +
`Capability axis (\`evolution_stage\`) is not scored here by design: a spec states what a product claims, not what it does. ` +
`[The standard](https://github.com/auxfirst/trustkit) · [auxfirst.com](https://auxfirst.com)</sub>`,
);
lines.push("");
Expand Down
2 changes: 1 addition & 1 deletion packages/aux-audit/src/score.ts
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ export function audit(spec: AgentSpec, options: AuditOptions = {}): AuditReport
grade: grade(score),
trust_stage,
evolution_stage: null,
evolution_stage_status: "schema-undefined",
evolution_stage_status: "human-assessed",
heuristics,
trust_stages,
issues,
Expand Down
13 changes: 9 additions & 4 deletions packages/aux-audit/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,12 +70,17 @@ export interface AuditReport {
grade: Grade;
trust_stage: string | null;
/**
* Null until the Evolution Curve is published as a schema. The capability
* axis is referenced by the AUX docs but has no canonical stage list, and
* aux-audit will not invent IDs in the `aux.` namespace.
* Always null, by design — not pending.
*
* A spec describes what a product *declares about itself*; capability is a
* claim about what it actually does. No schema turns a declaration into an
* observation, so the Evolution Curve is assessed by a human in
* `agent-ux-teardowns` and is out of scope for this tool permanently.
*
* Decided in auxfirst/trustkit#5 (option B).
*/
evolution_stage: null;
evolution_stage_status: "schema-undefined";
evolution_stage_status: "human-assessed";
heuristics: HeuristicResult[];
trust_stages: TrustStageResult[];
issues: Issue[];
Expand Down
11 changes: 7 additions & 4 deletions packages/aux-audit/test/audit.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,13 @@ test("the report is deterministic for the same spec", () => {
assert.deepEqual(audit(spec, { now }), audit(spec, { now }));
});

test("evolution_stage stays null while the Evolution Curve has no schema", () => {
const report = strong();
assert.equal(report.evolution_stage, null);
assert.equal(report.evolution_stage_status, "schema-undefined");
test("evolution_stage is permanently out of scope, not pending", () => {
// trustkit#5 option B: capability is assessed by a human in teardowns,
// never computed from a spec. The status must not read as future work.
for (const report of [strong(), weak()]) {
assert.equal(report.evolution_stage, null);
assert.equal(report.evolution_stage_status, "human-assessed");
}
});

test("level 3 requires evidence — a spec with no transcripts is capped at present", () => {
Expand Down
5 changes: 3 additions & 2 deletions repos/agent-ux-teardowns/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ Every `analysis.md`:
3. **Where the contract breaks** — named failures from [`trust-gap-classifier`](../trust-gap-classifier).
4. **Heuristic scorecard** — 10 heuristics, scored 0–3, with evidence.
5. **Trust stage** — placed on the Trust Architecture ladder.
6. **The fix** — one concrete pattern that would move it up a stage.
6. **Evolution stage** — placed on the capability axis. This is the *only* place the capability axis is assessed: `aux-audit` reads a spec, and a spec states what a product claims rather than what it does. Stage names are still open — see [trustkit#5](https://github.com/auxfirst/trustkit/issues/5) — so `evolution_stage` stays `null` in teardowns until they land.
7. **The fix** — one concrete pattern that would move it up a stage.

Every `score.json`:

Expand All @@ -40,7 +41,7 @@ Every `score.json`:
"product": "ChatGPT",
"version": "4o, 2026-04",
"trust_stage": "contextual",
"evolution_stage": "personally-intelligent",
"evolution_stage": null,
"heuristics": {
"aux.H01": 2, "aux.H02": 1, "aux.H03": 3,
"aux.H04": 2, "aux.H05": 1, "aux.H06": 2,
Expand Down
4 changes: 2 additions & 2 deletions repos/aux-audit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ npx aux-audit run ./agent-spec.yaml
score: 72
grade: B
trust_stage: contextual
evolution_stage: null # schema-undefined until the Evolution Curve ships
evolution_stage: null # human-assessed in teardowns, never computed here
issues:
- id: aux.H01
type: visibility_of_intent
Expand All @@ -41,7 +41,7 @@ recommendations:

1. **Parses** an agent spec (JSON/YAML describing the product: surface, memory model, tool use, autonomy model, failure handling).
2. **Runs** the rule set from [`aux-frameworks`](../aux-frameworks) against it.
3. **Scores** trust maturity from the Trust Architecture. Capability (`evolution_stage`) is `null` until an Evolution Curve schema is published.
3. **Scores** trust maturity from the Trust Architecture. Capability (`evolution_stage`) is always `null` here — it is assessed by a human in [`agent-ux-teardowns`](../agent-ux-teardowns), per [trustkit#5](https://github.com/auxfirst/trustkit/issues/5).
4. **Emits** a scored report as Markdown, JSON, or SARIF (for CI integration).

## Why CLI-first
Expand Down
3 changes: 0 additions & 3 deletions repos/aux-frameworks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@
/trust-architecture # 4-stage trust maturity ladder
schema.yaml
stages.yaml
/aux-evolution-curve # capability maturity axis
schema.yaml
stages.yaml
/aux-audit # audit rules derived from the above
schema.yaml
rules.yaml
Expand Down
Loading