Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,21 @@ Publishing the GitHub release and tagging `v0.2` is a manual step after merge.
- `CITATION.cff` for GitHub’s *Cite this repository* button.
- Project-home links to [auxfirst.com](https://auxfirst.com).

## aux-audit 0.2.0

- Scores **agent-spec v1.0**. Both formats are read, detected from the document
rather than the filename, and `meta.spec_version` records which produced a
score — they are not comparable.
- Under v1 the rules ask a better question. `aux.H01` and `aux.H03` read
per-action authority and the five human controls instead of one label;
`aux.H05` computes the gap between what the credentials permit and what the
mandate governs; a row enforced by "the system prompt" counts as unenforced.
- `aux.H08` reports *not scoreable* under v1, which has no memory field, rather
than scoring zero. A trust stage whose entire backing is unscoreable is now
reported **not assessable** and is never counted as earned — previously an
empty shortfall read as success on zero evidence.
- v0.1.0 scoring is unchanged.

### Notes

- `agent-spec.schema.yaml` is now v1.0: real JSON Schema draft 2020-12, with a
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ In CI, as a GitHub Action — SARIF into the Security tab, a scorecard on the PR
fail-on: high
```

It grades the **spec**, not the running product: it can prove a mechanism was never declared, never that a declared one works. A spec with no evaluation transcripts is capped at *present* on every heuristic, so the score cannot be gamed with better prose. See **[packages/aux-audit](packages/aux-audit/)**.
It grades the **spec**, not the running product: it can prove a mechanism was never declared, never that a declared one works. A spec with no evidence of testing is capped at *present* on every heuristic, so the score cannot be gamed with better prose.

Under agent-spec v1 it computes the gap between what an agent's credentials permit and what its mandate governs — the attack surface — and treats a row enforced by "the system prompt" as unenforced. See **[packages/aux-audit](packages/aux-audit/)**.

## Start here

Expand Down Expand Up @@ -80,7 +82,7 @@ Or read **[onboarding/README.md](onboarding/README.md)** — the router with all
│ ├── aux-evolution-curve.yaml ← the 4 capability stages (teardowns only)
│ ├── trust-contract.yaml ← example Trust Contract
│ ├── agent-spec.schema.yaml ← v1.0, real JSON Schema — per-action mandate
│ ├── agent-spec.v0.yaml ← v0.1.0, frozen; what aux-audit 0.1.x reads
│ ├── agent-spec.v0.yaml ← v0.1.0, frozen; still scored, not comparable
│ ├── MIGRATION.md ← v0 → v1, and why
│ ├── migrate-v0-to-v1.py ← converter; leaves TODOs rather than guessing
│ ├── brain-spec.schema.yaml ← Agency Brain input format (JSON Schema)
Expand Down
2 changes: 1 addition & 1 deletion docs/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ npx aux-audit run ./agent-spec.yaml

## Contract

- **Input:** an agent spec matching [`schemas/agent-spec.v0.yaml`](../schemas/agent-spec.v0.yaml). The canonical [`agent-spec.schema.yaml`](../schemas/agent-spec.schema.yaml) is now v1.0; aux-audit 0.1.x does not score it yet and says so plainly rather than reporting missing fields — see [MIGRATION.md](../schemas/MIGRATION.md) and [#10](https://github.com/auxfirst/trustkit/issues/10).
- **Input:** an agent spec in either format — [`agent-spec.schema.yaml`](../schemas/agent-spec.schema.yaml) v1.0 or [`agent-spec.v0.yaml`](../schemas/agent-spec.v0.yaml). The version is detected from the document, not the filename, and `meta.spec_version` records which one was scored. Scores are not comparable across the two; see [MIGRATION.md](../schemas/MIGRATION.md).
- **Output:** `score`, `grade`, `trust_stage`, `issues[]`, `recommendations[]`.
- **Formats:** Markdown (default), JSON, SARIF 2.1.0.
- **Exit codes:** `0` clean · `1` findings at or above `--fail-on` · `2` invalid spec (nothing graded).
Expand Down
8 changes: 6 additions & 2 deletions packages/aux-audit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,13 @@ Change the YAML in `schemas/`, and the audit changes with it. No heuristic text,

**aux-audit grades the spec, not the running product.** It can prove a mechanism was never declared. It cannot prove a declared one works.

There is a sharper limit in the format this version reads, tracked in [trustkit#10](https://github.com/auxfirst/trustkit/issues/10): `agent-spec.v0.yaml` carries a single per-agent `autonomy` label and free-text `guarantees`, so a score reflects a **declared posture**, not an enforced boundary.
**Both spec versions are scored**, detected from the document rather than the filename.

The replacement, `agent-spec.schema.yaml` v1.0, is published and assigns authority action by action with a required `enforced_by` on every non-autonomous row. **aux-audit 0.1.x does not score it.** Hand it a v1 document and it says so in one line and exits 2, rather than reporting the fields v1 removed on purpose as missing. v1 scoring lands in 0.2.0.
Under **v1.0** the audit asks a better question. Instead of *"what did you call your autonomy level"*, `aux.H01` and `aux.H03` read per-action authority and the five human controls, and `aux.H05` computes the **gap between what the credentials permit and what the mandate governs** — the attack surface, in the schema's own words. A row enforced by "the system prompt" is reported as unenforced, because the canon is explicit that a prompt is a request and a tool boundary is a control.

Under **v0.1.0** nothing changed: the same label and the same prose guarantees produce the same scores as before. Scores are not comparable across the two, so `meta.spec_version` records which one produced them.

One thing v1 cannot express: **memory**. It has no field for persistence, scoping, or retention, so `aux.H08` is reported as not scoreable rather than scored zero — and `aux.T02` Contextual Trust, whose only backing heuristic that is, comes back *not assessable* rather than earned. See [trustkit#10](https://github.com/auxfirst/trustkit/issues/10).

Two consequences, both deliberate:

Expand Down
212 changes: 212 additions & 0 deletions packages/aux-audit/fixtures/v1-strong-spec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
# Example agent spec — validates against agent-spec.schema.yaml
#
# A realistic, deliberately unglamorous agent: chasing overdue invoices.
# Note what the spec forces into the open — the send action is HIGH band on
# exposure and commitment alone, so it cannot be autonomous no matter how
# reliable the agent has proved to be on the read actions.
#
# Validate with: python3 schemas/validate.py schemas/example-agent-spec.yaml
#
# CC BY 4.0 · auxfirst agency 2026

spec_version: "1.0"
id: ar-followup
name: Accounts Receivable Follow-Up
purpose: >-
Chase overdue invoices on a schedule so finance staff spend their time only
on disputed and exceptional cases.

owners:
business:
name: Anna Nowak
role: Head of Finance Operations
contact: anna.nowak@example.com
technical:
name: Piotr Lis
role: Platform Engineer
contact: piotr.lis@example.com

trigger:
kind: schedule
detail: Weekdays 07:00 Europe/Warsaw

users:
- role: Accounts receivable clerk
can_invoke: true
can_approve: true
- role: Account executive
can_invoke: false
can_approve: true

systems:
data_sources:
- Billing database (invoices, payment status)
- CRM (contact of record, account owner)
connected:
- name: Billing service
auth: own_identity
scopes: [invoice:read, invoice:note:write]
- name: Mail gateway
auth: own_identity
scopes: [draft:write] # send scope deliberately withheld
- name: CRM
auth: delegated_identity
scopes: [contact:read]

# What the credentials permit. Compare against mandate below —
# the difference is the attack surface.
capability:
can_read:
- invoices
- payment status
- contacts of record
can_change:
- invoice internal note
- email drafts

mandate:
- action: Read invoice and payment status
authority: autonomous
enforced_by: Scoped read token, invoice:read only
heat: { reversibility: 0, blast_radius: 0, exposure: 0, commitment: 0, authority: 0, band: LOW }
autonomy_level: L1

- action: Identify overdue invoices and rank by age
authority: autonomous
enforced_by: Read-only query path, no write scope
heat: { reversibility: 0, blast_radius: 0, exposure: 0, commitment: 0, authority: 0, band: LOW }
autonomy_level: L1

- action: Write an internal note on the invoice
authority: autonomous
enforced_by: Field-scoped write API, note field only
heat: { reversibility: 0, blast_radius: 0, exposure: 0, commitment: 0, authority: 1, band: LOW }
autonomy_level: L4

- action: Draft a reminder email
authority: autonomous
enforced_by: Mail gateway draft scope; send scope not granted
heat: { reversibility: 0, blast_radius: 0, exposure: 0, commitment: 0, authority: 1, band: LOW }
autonomy_level: L3

# The hinge. Three dials are cool and the action is still HIGH,
# because it leaves the building and asks for money.
- action: Send a reminder email to the customer
authority: human_approval
enforced_by: Send scope withheld until an approval token is issued
approver: Account executive on the account
heat: { reversibility: 1, blast_radius: 0, exposure: 3, commitment: 3, authority: 1, band: HIGH }
autonomy_level: L3

- action: Apply a late-payment fee
authority: human_only
enforced_by: Agent has no write path to the fee service
approver: Head of Finance Operations
heat: { reversibility: 2, blast_radius: 0, exposure: 3, commitment: 4, authority: 2, band: CRITICAL }
autonomy_level: L2

- action: Escalate to a collections agency
authority: prohibited
enforced_by: Integration not provisioned for this agent identity
heat: { reversibility: 4, blast_radius: 1, exposure: 4, commitment: 4, authority: 3, band: CRITICAL }

human_control:
observe:
available: true
mechanism: Per-run log with the invoices touched and drafts produced
audience: affected_user
interrupt:
available: true
mechanism: Pause the approval queue; in-flight drafts are held
audience: operator
approve:
available: true
mechanism: Approval queue, one entry per outbound email
audience: operator
override:
available: true
mechanism: Edit the draft before approving, or reject with a reason
audience: operator
disable:
available: true
mechanism: Feature flag plus token revocation
audience: admin_only

exceptions:
- condition: identifier_missing
detail: No contact of record on the account
response: route
escalation:
recipient: Anna Nowak
deadline_minutes: 480
on_timeout: stop
context_preserved: true

- condition: records_conflict
detail: Billing and CRM disagree on payment status
response: ask_human
escalation:
recipient: Anna Nowak
deadline_minutes: 240
on_timeout: stop
context_preserved: true

- condition: result_disputed
detail: Customer replies contesting the invoice
response: route
escalation:
recipient: Account executive on the account
deadline_minutes: 120
on_timeout: page_secondary
context_preserved: true

- condition: tool_unavailable
detail: Billing service unreachable
response: stop

- condition: confidence_low
response: ask_human
escalation:
recipient: Anna Nowak
deadline_minutes: 480
on_timeout: stop
context_preserved: true

- condition: sensitive_data_present
detail: Bank details appear in a customer reply
response: refuse

escalation_default:
recipient: Anna Nowak
deadline_minutes: 480
on_timeout: stop
context_preserved: true

supervision:
agent_identity_disclosed: true
action_receipt: true
reversal: true
consequence_scaled_approval: true
escalation_handoff: true
provenance_at_decision: false # known gap — see note below

shutdown:
procedure: >-
Disable the feature flag, revoke the billing and mail gateway tokens, drain
the approval queue, and notify the two owners.
tested: true
last_tested: "2026-08-20"
revokes_access: true

model:
provider: anthropic
name: claude-sonnet-5
version_pinned: true

trust_stage: aux.T02
status: live

# Known gap, recorded rather than hidden: provenance_at_decision is false.
# The approver sees the draft but not the payment-history evidence behind the
# ranking. Tracked as tg.judgment.confident_nonsense risk; fix is the
# confidence-cues pattern on the approval queue.
36 changes: 36 additions & 0 deletions packages/aux-audit/fixtures/v1-weak-spec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# A v1 spec that satisfies the schema and still fails the doctrine: everything
# autonomous, enforcement by prompt, no supervision primitives, untested kill
# switch. v1 makes these visible where v0 could not express them at all.
spec_version: "1.0"
id: inbox-autopilot
name: Inbox Autopilot
purpose: Clear the shared inbox without human involvement.
owners:
business: { name: "Unassigned", role: "TBD" }
technical: { name: "Platform Team", role: "Engineering" }
trigger:
kind: continuous
systems:
data_sources: ["mailbox"]
connected:
- name: gmail
auth: own_identity
capability:
can_read: ["mailbox", "contacts", "calendar"]
can_change: ["send_email", "delete_email", "calendar_event"]
mandate:
- action: send_email
authority: autonomous
enforced_by: "the system prompt tells it to be careful"
human_control:
observe: { available: false }
interrupt: { available: false }
approve: { available: false }
override: { available: false }
disable: { available: true, mechanism: "revoke the OAuth token", audience: admin_only }
exceptions:
- condition: tool_unavailable
response: retry
shutdown:
procedure: "Revoke the OAuth token in the admin console."
tested: false
54 changes: 54 additions & 0 deletions packages/aux-audit/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading