Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
# - YAML syntax errors in /schemas/
# - broken internal links across onboarding/forwardables/repos
# - aux-audit failing its own test suite, or drifting from the canonical schemas
# - a taxonomy gap with no pattern, or a pattern missing one of its four files
#
# Local repro:
# yamllint schemas/ .github/
Expand All @@ -22,6 +23,7 @@ on:
- "profile/**"
- "docs/**"
- "packages/**"
- "repos/agentic-ux-patterns/**"
- "action.yml"
- "*.md"
- ".github/workflows/**"
Expand Down Expand Up @@ -110,3 +112,21 @@ jobs:
if node dist/cli.js run fixtures/invalid-spec.yaml > /dev/null 2>&1; then
echo "expected an invalid spec to be refused"; exit 1
fi

patterns:
name: pattern coverage
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- run: pip install --quiet pyyaml

# CONTRIBUTING says a pattern PR is rejected if any of the four files are
# missing, and the patterns README claims every named gap is closed.
# Both are executed here rather than trusted.
- name: Every gap closed, every pattern complete, every example runnable
run: python repos/agentic-ux-patterns/check-coverage.py
28 changes: 21 additions & 7 deletions repos/agentic-ux-patterns/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,35 @@ This is a **Reference** repo. Every pattern here:

## Patterns

| Slug | Closes | Heuristic |
|---|---|---|
| **[intent-handshake](./patterns/intent-handshake)** | `tg.judgment.overreach_in_ambiguity` | `aux.H01` |
| **[memory-in-motion](./patterns/memory-in-motion)** | `tg.contextual.memory_amnesia` | `aux.H08` |
Seven patterns close all **12 named gaps** in [`trust-gap-taxonomy.yaml`](../../schemas/trust-gap-taxonomy.yaml). Coverage is enforced in CI by [`check-coverage.py`](./check-coverage.py) — adding a gap without a pattern fails the build.

| Pattern | Closes | Heuristic | Trust stage |
|---|---|---|---|
| **[intent-handshake](./patterns/intent-handshake)** | `tg.judgment.overreach_in_ambiguity` | `aux.H01` | `aux.T03` |
| **[memory-in-motion](./patterns/memory-in-motion)** | `tg.contextual.memory_amnesia`<br>`tg.contextual.preference_ignored` | `aux.H08` | `aux.T02` |
| **[confidence-cues](./patterns/confidence-cues)** | `tg.functional.hallucination`<br>`tg.judgment.confident_nonsense` | `aux.H06` | `aux.T01`, `aux.T03` |
| **[behavioral-contract](./patterns/behavioral-contract)** | `tg.functional.silent_degradation`<br>`tg.functional.inconsistent_output` | `aux.H10` | `aux.T01` |
| **[memory-policy-scoping](./patterns/memory-policy-scoping)** | `tg.contextual.context_leak` | `aux.H08` | `aux.T02` |
| **[escalation-handoff](./patterns/escalation-handoff)** | `tg.judgment.refusal_when_escalation_needed` | `aux.H07` | `aux.T03` |
| **[user-aligned-objective](./patterns/user-aligned-objective)** | `tg.advocacy.metric_over_user`<br>`tg.advocacy.incentive_misalignment`<br>`tg.advocacy.loyalty_leak` | `aux.H07` | `aux.T04` |

Every pattern ships four files. `example.py` is runnable as-is: `python patterns/<slug>/example.py`.

## Pattern template

```
/patterns/<slug>/
├── README.md # what, why, when, how
├── diagram.svg # one diagram
├── example.{py,ts,...} # runnable
├── README.md # what, why, when, how — embeds the diagram
├── diagram.svg # one diagram, light/dark aware
├── example.py # runnable: python example.py
└── anti-pattern.md # what this is not
```

The anti-pattern is not decoration. Every one of these failure modes ships in
real products *because it looks like the pattern* — hedging every sentence
looks like calibration, a snapshot test looks like a contract, a `user_id`
filter looks like scoping. Naming the near-miss is most of the work.

## Contributing

See [`CONTRIBUTING.md`](../../CONTRIBUTING.md). A pattern PR is rejected if any of the four files are missing.
Expand Down
77 changes: 77 additions & 0 deletions repos/agentic-ux-patterns/check-coverage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#!/usr/bin/env python3
"""Enforce the two claims this repo makes about itself.

1. Every named gap in trust-gap-taxonomy.yaml has a pattern that closes it.
2. Every pattern ships all four files, and example.py actually runs.

CONTRIBUTING says "a pattern PR is rejected if any of the four files are
missing". This is that rule, executed rather than asserted.

Usage: python repos/agentic-ux-patterns/check-coverage.py
"""
import subprocess
import sys
from pathlib import Path

import yaml

HERE = Path(__file__).resolve().parent
REPO = HERE.parent.parent
PATTERNS = HERE / "patterns"
REQUIRED = ("README.md", "diagram.svg", "example.py", "anti-pattern.md")


def main() -> int:
taxonomy = yaml.safe_load((REPO / "schemas" / "trust-gap-taxonomy.yaml").read_text())
gaps = taxonomy["gaps"]
present = {d.name for d in PATTERNS.iterdir() if d.is_dir()}
problems: list[str] = []

# 1. coverage
uncovered = [g for g in gaps if g["fix_pattern"] not in present]
for gap in uncovered:
problems.append(
f"gap {gap['id']} names fix_pattern '{gap['fix_pattern']}', "
f"which has no folder under patterns/"
)

# An orphan is not an error — a pattern may exist before a gap names it —
# but it should be deliberate, so say so.
named = {g["fix_pattern"] for g in gaps}
orphans = sorted(present - named)

# 2. completeness and runnability
for slug in sorted(present):
for filename in REQUIRED:
if not (PATTERNS / slug / filename).exists():
problems.append(f"pattern '{slug}' is missing {filename}")

example = PATTERNS / slug / "example.py"
if example.exists():
run = subprocess.run(
[sys.executable, example.name],
cwd=example.parent,
capture_output=True,
text=True,
timeout=30,
)
if run.returncode != 0:
tail = (run.stderr or run.stdout).strip().splitlines()[-1:]
problems.append(f"pattern '{slug}': example.py exited {run.returncode} — {tail}")

covered = len(gaps) - len(uncovered)
print(f"gaps covered: {covered}/{len(gaps)} patterns: {len(present)}")
if orphans:
print(f"patterns not named by any gap (fine, but deliberate?): {', '.join(orphans)}")

if problems:
print("\nFAILED:")
for problem in problems:
print(f" - {problem}")
return 1
print("all gaps closed, all patterns complete, all examples run")
return 0


if __name__ == "__main__":
sys.exit(main())
77 changes: 77 additions & 0 deletions repos/agentic-ux-patterns/patterns/behavioral-contract/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Pattern · Behavioral Contract

> The agent's behaviour is pinned by golden cases and a version. When it changes, the change is announced — never discovered.

![A pin change re-runs the golden cases; unacknowledged drift blocks the release, acknowledged drift becomes a changelog entry.](./diagram.svg)

- **Heuristic:** `aux.H10` Consistency of Behavior
- **Closes gaps:** `tg.functional.silent_degradation`, `tg.functional.inconsistent_output`
- **Trust stage:** `aux.T01` Functional

## What it is

A versioned artifact that travels with the agent and states what will not change without notice:

1. **Golden cases.** A small set of inputs with expected-behaviour assertions — not exact strings, but the properties that must hold ("escalates", "cites a source", "does not send email").
2. **A pin.** Model version, prompt version, tool versions. Anything that can move behaviour is named.
3. **A gate.** Changing the pin re-runs the golden cases. Drift does not block the release — it blocks a *silent* release. The gate's output is a changelog entry.

## Why it works

Functional trust is the claim "same input, same output, across sessions." Every agent breaks that claim eventually: a model is deprecated, a prompt is tuned, a tool changes its response shape. The trust-destroying part is not the change — it's that the user finds out by being burned.

A contract converts an invisible regression into a visible release note. That is a much smaller promise than "we will never change", and unlike that promise, it can be kept.

## When to use it

- Any agent behind a model you do not control the release schedule of.
- Any behaviour a user has been told to rely on ("it always asks before sending").
- Any agent whose output feeds another system.

## When NOT to use it

- Prototypes with no users. A contract with no one on the other side is ceremony.
- Assertions on exact wording. Pinning prose makes the suite fail on every harmless rephrase, and a suite that cries wolf gets deleted. That's the **[anti-pattern](./anti-pattern.md)**.

## Minimal implementation

```python
def check(contract: Contract, agent: Agent) -> list[Drift]:
drift = []
for case in contract.golden:
result = agent.run(case.given)
for prop, expected in case.must.items():
holds = PROPERTIES[prop]
if holds(result) != expected:
drift.append(Drift(case.name, prop, expected))
return drift


def release(contract, agent, new_pin) -> str:
drift = check(contract, agent)
if drift and not contract.acknowledged(drift):
raise SilentChangeBlocked(drift) # ship it, but say so
return changelog_entry(contract.pin, new_pin, drift)
```

## What to assert

| Assert on | Don't assert on |
|---|---|
| "escalates to a human" | the exact escalation wording |
| "cites at least one source" | which source it picked |
| "does not call `email.send`" | the order of the other tool calls |
| "refuses" / "asks before acting" | the phrasing of the refusal |
| output parses as the declared schema | field ordering |

The rule: **assert on the properties you promised the user, not on the prose you happened to ship.** A contract that fails when nothing a user would notice has changed will be disabled within a month, and then the real regression ships unannounced.

## Relationship to `aux-audit`

`aux.H10` reaches *present* only when a spec declares `evaluation.golden_transcripts`. That field is this contract's golden cases, pointed at from the spec — the audit checks that a contract exists; this pattern is what goes in it.

## Anti-pattern

See **[anti-pattern.md](./anti-pattern.md)**.

TL;DR: a snapshot test on the exact output string is not a behavioral contract. It's a tripwire that fires on rephrases and stays silent on regressions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Anti-pattern · Snapshot-testing the prose

What it looks like:

```python
def test_refund_flow():
out = agent.run("customer wants a refund on order 4831")
assert out == (
"I've looked up order #4831. I'll draft a refund email for your "
"review. I won't send anything without your approval."
)
```

Six weeks later the suite is red on every run, someone adds `--update-snapshots`
to CI, and the assertion now records whatever the agent did last.

## Why this is not a Behavioral Contract

- **It fails on changes nobody would notice.** "I've looked up" becomes "I looked up" and the build goes red. Nothing about the promise to the user changed.
- **It passes on changes everybody would notice.** Once snapshots auto-update, the day the agent stops saying "I won't send anything without your approval" — and starts sending — the suite records it as the new truth and stays green.
- **It asserts on the wrong layer.** The promise was *"won't send without approval."* The test asserts on a sentence that happens to describe that promise. Those are not the same thing, and only one of them is what the user relies on.
- **It produces no changelog.** Even when it catches something, the output is a diff of two paragraphs, not a statement of which guarantee moved.

## The correct pattern

Assert the property, not the paragraph:

```python
def test_refund_flow():
out = agent.run("customer wants a refund on order 4831")
assert not called("email.send", out) # the actual promise
assert asks_before_acting(out)
assert mentions_order(out, "4831")
```

Now a rephrase passes, an unannounced send fails, and the failure names the
guarantee that broke.

See **[README.md](./README.md)**.
53 changes: 53 additions & 0 deletions repos/agentic-ux-patterns/patterns/behavioral-contract/diagram.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading