diff --git a/CLAUDE.md b/CLAUDE.md index 6c5eb0a..b29808e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -89,6 +89,7 @@ Entry point: `/optimize`. Audits an **existing** product: extracts artifacts fro | `07_ux_optimization/08_gap_analysis` | always | `hypothesis_backlog` | `user_flow`, `proto_persona`, `value_prop_inventory`, `core_objective` | | `07_ux_optimization/09_optimization_spec` | always | `optimization_spec` | `hypothesis_backlog` | | `07_ux_optimization/10_live_site_audit` | context_gated (deployed URL + browser; Tier 2) | `live_site_audit` | `user_flow` | +| `07_ux_optimization/11_runtime_audit` | context_gated (locally runnable + toolchain; Tier 2) | `runtime_audit` | `user_flow` | After validation, the registry matches forward-pass Phases 00-04 and the standard downstream (product-flow, tech-spec, `/compile`) applies unchanged. Visual-layer defects are handed off to a design-system audit, not itemized in the spec. diff --git a/checks/word_limits.json b/checks/word_limits.json index 7827db9..e532e01 100644 --- a/checks/word_limits.json +++ b/checks/word_limits.json @@ -2,6 +2,7 @@ "_comment": "Per-prompt word limits from the Pattern Guide Tier 1 table (docs/updates/00_ENHANCEMENT_PATTERN.md), keyed by frontmatter `name`. Counting rule per ADR 0009 Enforcement: body words excluding the block. Prompts absent here are uncovered, not checked.", "analyze-optimization-gaps": 300, "audit-live-site": 350, + "audit-product-runtime": 400, "capture-initial-idea": 250, "compose-optimization-spec": 300, "create-problem-statement": 200, diff --git a/docs/DEFERRED.md b/docs/DEFERRED.md new file mode 100644 index 0000000..4c6ec0c --- /dev/null +++ b/docs/DEFERRED.md @@ -0,0 +1,16 @@ +# Deferred Work Log + +Framework-side items consciously deferred, with their trigger conditions. +Owner triages; delete entries when done or dropped. (Product-side findings +for audited targets live in their delivered packages, not here.) + +| # | Item | Origin | Trigger / next step | +|---|------|--------|---------------------| +| D1 | Spec-package `deltas/` layer for reverse-pass output (current-state vs target-state in compiled packages) | ADR 0013 §6 | First `/compile` run on a reverse-pass registry; would amend ADR 0010 | +| D2 | Tier 2 depth prompts for `02_evidence_mining` (deep-dive interviews/analytics mining) | ADR 0013 deferred list | A target repo with rich real user research shows up | +| D3 | `/package` output mode: formalize the README + manifest.yaml + evidence-files deliverable shape (used twice by hand: TrustedRouter, QuillCode) as a command like `/compile` | pilot notes, packaging addendum | Third manual package assembly — rule of three | +| D4 | Evidence-file embedding conventions (screenshots in packages): naming, referencing from manifest `role: evidence`, size budgets | QuillCode runtime pass | Fold into D3 | +| D5 | Headless environments silently lose Tier 2 verification modes (no browser / no toolchain / no OS permissions) — consider a "verification coverage" line in the spec output stating which modes ran and why others didn't | ADR 0014 consequences; ADR 0015 boundary handling | Next skill iteration; small prompt edit to `09_optimization_spec` | +| D6 | Interaction-level native driving (beyond launch + capture) requires Accessibility permission — document a permission-request checklist for runtime audits on macOS | QuillCode runtime pass boundary | When a runtime audit next needs in-app interaction | +| D7 | Forward-pass parity check: reverse pass now produces `core_objective`; audit whether forward-pass Phase 02 prompts and reverse `07_objective_metrics` stay consistent in format as either evolves | ADR 0014 | Any edit to either objective prompt; candidate warn-only check | +| D8 | ADR 0012 check suite: pre-existing warnings (5 placeholder-resolvability, 1 word-limit, 6 filename-convention, 2 constraints-count) predate ADR 0013 work and remain untriaged | check runs during ADR 0013-0015 implementation | Dedicated cleanup pass; all warn-only | diff --git a/docs/adrs/0015-runtime-audit-mode.md b/docs/adrs/0015-runtime-audit-mode.md new file mode 100644 index 0000000..aa26ecf --- /dev/null +++ b/docs/adrs/0015-runtime-audit-mode.md @@ -0,0 +1,83 @@ +# ADR 0015: Runtime Audit Mode for Locally Runnable Products + +## Status + +Accepted (2026-07-18) — owner-directed ("take any learnings from this test and bake it in to the plugin") + +## Context + +The reverse pass had two verification modes: code-only (always) and live-site +browser audit (ADR 0014, gated on a deployed URL). The QuillCode production +run exposed the missing third mode: products that are **locally runnable but +not deployed** — CLIs, desktop apps, native tools. Its gate correctly skipped +the live-site step, leaving the pass "strategy-rich, experience-blind" for +exactly the product class whose UX is mostly runtime feel. + +A manual build-and-run pass (2026-07-18) proved the mode's value with finding +classes that neither code reading nor browser auditing can reach: + +1. **Failure-path quality**: the product's primary command, run unconfigured, + died with a raw `NSError` — while its own `doctor` command diagnosed the + exact condition with the exact fix. Knowledge present, unwired. +2. **Fixture-vs-demo distinction**: the static pass had recommended marketing + "mock mode" as a zero-setup demo; running it showed a 24ms canned response + that never engages the prompt — a test fixture, not a demo. +3. **Configured-state misdiagnosis**: a placeholder credential surfaced as + "Balance unavailable" (reads as an outage) rather than an invalid-key + error — wrong self-diagnosis at the activation step. +4. **State side effects**: silent adoption of the launch directory as a + project; one surface honoring config isolation while another ignored it. +5. **Proxy-vs-product**: the product's UI test harness lacked the + activation-critical surface entirely — only observable by driving both. + +## Decision + +Add **`11_runtime_audit.md`** (produces `runtime_audit`, requires +`user_flow`, **Tier 2**, context_gated on "product is locally runnable and +its build toolchain is available"). The prompt encodes the tested method: +isolate state and verify each surface honors the isolation; failure paths +first, graded against the product's own diagnostics; first-launch +observation; configured-state delta with placeholder credentials only; +fixture-vs-demo testing; window-scoped native captures within granted OS +permissions; complete cleanup. Findings are tagged `[confirms]` / `[NEW]` / +`[corrects]` against the static pass. + +`09_optimization_spec` consumes `runtime_audit` when present, alongside +`live_site_audit`. The three verification modes are now: code-only (always) +→ live-site (deployed URL + browser) → runtime (locally runnable + toolchain); +a product may qualify for both gated modes. + +Safety boundaries are part of the decision, not implementation detail: no +real credentials, no account actions, no permission workarounds, no residue. + +## Consequences + +**Positive:** +- The reverse pass covers all three product shapes end-to-end (web site/app, + deployed service, local CLI/native), each with a verification mode. +- The `[corrects]` tag creates a feedback loop: runtime evidence can revise + static-spec recommendations before the owner sees them. + +**Negative:** +- Building arbitrary products is the heaviest gate yet (toolchains, build + time, platform variance); failed builds must degrade to the code-only pass + without blocking. +- Native visual coverage depends on user-granted OS permissions that vary by + session and cannot be assumed. + +## Enforcement + +- **Testable consequences:** + - TC-1: `plugin/prompts/07_ux_optimization/` contains 11 prompts (01-11) with filename order matching the SKILL.md sequence table. + - TC-2: `11_runtime_audit.md` is `tier: 2`, `context_gated`, with a `run_when` naming local runnability and toolchain availability. + - TC-3: `09_optimization_spec.md` references `runtime_audit` as an optional input. +- **Checks:** covered by the existing suite (frontmatter-v2, dependency-graph, filename-convention, word-limit via `checks/word_limits.json` entry); no new check code. +- **Not machine-checkable:** cleanup completeness at run time; credential-safety compliance; permission-boundary honesty. +- **Graduation log:** _(empty)_ + +## References + +- ADR 0013 (reverse pass), ADR 0014 (live-site mode — this ADR completes the + mode triad), ADR 0012 (enforcement). +- Evidence: `docs/references/ux-optimization-pilot-notes.md` (runtime-pass + addendum forthcoming with this change). diff --git a/docs/references/ux-optimization-pilot-notes.md b/docs/references/ux-optimization-pilot-notes.md index 5e0a496..284c005 100644 --- a/docs/references/ux-optimization-pilot-notes.md +++ b/docs/references/ux-optimization-pilot-notes.md @@ -127,6 +127,33 @@ Lessons now encoded in ADR 0014 and prompts 07/10: spec items → fix locations) + numbered docs + strategy dir with provenance frontmatter. Candidate for a future `/package` output mode. +## Runtime-audit pass on QuillCode (2026-07-18 addendum) — became ADR 0015 + +The skill's first production run (QuillCode: Swift desktop app + CLI, no +deployment — live-site gate correctly skipped) was followed by a manual +build-and-run pass that defined the third verification mode: + +- **Failure-path grading against the product's own diagnostics** found the + single best item in the spec: primary command dies with a raw NSError + while `doctor` explains the exact fix. New finding class: "knowledge + present, unwired." +- **`[corrects]` matters:** runtime evidence overturned a static + recommendation (mock mode is a 24ms canned fixture, not a demo — the + "market the mock as a demo" advice was wrong). The tag set became + confirms/NEW/corrects. +- **Placeholder-credential delta** exposed misdiagnosis at activation + (invalid key → "Balance unavailable" chip, reads as outage). +- **State observation** caught silent project adoption of the launch + directory and one surface ignoring config isolation (`QUILLCODE_HOME`). +- **Permission honesty:** screen recording enabled window-scoped native + captures; accessibility remained absent so the settings sheet stayed + unphotographed — documented as a boundary, not worked around. +- **Proxy-vs-product:** driving the product's own JS test harness alongside + the real app proved the harness lacks the activation surface entirely — + only observable by running both. + +All encoded in `11_runtime_audit.md` (ADR 0015). + ## Open questions for the ADR 1. Naming: `optimize-ux` (Chris's working name) vs `product-audit`; command `/audit` vs `/optimize`. diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index e6af3b3..d0d7e5c 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "product-dev", "description": "AI-assisted product development framework. Guides from vague idea to technical spec through structured UX research, hypothesis formation, and prototype planning.", - "version": "0.3.0", + "version": "0.4.0", "author": { "name": "clownware" }, diff --git a/plugin/prompts/07_ux_optimization/09_optimization_spec.md b/plugin/prompts/07_ux_optimization/09_optimization_spec.md index 9d3cdc6..f98dce1 100644 --- a/plugin/prompts/07_ux_optimization/09_optimization_spec.md +++ b/plugin/prompts/07_ux_optimization/09_optimization_spec.md @@ -19,7 +19,7 @@ traces to a journey, a persona, and evidence — a reader should never wonder Hypothesis backlog: {{hypothesis_backlog}} -Also read from `.product-dev/artifacts/`: `existing_feedback.md` if present (pre-existing tactical feedback from the owner or a reviewer — slot every item into the priority structure rather than appending it as a list), `live_site_audit.md` if present (browser-verified findings — cite its measurements in item rationales and mark code-only claims as unverified-live), and `validation_log.md` for outstanding unvalidated flags. +Also read from `.product-dev/artifacts/`: `existing_feedback.md` if present (pre-existing tactical feedback from the owner or a reviewer — slot every item into the priority structure rather than appending it as a list), `live_site_audit.md` and/or `runtime_audit.md` if present (verified findings — cite their measurements and observed behavior in item rationales, apply any `[corrects]` revisions, and mark code-only claims as unverified), and `validation_log.md` for outstanding unvalidated flags. Produce the optimization spec: diff --git a/plugin/prompts/07_ux_optimization/11_runtime_audit.md b/plugin/prompts/07_ux_optimization/11_runtime_audit.md new file mode 100644 index 0000000..70deaeb --- /dev/null +++ b/plugin/prompts/07_ux_optimization/11_runtime_audit.md @@ -0,0 +1,55 @@ +--- +name: audit-product-runtime +description: > + Build and run a CLI or native-app product to verify code-suspected UX + defects at runtime and find the failure-path, first-launch, and + state-behavior defects only execution can reveal. +run: context_gated +run_when: Product is locally runnable (CLI/native/desktop) and its build toolchain is available +produces: runtime_audit +requires: [user_flow] +tier: 2 +--- + + +You are a UX auditor running the actual product. Static analysis predicts; +execution proves — and the failure paths, first-launch behavior, and state +side effects are where running the product earns its cost. Isolate +everything, clean up everything, and never exceed granted OS permissions. + + +Traced journeys: +{{user_flow}} + +Build from source, then drive the real binaries — never a test harness or mock UI as a stand-in (audit those separately as design-intent, labeled as such). + +**Method:** +1. **Isolate state**: use the product's home/config isolation flags; verify each surface honors them (a surface that ignores isolation is itself a finding). Record every path the product writes. +2. **Failure paths first**: run the primary command/flow unconfigured (no key, no login, no setup). Grade the error message: does it name the cause and the fix? Compare it against the product's own diagnostics (doctor/status commands) — knowledge that exists but isn't wired into errors is a top finding class. +3. **First launch**: observe what appears, what is adopted or written silently, and whether the product's mode (demo/mock vs live) is communicated. +4. **Configured-state delta**: configure with a placeholder credential where safe; diff the experience against unconfigured — and check misdiagnosis (does an invalid credential produce an accurate error?). +5. **Fixtures vs demos**: test any mock/demo mode for whether it actually demonstrates the product or returns canned output — never recommend marketing a fixture as a demo. +6. **Native visuals**: capture window-scoped screenshots where OS permissions allow; state plainly which permissions were unavailable and what remains unaudited. +7. **Clean up completely**: clear credentials, remove created state, stop processes. + +**Output**: findings grouped by surface (CLI / app / harness), tagged `[confirms ]`, `[NEW]`, or `[corrects ]`, each with the observed behavior verbatim. Close with provenance frontmatter (`mode: extracted`, `provenance:` commands run × surfaces × permission boundaries, `confidence:`, `validation_status: pending`). + + +- Do NOT use real credentials or perform account actions — placeholder values only, cleared afterward +- Do NOT report harness/mock-UI behavior as product behavior — label proxies as proxies +- Do NOT leave any state behind: keys, config homes, processes, files +- Do NOT exceed granted OS permissions or work around permission boundaries — document them instead +- Do NOT exceed 500 words + + + +**CLI (fresh --home, sample repo)** +- `[NEW]` Unconfigured `steeply sync` exits with `Error: nil response` — no mention of the missing account, while `steeply doctor` correctly diagnoses "no account configured → run steeply login". Wire doctor's check into the error. +- `[corrects]` `--demo` returns a canned cabinet, not the user's data — a fixture, not a demo; static spec's "demo mode" framing revised. + +**App (built, launched, window-captured)** +- `[confirms]` First launch shows no sync indicator; `[NEW]` app ignores `STEEPLY_HOME` and wrote `~/.steeply` (CLI honors it). +- Boundary: accessibility permission absent — settings sheet uninspected. + +`mode: extracted · provenance: build + 6 commands × 2 surfaces; screen-recording granted, accessibility absent · confidence: high (CLI), medium (app) · validation_status: pending` + diff --git a/plugin/skills/ux-optimization/SKILL.md b/plugin/skills/ux-optimization/SKILL.md index 15fbb80..969a25c 100644 --- a/plugin/skills/ux-optimization/SKILL.md +++ b/plugin/skills/ux-optimization/SKILL.md @@ -35,6 +35,7 @@ Prompts live in `${CLAUDE_PLUGIN_ROOT}/prompts/07_ux_optimization/`: | 8 | `08_gap_analysis.md` | always | `hypothesis_backlog` | | 9 | `09_optimization_spec.md` | always | `optimization_spec` | | 10 | `10_live_site_audit.md` | context_gated (Tier 2) | `live_site_audit` | +| 11 | `11_runtime_audit.md` | context_gated (Tier 2) | `runtime_audit` | **Step 2 gate**: "Target repo contains research, interview, marketing, or outreach material." When skipping: "No research or marketing material found — skipping evidence mining. The problem statement will be inferred from code and README only, at reduced confidence. If research exists elsewhere (notes, call transcripts), share it and we'll run this step." @@ -52,7 +53,7 @@ Prompts live in `${CLAUDE_PLUGIN_ROOT}/prompts/07_ux_optimization/`: **Objective, gap analysis, and spec (steps 7-9) — in-chat.** The objective anchors gap-analysis priorities; the spec integrates `existing_feedback` if present and carries the visual-layer handoff line (design-system defects belong to a design-audit skill, not this spec). -**Live-site audit (step 10) — Tier 2, browser required.** When the deployed product is reachable and browser tooling exists, offer it after journey tracing (its findings feed the spec) or after the spec ships (as verification). Run it via a subagent or in-chat with browser tools; it upgrades code-suspected defects into measured, confirmed findings and catches render-only issues (orphans, theme seams, missing responsive collapse) that code review cannot see. +**Verification modes (steps 10-11) — Tier 2, gated.** Three verification levels exist: code-only (the default pass), **live-site audit** (step 10; deployed URL + browser — measured visual/render findings), and **runtime audit** (step 11; locally runnable CLI/native product + build toolchain — build it, drive the real binaries, grade failure paths against the product's own diagnostics, observe first-launch and state behavior). A product may qualify for both. Offer whichever gates pass after journey tracing, or post-spec as verification; their findings feed `09_optimization_spec`, including `[corrects]` revisions of static recommendations. If a build fails, degrade to code-only without blocking. ## Registry