From 6bc3296dfeebe67067cfb99a3d9c32d36d8e3ca8 Mon Sep 17 00:00:00 2001 From: m Date: Sat, 5 Sep 2026 11:02:27 +0200 Subject: [PATCH] docs: add intent, specifications and agent delivery packets (G1.3/G1.5) --- .agents/skills/swapos-deliver/SKILL.md | 14 ++++ .agents/skills/swapos-verify/SKILL.md | 14 ++++ AGENTS.md | 4 + PR_SUMMARY.md | 62 ++------------ README.md | 22 +++++ STATE.md | 7 ++ docs/execution/CONTEXT.md | 40 +++++++++ docs/execution/INTENT.md | 20 +++++ docs/execution/README.md | 45 +++++++++++ docs/execution/ROADMAP.md | 14 ++++ docs/execution/TECHNICAL_SPEC.md | 35 ++++++++ docs/execution/WORK_PACKETS.md | 107 +++++++++++++++++++++++++ 12 files changed, 329 insertions(+), 55 deletions(-) create mode 100644 .agents/skills/swapos-deliver/SKILL.md create mode 100644 .agents/skills/swapos-verify/SKILL.md create mode 100644 README.md create mode 100644 docs/execution/CONTEXT.md create mode 100644 docs/execution/INTENT.md create mode 100644 docs/execution/README.md create mode 100644 docs/execution/ROADMAP.md create mode 100644 docs/execution/TECHNICAL_SPEC.md create mode 100644 docs/execution/WORK_PACKETS.md diff --git a/.agents/skills/swapos-deliver/SKILL.md b/.agents/skills/swapos-deliver/SKILL.md new file mode 100644 index 0000000..b70fd7e --- /dev/null +++ b/.agents/skills/swapos-deliver/SKILL.md @@ -0,0 +1,14 @@ +--- +name: swapos-deliver +description: Implement a bounded ModelSwapper work packet using its source map, contracts and acceptance criteria. Use for delivery in this repository. +--- + +# Deliver one SwapOS packet + +Read root instructions and `docs/execution/README.md`, then load only the selected packet in `docs/execution/WORK_PACKETS.md` and its contract in `TECHNICAL_SPEC.md`. Resolve paths from the repository root, not this skill directory. + +Use `docs/execution/CONTEXT.md` for the actual entry points and known gaps. Recheck the source SHA and packet dependencies against the current checkout. Keep benchmark tasks, graders and baseline runners outside the edit set. Check whether the selected backend is per-phase, resident or shared before changing cleanup or caching. + +Restate the numbered acceptance criteria and reserve the named files before edits. Use the fixtures and commands in the packet. Preserve the existing public contract unless the packet explicitly specifies its migration. Do not repair adjacent issues or edit a historical result to make an outcome look successful. + +Hand the verifier the resulting commit, diff and reproduction commands using the execution-package handoff record. Keep the task incomplete until the required tests and review are recorded. If a dependency is unavailable, finish the bounded local preparation and record the missing gate precisely. diff --git a/.agents/skills/swapos-verify/SKILL.md b/.agents/skills/swapos-verify/SKILL.md new file mode 100644 index 0000000..b89e610 --- /dev/null +++ b/.agents/skills/swapos-verify/SKILL.md @@ -0,0 +1,14 @@ +--- +name: swapos-verify +description: Independently verify a ModelSwapper work packet against its technical contract and failure cases. Use when reviewing an implementation in this repository. +--- + +# Verify one SwapOS packet + +Read root instructions, `docs/execution/README.md`, the selected work packet and `TECHNICAL_SPEC.md` before reading the implementer's explanation. Resolve all paths from the repository root. + +Record the exact commit under review. Derive at least one success case and the named refusal/failure cases from the contract. Check same-size model replacement and backend cleanup with temporary files/fakes. For measurement packets require raw paired results and the actual host, not a projection from faster prefill. + +Use the packet's test commands and the context file's environment prerequisites. Separate source inspection, fake-backed tests and live evidence. Inspect shared-state, error and rollback behavior within the packet's scope. Check that no source fixture, oracle, budget or threshold was weakened to obtain a pass. + +Return the handoff record with an outcome for every acceptance criterion and a verdict of `pass`, `changes_requested` or `blocked`. Missing hardware, dependencies or services are explicit `not_run` entries. A preparation-only pass cannot close an execution or launch gate. Do not modify production code, create external artifacts, merge or deploy as part of verification. diff --git a/AGENTS.md b/AGENTS.md index bb01c3c..fe56103 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,3 +60,7 @@ benchmarks/ # tasks/ (the 50-task suite), harness/ (grader + runners — sac hardware/ # per-tier configs and tuning profiles (tiers.yaml) docs/adr/ # ADRs, one per non-obvious choice ``` + +## Execution package (reviewed 2026-09-05) + +For the requested intent, roadmap, technical contracts and developer/test work packets, start at [docs/execution/README.md](docs/execution/README.md). Repository-scoped delivery and verification skills are under `.agents/skills/`. Existing instructions and domain constraints above remain in force. diff --git a/PR_SUMMARY.md b/PR_SUMMARY.md index 5ce5285..66e6a95 100644 --- a/PR_SUMMARY.md +++ b/PR_SUMMARY.md @@ -1,61 +1,13 @@ -# PR_SUMMARY — 2026-08-28 landing: Step-0 symmetric baseline + p-value amendment + benchmark page (gate 4) + validation-program templates (Engine 1) +# Agent execution documentation — G1.3 / G1.5 -Branch: `wt/land-2026-08-28` (merged onto main as commit) · goal refs: **Step 0 / Gate 4 / Engine 1** (docs/roadmap-to-revenue.md, audited 2026-08-21) +Existing plans and status notes do not consistently match the implemented repository. Agents need a reliable source map and bounded contracts before taking implementation tickets. -## What +This documentation change adds intent, a dependency roadmap, reviewed context/findings, technical specifications and developer/test work packets under `docs/execution/`. It adds two repository-scoped skills and links the package from root instructions. Historical evidence and existing domain constraints are preserved; stale status statements are qualified with a dated review checkpoint. -Four reviewer-approved strands merged onto main in one landing, plus the one -review-required page fix: +The work packets identify implementation ownership, independent verification, input/output/error behavior, acceptance criteria, dependencies and rollback. Readiness is separate from future implementation completion. This PR does not implement the proposed application fixes or claim that their acceptance tests already pass. -1. **Step 0 symmetric baseline (review t_d24fd5e4 APPROVED)** — the falsification - experiment: `benchmarks/run_symmetric_baseline.py` (deepseek-v4-pro with the - pipeline's identical REASON → CODE → grader → CRITIC → retry loop), - evidence `benchmarks/results/symmetric-baseline-20260825.json` - (48/50, pass@1 46, $0.2461), narrative `docs/symmetric-baseline.md`. - Verdict: falsification did NOT produce ~50/50; parity claim survives - (47 vs 48, McNemar p≈1.0). -2. **pass@1 p-value amendment (t_fd73b8df)** — `docs/four-number-proof.md` §2.1 - and `docs/parity-report-phase1.md` Addendum 4 item 3 corrected to exact - McNemar p = 0.016 (b=7 c=0) for the pass@1-only comparison (was p≈1.0). -3. **Benchmark page + methodology standard (gate 4, review t_23f2732a — 1 - required fix applied, see below)** — `docs/methodology-standard.md` - (open-funnel validation methodology), `site/build_site.py` + `site/index.html` - (deterministic static parity-evidence page, zero external assets), - canonical run evidence `benchmarks/results/symmetric-baseline-deepseek-v4-pro-20260825-195414.json` - (48/50, pass@1 46, $0.4539), and the reconciliation note ($0.25–$0.45 per suite). -4. **Validation-program templates (review t_2e6095e4 APPROVED)** — Engine 1 - deliverables `templates/validation-program/{one-pager,validation-report,README}.md` - (partner-led validation program shape, roadmap-approved price bands only, - EXAMPLE blocks with measured numbers). +Validation: 37 capsule/grader/pipeline tests passed with isolated CPU dependencies; 50 benchmark tasks passed structural validation. No hardware or API baseline run. -**Required review fix applied in this landing (t_23f2732a):** `site/build_site.py` -PIPELINE_DOC.failed and LOOP_MCNEMAR.api_only_tasks now list `refactor-02` -(pipeline's real third failure, per committed JSON) instead of `refactor-11` -(passed 7/7, iteration 1). Recommended fixes also applied: build now asserts -pipeline failed-task ids against the parsed JSON, and `docs/parity-report-phase1.md` -Addendum 5 pipeline-failed cell aligned to `refactor-02`. +All 12 repository-scoped skills across this six-repository review passed the skill validator. New documentation links and bounded packet structure were checked locally. No merge, deployment, training, external send or customer-data operation is included. -## Why - -The 08-27 operating cycle was blocked by a DeepSeek billing 402 before creating -anything — three reviewer-approved strands had been sitting off-main for 1–3 -cycles. This landing closes that stranding and delivers the roadmap's -publication-critical pieces (Step-0 verdict, page, methodology) plus the -Engine-1 sales templates onto main. - -## How tested (gate steps, real output) - -1. **Quality gate:** `uv run --with pytest pytest capsule/tests benchmarks/harness/tests pipeline/tests` → **37 passed** (rerun independently by wrapper after the fix). -2. **Sacred rule:** `git diff main..HEAD -- benchmarks/tasks benchmarks/harness` = **0 lines** (only NEW files added under benchmarks/: the runner + two result JSONs). -3. **Build determinism:** `python3 site/build_site.py` run twice → **byte-identical index.html**; page asserts invariants against committed JSONs (incl. the new failed-task invariant). -4. **Serve check:** `python3 -m http.server -d site` + curl → headline "48/50 on two independent runs, $0.25–$0.45 per suite" renders. -5. **Merge hygiene:** both lineage-A (approved base b762749) and lineage-B (page/templates base 3bea626) strands merged; only conflict was PR_SUMMARY.md (add/add) — resolved as this combined summary; parity-report Addendum 4 amendment + Addendum 5 both present; roadmap Step 0 DONE recorded; g2.4/notebooks intact. - -## Reconciliation note - -Two Step-0 runs are published on the page, both 48/50: canonical -`symmetric-baseline-deepseek-v4-pro-20260825-195414.json` ($0.4539) and -reproduction `symmetric-baseline-20260825.json` ($0.2461) — headline "$0.25–$0.45 -per suite". No "1/50th cost" or any cost-ratio claim anywhere (retracted red -line); hardware floor (G1.5) stated as open item, no placeholders; zero external -assets. +Goal references: G1.3 and G1.5. Four-number impact: this user-requested documentation prepares quality, latency and hardware-floor measurements; it records no new numerical improvement. It preserves the frozen task/grader/baseline boundaries. diff --git a/README.md b/README.md new file mode 100644 index 0000000..6435b9e --- /dev/null +++ b/README.md @@ -0,0 +1,22 @@ +# SwapOS / ModelSwapper + +A research runtime for local specialist-model pipelines with Context Capsule handoff. + +Start with [the execution package](docs/execution/README.md) for reviewed intent, +source context, a dependency roadmap, technical contracts, and bounded developer/test tickets. + +Read [AGENTS.md](AGENTS.md) and [MASTER-PROMPT.md](MASTER-PROMPT.md) before code changes. +The [methodology standard](docs/methodology-standard.md) explains benchmark interpretation; +[STATE.md](STATE.md) records progress and outstanding measurements. + +## Local verification + +With Python 3.11+, pytest and jsonschema installed, from the repository root: + +```bash +python3 -m pytest -q capsule/tests benchmarks/harness/tests pipeline/tests +python3 benchmarks/harness/validate_tasks.py --tasks-dir benchmarks/tasks +``` + +Actual inference requires the appropriate local runtime and model artifacts. Hardware +certification and performance claims require measured evidence on the stated host. diff --git a/STATE.md b/STATE.md index a001afc..7684a66 100644 --- a/STATE.md +++ b/STATE.md @@ -1,5 +1,12 @@ # STATE.md — ModelSwapper (SwapOS) +## Review checkpoint — 2026-09-05 + +Source review confirms implemented `pipeline/` and `router/` code. The historical empty-stub description below is obsolete. G1.3 and G1.5 remain open measurements. See [reviewed context](docs/execution/CONTEXT.md) and [work packets](docs/execution/WORK_PACKETS.md); this documentation establishes no new performance result. + +The following sections are historical status notes; use the checkpoint above and the linked source map for current code presence. + + ## Current state - **G1.3 lever (issue #16) IMPLEMENTED + MEASURED + reviewer-approved:** diff --git a/docs/execution/CONTEXT.md b/docs/execution/CONTEXT.md new file mode 100644 index 0000000..b1e7077 --- /dev/null +++ b/docs/execution/CONTEXT.md @@ -0,0 +1,40 @@ +# Code and review context + +## Sources + +| Path | Responsibility / observation | +| --- | --- | +| `MASTER-PROMPT.md`, `AGENTS.md` | Goals, four metrics, hardware rules and benchmark ownership | +| `pipeline/contracts.py` | `ModelBackend`, `GenerationResult`, `TaskRunResult` | +| `pipeline/loop.py` | `run_task`, capsule handoff, bounded critic feedback, checkpoint naming | +| `runtime/llama_backend.py`, `runtime/overlap_backend.py`, `runtime/overlap.py` | Process and two-slot backend implementations | +| `capsule/capsule.py`, `capsule/compress.py` | Capsule 0.1.0, atomic persistence and separate compression mechanism | +| `router/rules.py` | Deterministic phase routing; inspect actual use before asserting G1.4 coverage | +| `pipeline/run_pipeline.py` | Runner and persisted summary aggregation | +| `benchmarks/harness/`, `benchmarks/tasks/` | Grader, baseline code and frozen 50-task suite | +| `docs/methodology-standard.md`, `docs/symmetric-baseline.md` | Comparison protocol and historical evidence | +| `hardware/tiers.yaml`, `docs/t0-air-RUN.md` | Hardware configuration and uncompleted T0 measurement | + +## Findings from source inspection + +1. The pipeline and router contain implementation; the older STATE statement calling them empty stubs is stale. +2. `kv_checkpoint_name` includes a model path and file size, but not a content digest. Two different same-size artifacts at the same path can produce the same checkpoint identity. SWAP-01 specifies a correction without claiming a reproduced inference failure. +3. Nonresident generation has a `finally` cleanup; resident-mode cleanup is after the loop. An unexpected exception from grading or downstream processing can bypass that final stop. SWAP-02 specifies lifecycle coverage. +4. CI runs capsule and grader tests, but omits `pipeline/tests`, despite the broader gate in AGENTS. SWAP-02 closes that coverage gap. +5. Compression is deliberately not wired into the loop; do not treat that design choice as an accidental omission. + +## Commands and prerequisites + +From repository root, with Python 3.11+, pytest and jsonschema installed: + +```bash +python3 -m pytest -q capsule/tests benchmarks/harness/tests pipeline/tests +python3 benchmarks/harness/validate_tasks.py --tasks-dir benchmarks/tasks +python3 pipeline/run_pipeline.py --help +``` + +Real inference requires llama.cpp and explicitly supplied model artifacts. No GPU/model download or external API run is needed for the first two packets. Follow the existing hardware runbook for real measurements, preserving its complete command, model identity and host details. + +## Review evidence + +The 50-task structural validation passed in this review environment. After installing isolated CPU test dependencies, the capsule, grader and pipeline suites passed: 37 tests. The command used the isolated dependency directory on PYTHONPATH. No GPU run, new cost benchmark, T0 memory measurement or baseline API call was performed. Existing JSON results were inspected as historical evidence. diff --git a/docs/execution/INTENT.md b/docs/execution/INTENT.md new file mode 100644 index 0000000..57b6596 --- /dev/null +++ b/docs/execution/INTENT.md @@ -0,0 +1,20 @@ +# Intent + +Enable a developer to run a reason → code → mechanical review → critic/retry workflow using local specialist models and a portable Context Capsule. Success is measured on the same frozen tasks against an explicitly identified baseline. + +## Users and value + +The first user is a technical evaluator with local compute and privacy requirements. The immediate deliverable is a reproducible runtime evaluation: code output, test outcome, phase timings, memory observations and a complete run manifest. Installation and licensing follow evidence and a validated user need. + +## Outcomes + +- Quality: satisfy the G1.2 ratio on the frozen suite; report absolute counts and denominators, including all failed runs. +- Latency: close G1.3 using measured total task time, not only a faster subphase. +- Hardware: independently measure G1.5 on the specified 24 GB machine; an 8 GB or large GPU run is a different evidence lane. +- Economics: preserve actual cost basis and unknown values; the historical 1/50th-cost thesis is not an established result. + +## Boundaries + +The core uses local inference and specialists at most 32B class. Explicit API baselines are separate experiments. Capsule handoff must remain model-agnostic; a KV checkpoint is an optional optimization for a compatible model and prefix. Frozen oracles are never changed to rescue a candidate. + +The master prompt defines the long-term research goals. This package sequences reliability and measurement work within those goals. It does not authorize publication, paid compute, additional hardware or a new commercial commitment. diff --git a/docs/execution/README.md b/docs/execution/README.md new file mode 100644 index 0000000..a40be61 --- /dev/null +++ b/docs/execution/README.md @@ -0,0 +1,45 @@ +# SwapOS execution package + +Reviewed 2026-09-05 against `379064ea2af77a0deed738296da36337448bde94` on `main`. This is a source review and a proposed delivery plan. A specification describes intended behavior; it does not establish that the behavior already exists. Recorded historical results are not new measurements. + +## Read in this order + +1. Repository instructions at the root, then [Intent](INTENT.md). +2. [Current code and context](CONTEXT.md), including the verification limits. +3. [Roadmap](ROADMAP.md) to select a dependency-ready work packet. +4. [Technical specification](TECHNICAL_SPEC.md) and the selected [work packet](WORK_PACKETS.md). + +Existing product goals, ADRs, schemas and task evidence remain authoritative for their domains. These documents make the next work explicit; they do not silently retire an existing decision. Recheck the target branch and open PRs before executing an old packet. + +## Developer and test pair + +The coordinator hands both roles the same packet ID, base commit, acceptance criteria, permitted files and fixture contract. One implementer owns the code branch. The verifier independently derives failure cases from the specification and records its own observations. When two agents are unavailable, perform separate implementation and verification passes and disclose that independence was unavailable. + +The implementer reads the named sources, reproduces the current behavior, changes only the assigned scope and hands over a commit plus commands. The verifier reads the contract before the implementation explanation, tests success and refusal/error paths, inspects the diff and returns `pass`, `changes_requested` or `blocked` against that exact commit. A test that never ran is `not_run`, never a pass. Do not mark an existing unmet gate complete merely because a packet's preparation is complete. + +Use distinct branches/worktrees when executing concurrently. Packets with overlapping owned files run serially unless the coordinator explicitly partitions the edits. The coordinator integrates one reviewed change at a time, reruns the affected integration gate and reconciles status. Tests may add fixtures under their assigned test paths; shared configuration and schema edits have one owner. + +## Handoff record + +```yaml +packet_id: selected ID +base_commit: exact SHA +implementation_commit: exact SHA +acceptance_results: # one entry for every numbered criterion + - criterion: AC1 + outcome: pass | fail | not_run + command: exact command and working directory + evidence: result or repository artifact path +changed_files: [] +remaining_risks: [] +verifier: independent reviewer identity or independence unavailable +verdict: pass | changes_requested | blocked +``` + +Keep test data synthetic and generated evidence free of credentials. Work packets authorize their stated code/document scope, not external sends, purchases, model training, production deployment or data migration. Respect the repository's existing authorization boundaries. + +## Skills + +Repository-scoped skills are in `.agents/skills/swapos-deliver/` and `.agents/skills/swapos-verify/`. Invoke the applicable skill with a packet ID. If the client does not discover repository skills, open its `SKILL.md` explicitly; no personal skill installation is required. + +ModelSwapper packets retain the original G-goal IDs. Use the branch conventions in `AGENTS.md` and an ADR for architectural changes. Benchmark tasks, graders and baseline runners remain outside implementation ownership. This documentation prepares quality, latency and hardware measurements; it does not claim a numerical improvement. diff --git a/docs/execution/ROADMAP.md b/docs/execution/ROADMAP.md new file mode 100644 index 0000000..f714647 --- /dev/null +++ b/docs/execution/ROADMAP.md @@ -0,0 +1,14 @@ +# Dependency roadmap + +| Order | Packet / milestone | Dependency | Exit evidence | +| --- | --- | --- | --- | +| 1 | SWAP-01: bind checkpoints to artifact identity | None | Same-size replacement cannot restore a stale checkpoint; fake-backed tests | +| 1 | SWAP-02: close lifecycle and CI coverage gaps | None, but serialize shared `pipeline/loop.py` edits with SWAP-01 | All success/failure paths stop owned backends; pipeline suite runs in CI | +| 2 | SWAP-03: paired latency experiment for G1.3 | SWAP-01 + SWAP-02; authorized hardware | Immutable run pair, whole-task latency, failures and quality comparison | +| 2 | SWAP-04: T0 certification for G1.5 | Reliable runner + actual 24 GB host | Full-suite memory/timing evidence on the stated host | +| 3 | G3 tier and registry work | Review measured hardware findings | Reproducible configs, model manifests and ADR-backed residency rules | +| 4 | G4 install and external harness integration | Measurement gates and user need | Unaided install and task completion by an external evaluator | + +SWAP-01/02 are small code tickets. SWAP-03/04 permit local preparation now, but the measurement portions remain blocked on actual hardware and the existing execution authorization. Do not turn that prerequisite into fabricated estimates. Work beyond these four packets needs a newly bounded specification before delegation. + +Keep G1.3 and G1.5 open until their existing numerical bars are measured. Documentation or a passing CPU fake is insufficient. Historical G2 closure is preserved; newly observed limitations must be recorded with evidence, not silently used to rewrite prior measurements. diff --git a/docs/execution/TECHNICAL_SPEC.md b/docs/execution/TECHNICAL_SPEC.md new file mode 100644 index 0000000..2bea7df --- /dev/null +++ b/docs/execution/TECHNICAL_SPEC.md @@ -0,0 +1,35 @@ +# Technical specification + +## Existing interfaces to preserve + +`ModelBackend.start()` loads a backend; `generate(prompt, max_tokens, temperature, prefetch_model, kv_restore, kv_save)` returns `GenerationResult`; `stop()` returns eviction seconds. Optional cache support must not become mandatory for fakes or other backends. `TaskRunResult` keeps task ID, category, pass result, iterations, test counts, phases, capsule bytes, wall time and an optional error. + +A successful task is decided by the mechanical grader. Model self-review cannot change `passed` to true. Preserve seconds for backend timing fields, milliseconds for capsule swap history and the existing `peak_rss_kb` field; do not equate a process RSS observation with total unified-memory usage. + +## SWAP-01: cache identity and reuse + +The checkpoint identity is SHA-256 over a canonical, sorted JSON object containing identity format version `1`, full model artifact SHA-256, task ID, exact problem, starter and plan text. UTF-8 encoding is explicit. Use the entire digest in `kv-.bin`. The backend compatibility namespace must additionally include runtime/build identity and context settings before cross-run reuse is enabled. + +Hash model contents once per run/model and pass that immutable identity into checkpoint creation; do not rehash gigabytes on every retry. Use a run-private cache namespace initially so unknown runtime metadata cannot cause cross-run reuse. If an artifact cannot be read, disable checkpoint restore/save for it and perform normal generation; do not substitute an empty or all-zero digest. Existing cache files are optimization artifacts and may be ignored; no migration of result JSON is needed. + +Fixtures: two temporary model files with the same byte count but different content; identical bytes copied to a different path; changed plan text; unreadable/missing file. No actual model weights are required. Expected identity follows content and prefix, not path alone. Existing correct behavior on cache failure still falls back to full prefill. + +## SWAP-02: backend ownership + +Every successfully constructed backend whose start is attempted receives exactly one cleanup attempt by its owning layer, including start failure, generation failure, grader exception and exhausted retry budget. A resident backend is owned by an outer `try/finally` around the task lifecycle; a per-phase backend is owned by that phase. An overlap engine retains its existing outer ownership in the runner; avoid double cleanup of a shared engine. + +Preserve the original failure as the primary error if cleanup also fails. All failures remain failed tasks, and error evidence names the phase. Do not change grader semantics. Add `pipeline/tests` to the existing CI pytest invocation; preserve structural suite validation. + +## SWAP-03: latency experiment contract + +Prepare a comparison manifest containing commit, task-pack digest, complete model digests/quantization, llama.cpp build, hardware, context size, temperature, output budget, retry budget, warm/cold policy, baseline result reference, run IDs and commands. Hold all values except the declared treatment fixed. Use the same frozen tasks and an identical loop protocol. Keep failed tasks in denominators and record infrastructure failures distinctly. + +Record per-task quality, total wall time, phase load/evict/generate/prefill timings and cache hit/miss evidence in new result files. Compare paired task outcomes and mean total time. State sampling uncertainty and the small-suite limitation; failure to reject a difference does not establish equivalence. G1.3 passes only at the original total-time bar, with its stated baseline and G1.2 maintained. Report subphase improvement separately. + +## SWAP-04: hardware evidence contract + +Follow the T0 runbook on a verified 24 GB host. Record host model, OS, memory capacity, model digests, runtime build, sampler/method and peak memory for every task. G1.5 requires peak unified-memory use below 20 GB for every task, with operating-system headroom; if the sampler measures only process RSS, label the gate unproven. OOM, swap pressure and unfinished tasks remain visible. A reduced suite or remote large-GPU run cannot certify T0. + +## Compatibility and rollback + +These tickets do not change benchmark tasks or historical result files. Cache naming can be reverted or disabled without affecting correctness. Lifecycle fixes preserve return schemas. Measurement outputs are append-only new artifacts; an invalid run is retained and labeled invalid rather than deleted. diff --git a/docs/execution/WORK_PACKETS.md b/docs/execution/WORK_PACKETS.md new file mode 100644 index 0000000..64fae10 --- /dev/null +++ b/docs/execution/WORK_PACKETS.md @@ -0,0 +1,107 @@ +# Bounded work packets + +These are proposed implementation tickets, not claims of completed work. `Ready` means sufficiently specified to begin local work. `Blocked` names an unmet dependency or decision. Historical task ledgers retain completed-task evidence. + +## SWAP-01 — Make KV checkpoint identity content-addressed + +**Readiness:** Ready. **Depends on:** None. **Parent goal:** G1.3 reliability prerequisite. + +**Outcome:** Prevent reuse of a cache derived from different model bytes. + +**Read:** `pipeline/loop.py`, `runtime/llama_backend.py`, `pipeline/tests/test_loop.py`. + +**Implementation ownership:** `pipeline/loop.py`, `pipeline/run_pipeline.py`, `runtime/llama_backend.py` only as required to pass identity; new ADR under `docs/adr/`. + +**Verifier ownership:** New identity fixtures/tests in `pipeline/tests/`. + +**Contract:** TECHNICAL_SPEC SWAP-01. + +**Acceptance criteria:** + +1. **AC1:** Same-size different bytes produce different names; unchanged bytes/prefix in the same compatible namespace produce the same name. +2. **AC2:** Missing identity disables reuse while normal generation remains available. +3. **AC3:** Digest work occurs once per model per run, and existing cache fallback tests still pass. + +**Verification:** `python3 -m pytest -q pipeline/tests` then the root quality gate. + +**Non-goals:** Cross-model KV reuse, benchmark edits, GPU runs. + +**Failure/rollback:** Disable the optional cache or revert naming changes; do not delete historical results. + +## SWAP-02 — Guarantee backend cleanup and include pipeline tests in CI + +**Readiness:** Ready; serialize after SWAP-01 if both are selected. **Depends on:** None. **Parent goal:** G1.3 / G1.5 reliability prerequisite. + +**Outcome:** Ensure failed tasks release backend resources and CI exercises the runtime loop. + +**Read:** `pipeline/loop.py`, `pipeline/run_pipeline.py`, `.github/workflows/ci.yml`. + +**Implementation ownership:** `pipeline/loop.py`, `pipeline/run_pipeline.py`, `.github/workflows/ci.yml`. + +**Verifier ownership:** `pipeline/tests/test_loop.py` and new test fakes there. + +**Contract:** TECHNICAL_SPEC SWAP-02. + +**Acceptance criteria:** + +1. **AC1:** Instrumented fakes observe one owned stop on normal completion, start/generate failure, grading exception and exhausted budget. +2. **AC2:** The original error remains visible if stop also fails; failed work is never marked passed. +3. **AC3:** CI runs capsule, grader and pipeline suites and retains task structural validation. + +**Verification:** Full root quality gate; injected failures use no models or providers. + +**Non-goals:** Changing task oracles, retry policy or residency architecture. + +**Failure/rollback:** Revert the lifecycle patch if ownership regressions appear; preserve failing tests and error evidence. + +## SWAP-03 — Prepare and execute a controlled G1.3 comparison + +**Readiness:** Preparation ready; measurement blocked on authorized hardware. **Depends on:** SWAP-01, SWAP-02 for the corrected-cache comparison. **Parent goal:** G1.3. + +**Outcome:** Determine whether the chosen treatment reduces whole-task latency while preserving quality. + +**Read:** `docs/methodology-standard.md`, `docs/symmetric-baseline.md`, `pipeline/run_pipeline.py`. + +**Implementation ownership:** New experiment manifest/report in `docs/`; new immutable JSON under `benchmarks/results/`. + +**Verifier ownership:** Verifier owns manifest audit, paired analysis and full-gate evidence. + +**Contract:** TECHNICAL_SPEC SWAP-03. + +**Acceptance criteria:** + +1. **AC1:** Manifest fixes task pack, models, runtime and budgets; only the treatment varies. +2. **AC2:** Every task, failed attempt and infrastructure failure appears in the evidence. +3. **AC3:** Report states total-time ratio and quality denominators; G1.3 is closed only if its bar is actually met. + +**Verification:** Existing runner command recorded verbatim in the manifest; raw result paths and paired calculations attached after the hardware run. + +**Non-goals:** Benchmark mutation, cherry-picking tasks, automatic paid compute. + +**Failure/rollback:** Mark an invalid experiment invalid and schedule a corrected run; keep original raw evidence. + +## SWAP-04 — Run the T0 memory certification protocol + +**Readiness:** Blocked on an authorized 24 GB host; runbook preparation ready. **Depends on:** SWAP-02 before relying on the updated lifecycle. **Parent goal:** G1.5. + +**Outcome:** Establish the 24 GB hardware floor using directly measured memory. + +**Read:** `docs/t0-air-RUN.md`, `hardware/t0_air_run.sh`, `hardware/tiers.yaml`. + +**Implementation ownership:** T0 runbook amendments and new `benchmarks/results/` evidence only. + +**Verifier ownership:** Verifier owns host/sampler audit and per-task gate assessment. + +**Contract:** TECHNICAL_SPEC SWAP-04. + +**Acceptance criteria:** + +1. **AC1:** Host capacity and sampling method are recorded; RSS and unified memory are distinguished. +2. **AC2:** All 50 task outcomes and memory peaks are retained. +3. **AC3:** Every task remains below the existing 20 GB ceiling before G1.5 is marked satisfied. + +**Verification:** Run the existing T0 command on the named host; attach raw observations and the full report. + +**Non-goals:** Hardware purchase, estimated certification, broad tier implementation. + +**Failure/rollback:** An unsupported sampler or incomplete run leaves certification blocked; no thresholds are weakened.