Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .agents/skills/swapos-deliver/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
name: swapos-deliver
description: Implement a bounded ModelSwapper work packet using its source map, contracts and acceptance criteria. Use for delivery in this repository.
---

# Deliver one SwapOS packet

Read root instructions and `docs/execution/README.md`, then load only the selected packet in `docs/execution/WORK_PACKETS.md` and its contract in `TECHNICAL_SPEC.md`. Resolve paths from the repository root, not this skill directory.

Use `docs/execution/CONTEXT.md` for the actual entry points and known gaps. Recheck the source SHA and packet dependencies against the current checkout. Keep benchmark tasks, graders and baseline runners outside the edit set. Check whether the selected backend is per-phase, resident or shared before changing cleanup or caching.

Restate the numbered acceptance criteria and reserve the named files before edits. Use the fixtures and commands in the packet. Preserve the existing public contract unless the packet explicitly specifies its migration. Do not repair adjacent issues or edit a historical result to make an outcome look successful.

Hand the verifier the resulting commit, diff and reproduction commands using the execution-package handoff record. Keep the task incomplete until the required tests and review are recorded. If a dependency is unavailable, finish the bounded local preparation and record the missing gate precisely.
14 changes: 14 additions & 0 deletions .agents/skills/swapos-verify/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
name: swapos-verify
description: Independently verify a ModelSwapper work packet against its technical contract and failure cases. Use when reviewing an implementation in this repository.
---

# Verify one SwapOS packet

Read root instructions, `docs/execution/README.md`, the selected work packet and `TECHNICAL_SPEC.md` before reading the implementer's explanation. Resolve all paths from the repository root.

Record the exact commit under review. Derive at least one success case and the named refusal/failure cases from the contract. Check same-size model replacement and backend cleanup with temporary files/fakes. For measurement packets require raw paired results and the actual host, not a projection from faster prefill.

Use the packet's test commands and the context file's environment prerequisites. Separate source inspection, fake-backed tests and live evidence. Inspect shared-state, error and rollback behavior within the packet's scope. Check that no source fixture, oracle, budget or threshold was weakened to obtain a pass.

Return the handoff record with an outcome for every acceptance criterion and a verdict of `pass`, `changes_requested` or `blocked`. Missing hardware, dependencies or services are explicit `not_run` entries. A preparation-only pass cannot close an execution or launch gate. Do not modify production code, create external artifacts, merge or deploy as part of verification.
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,7 @@ benchmarks/ # tasks/ (the 50-task suite), harness/ (grader + runners — sac
hardware/ # per-tier configs and tuning profiles (tiers.yaml)
docs/adr/ # ADRs, one per non-obvious choice
```

## Execution package (reviewed 2026-09-05)

For the requested intent, roadmap, technical contracts and developer/test work packets, start at [docs/execution/README.md](docs/execution/README.md). Repository-scoped delivery and verification skills are under `.agents/skills/`. Existing instructions and domain constraints above remain in force.
62 changes: 7 additions & 55 deletions PR_SUMMARY.md
Original file line number Diff line number Diff line change
@@ -1,61 +1,13 @@
# PR_SUMMARY — 2026-08-28 landing: Step-0 symmetric baseline + p-value amendment + benchmark page (gate 4) + validation-program templates (Engine 1)
# Agent execution documentation — G1.3 / G1.5

Branch: `wt/land-2026-08-28` (merged onto main as commit) · goal refs: **Step 0 / Gate 4 / Engine 1** (docs/roadmap-to-revenue.md, audited 2026-08-21)
Existing plans and status notes do not consistently match the implemented repository. Agents need a reliable source map and bounded contracts before taking implementation tickets.

## What
This documentation change adds intent, a dependency roadmap, reviewed context/findings, technical specifications and developer/test work packets under `docs/execution/`. It adds two repository-scoped skills and links the package from root instructions. Historical evidence and existing domain constraints are preserved; stale status statements are qualified with a dated review checkpoint.

Four reviewer-approved strands merged onto main in one landing, plus the one
review-required page fix:
The work packets identify implementation ownership, independent verification, input/output/error behavior, acceptance criteria, dependencies and rollback. Readiness is separate from future implementation completion. This PR does not implement the proposed application fixes or claim that their acceptance tests already pass.

1. **Step 0 symmetric baseline (review t_d24fd5e4 APPROVED)** — the falsification
experiment: `benchmarks/run_symmetric_baseline.py` (deepseek-v4-pro with the
pipeline's identical REASON → CODE → grader → CRITIC → retry loop),
evidence `benchmarks/results/symmetric-baseline-20260825.json`
(48/50, pass@1 46, $0.2461), narrative `docs/symmetric-baseline.md`.
Verdict: falsification did NOT produce ~50/50; parity claim survives
(47 vs 48, McNemar p≈1.0).
2. **pass@1 p-value amendment (t_fd73b8df)** — `docs/four-number-proof.md` §2.1
and `docs/parity-report-phase1.md` Addendum 4 item 3 corrected to exact
McNemar p = 0.016 (b=7 c=0) for the pass@1-only comparison (was p≈1.0).
3. **Benchmark page + methodology standard (gate 4, review t_23f2732a — 1
required fix applied, see below)** — `docs/methodology-standard.md`
(open-funnel validation methodology), `site/build_site.py` + `site/index.html`
(deterministic static parity-evidence page, zero external assets),
canonical run evidence `benchmarks/results/symmetric-baseline-deepseek-v4-pro-20260825-195414.json`
(48/50, pass@1 46, $0.4539), and the reconciliation note ($0.25–$0.45 per suite).
4. **Validation-program templates (review t_2e6095e4 APPROVED)** — Engine 1
deliverables `templates/validation-program/{one-pager,validation-report,README}.md`
(partner-led validation program shape, roadmap-approved price bands only,
EXAMPLE blocks with measured numbers).
Validation: 37 capsule/grader/pipeline tests passed with isolated CPU dependencies; 50 benchmark tasks passed structural validation. No hardware or API baseline run.

**Required review fix applied in this landing (t_23f2732a):** `site/build_site.py`
PIPELINE_DOC.failed and LOOP_MCNEMAR.api_only_tasks now list `refactor-02`
(pipeline's real third failure, per committed JSON) instead of `refactor-11`
(passed 7/7, iteration 1). Recommended fixes also applied: build now asserts
pipeline failed-task ids against the parsed JSON, and `docs/parity-report-phase1.md`
Addendum 5 pipeline-failed cell aligned to `refactor-02`.
All 12 repository-scoped skills across this six-repository review passed the skill validator. New documentation links and bounded packet structure were checked locally. No merge, deployment, training, external send or customer-data operation is included.

## Why

The 08-27 operating cycle was blocked by a DeepSeek billing 402 before creating
anything — three reviewer-approved strands had been sitting off-main for 1–3
cycles. This landing closes that stranding and delivers the roadmap's
publication-critical pieces (Step-0 verdict, page, methodology) plus the
Engine-1 sales templates onto main.

## How tested (gate steps, real output)

1. **Quality gate:** `uv run --with pytest pytest capsule/tests benchmarks/harness/tests pipeline/tests` → **37 passed** (rerun independently by wrapper after the fix).
2. **Sacred rule:** `git diff main..HEAD -- benchmarks/tasks benchmarks/harness` = **0 lines** (only NEW files added under benchmarks/: the runner + two result JSONs).
3. **Build determinism:** `python3 site/build_site.py` run twice → **byte-identical index.html**; page asserts invariants against committed JSONs (incl. the new failed-task invariant).
4. **Serve check:** `python3 -m http.server -d site` + curl → headline "48/50 on two independent runs, $0.25–$0.45 per suite" renders.
5. **Merge hygiene:** both lineage-A (approved base b762749) and lineage-B (page/templates base 3bea626) strands merged; only conflict was PR_SUMMARY.md (add/add) — resolved as this combined summary; parity-report Addendum 4 amendment + Addendum 5 both present; roadmap Step 0 DONE recorded; g2.4/notebooks intact.

## Reconciliation note

Two Step-0 runs are published on the page, both 48/50: canonical
`symmetric-baseline-deepseek-v4-pro-20260825-195414.json` ($0.4539) and
reproduction `symmetric-baseline-20260825.json` ($0.2461) — headline "$0.25–$0.45
per suite". No "1/50th cost" or any cost-ratio claim anywhere (retracted red
line); hardware floor (G1.5) stated as open item, no placeholders; zero external
assets.
Goal references: G1.3 and G1.5. Four-number impact: this user-requested documentation prepares quality, latency and hardware-floor measurements; it records no new numerical improvement. It preserves the frozen task/grader/baseline boundaries.
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# SwapOS / ModelSwapper

A research runtime for local specialist-model pipelines with Context Capsule handoff.

Start with [the execution package](docs/execution/README.md) for reviewed intent,
source context, a dependency roadmap, technical contracts, and bounded developer/test tickets.

Read [AGENTS.md](AGENTS.md) and [MASTER-PROMPT.md](MASTER-PROMPT.md) before code changes.
The [methodology standard](docs/methodology-standard.md) explains benchmark interpretation;
[STATE.md](STATE.md) records progress and outstanding measurements.

## Local verification

With Python 3.11+, pytest and jsonschema installed, from the repository root:

```bash
python3 -m pytest -q capsule/tests benchmarks/harness/tests pipeline/tests
python3 benchmarks/harness/validate_tasks.py --tasks-dir benchmarks/tasks
```

Actual inference requires the appropriate local runtime and model artifacts. Hardware
certification and performance claims require measured evidence on the stated host.
7 changes: 7 additions & 0 deletions STATE.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# STATE.md — ModelSwapper (SwapOS)

## Review checkpoint — 2026-09-05

Source review confirms implemented `pipeline/` and `router/` code. The historical empty-stub description below is obsolete. G1.3 and G1.5 remain open measurements. See [reviewed context](docs/execution/CONTEXT.md) and [work packets](docs/execution/WORK_PACKETS.md); this documentation establishes no new performance result.

The following sections are historical status notes; use the checkpoint above and the linked source map for current code presence.


## Current state

- **G1.3 lever (issue #16) IMPLEMENTED + MEASURED + reviewer-approved:**
Expand Down
40 changes: 40 additions & 0 deletions docs/execution/CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Code and review context

## Sources

| Path | Responsibility / observation |
| --- | --- |
| `MASTER-PROMPT.md`, `AGENTS.md` | Goals, four metrics, hardware rules and benchmark ownership |
| `pipeline/contracts.py` | `ModelBackend`, `GenerationResult`, `TaskRunResult` |
| `pipeline/loop.py` | `run_task`, capsule handoff, bounded critic feedback, checkpoint naming |
| `runtime/llama_backend.py`, `runtime/overlap_backend.py`, `runtime/overlap.py` | Process and two-slot backend implementations |
| `capsule/capsule.py`, `capsule/compress.py` | Capsule 0.1.0, atomic persistence and separate compression mechanism |
| `router/rules.py` | Deterministic phase routing; inspect actual use before asserting G1.4 coverage |
| `pipeline/run_pipeline.py` | Runner and persisted summary aggregation |
| `benchmarks/harness/`, `benchmarks/tasks/` | Grader, baseline code and frozen 50-task suite |
| `docs/methodology-standard.md`, `docs/symmetric-baseline.md` | Comparison protocol and historical evidence |
| `hardware/tiers.yaml`, `docs/t0-air-RUN.md` | Hardware configuration and uncompleted T0 measurement |

## Findings from source inspection

1. The pipeline and router contain implementation; the older STATE statement calling them empty stubs is stale.
2. `kv_checkpoint_name` includes a model path and file size, but not a content digest. Two different same-size artifacts at the same path can produce the same checkpoint identity. SWAP-01 specifies a correction without claiming a reproduced inference failure.
3. Nonresident generation has a `finally` cleanup; resident-mode cleanup is after the loop. An unexpected exception from grading or downstream processing can bypass that final stop. SWAP-02 specifies lifecycle coverage.
4. CI runs capsule and grader tests, but omits `pipeline/tests`, despite the broader gate in AGENTS. SWAP-02 closes that coverage gap.
5. Compression is deliberately not wired into the loop; do not treat that design choice as an accidental omission.

## Commands and prerequisites

From repository root, with Python 3.11+, pytest and jsonschema installed:

```bash
python3 -m pytest -q capsule/tests benchmarks/harness/tests pipeline/tests
python3 benchmarks/harness/validate_tasks.py --tasks-dir benchmarks/tasks
python3 pipeline/run_pipeline.py --help
```

Real inference requires llama.cpp and explicitly supplied model artifacts. No GPU/model download or external API run is needed for the first two packets. Follow the existing hardware runbook for real measurements, preserving its complete command, model identity and host details.

## Review evidence

The 50-task structural validation passed in this review environment. After installing isolated CPU test dependencies, the capsule, grader and pipeline suites passed: 37 tests. The command used the isolated dependency directory on PYTHONPATH. No GPU run, new cost benchmark, T0 memory measurement or baseline API call was performed. Existing JSON results were inspected as historical evidence.
20 changes: 20 additions & 0 deletions docs/execution/INTENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Intent

Enable a developer to run a reason → code → mechanical review → critic/retry workflow using local specialist models and a portable Context Capsule. Success is measured on the same frozen tasks against an explicitly identified baseline.

## Users and value

The first user is a technical evaluator with local compute and privacy requirements. The immediate deliverable is a reproducible runtime evaluation: code output, test outcome, phase timings, memory observations and a complete run manifest. Installation and licensing follow evidence and a validated user need.

## Outcomes

- Quality: satisfy the G1.2 ratio on the frozen suite; report absolute counts and denominators, including all failed runs.
- Latency: close G1.3 using measured total task time, not only a faster subphase.
- Hardware: independently measure G1.5 on the specified 24 GB machine; an 8 GB or large GPU run is a different evidence lane.
- Economics: preserve actual cost basis and unknown values; the historical 1/50th-cost thesis is not an established result.

## Boundaries

The core uses local inference and specialists at most 32B class. Explicit API baselines are separate experiments. Capsule handoff must remain model-agnostic; a KV checkpoint is an optional optimization for a compatible model and prefix. Frozen oracles are never changed to rescue a candidate.

The master prompt defines the long-term research goals. This package sequences reliability and measurement work within those goals. It does not authorize publication, paid compute, additional hardware or a new commercial commitment.
45 changes: 45 additions & 0 deletions docs/execution/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# SwapOS execution package

Reviewed 2026-09-05 against `379064ea2af77a0deed738296da36337448bde94` on `main`. This is a source review and a proposed delivery plan. A specification describes intended behavior; it does not establish that the behavior already exists. Recorded historical results are not new measurements.

## Read in this order

1. Repository instructions at the root, then [Intent](INTENT.md).
2. [Current code and context](CONTEXT.md), including the verification limits.
3. [Roadmap](ROADMAP.md) to select a dependency-ready work packet.
4. [Technical specification](TECHNICAL_SPEC.md) and the selected [work packet](WORK_PACKETS.md).

Existing product goals, ADRs, schemas and task evidence remain authoritative for their domains. These documents make the next work explicit; they do not silently retire an existing decision. Recheck the target branch and open PRs before executing an old packet.

## Developer and test pair

The coordinator hands both roles the same packet ID, base commit, acceptance criteria, permitted files and fixture contract. One implementer owns the code branch. The verifier independently derives failure cases from the specification and records its own observations. When two agents are unavailable, perform separate implementation and verification passes and disclose that independence was unavailable.

The implementer reads the named sources, reproduces the current behavior, changes only the assigned scope and hands over a commit plus commands. The verifier reads the contract before the implementation explanation, tests success and refusal/error paths, inspects the diff and returns `pass`, `changes_requested` or `blocked` against that exact commit. A test that never ran is `not_run`, never a pass. Do not mark an existing unmet gate complete merely because a packet's preparation is complete.

Use distinct branches/worktrees when executing concurrently. Packets with overlapping owned files run serially unless the coordinator explicitly partitions the edits. The coordinator integrates one reviewed change at a time, reruns the affected integration gate and reconciles status. Tests may add fixtures under their assigned test paths; shared configuration and schema edits have one owner.

## Handoff record

```yaml
packet_id: selected ID
base_commit: exact SHA
implementation_commit: exact SHA
acceptance_results: # one entry for every numbered criterion
- criterion: AC1
outcome: pass | fail | not_run
command: exact command and working directory
evidence: result or repository artifact path
changed_files: []
remaining_risks: []
verifier: independent reviewer identity or independence unavailable
verdict: pass | changes_requested | blocked
```

Keep test data synthetic and generated evidence free of credentials. Work packets authorize their stated code/document scope, not external sends, purchases, model training, production deployment or data migration. Respect the repository's existing authorization boundaries.

## Skills

Repository-scoped skills are in `.agents/skills/swapos-deliver/` and `.agents/skills/swapos-verify/`. Invoke the applicable skill with a packet ID. If the client does not discover repository skills, open its `SKILL.md` explicitly; no personal skill installation is required.

ModelSwapper packets retain the original G-goal IDs. Use the branch conventions in `AGENTS.md` and an ADR for architectural changes. Benchmark tasks, graders and baseline runners remain outside implementation ownership. This documentation prepares quality, latency and hardware measurements; it does not claim a numerical improvement.
Loading
Loading