Skip to content

STATE.md

Latest commit

 

History

History
139 lines (115 loc) · 9.79 KB

STATE.md

File metadata and controls

139 lines (115 loc) · 9.79 KB
Raw

STATE.md — ModelSwapper (SwapOS)

Review checkpoint — 2026-09-05

Source review confirms implemented pipeline/ and router/ code. The historical empty-stub description below is obsolete. G1.3 and G1.5 remain open measurements. See reviewed context and work packets; this documentation establishes no new performance result.

The following sections are historical status notes; use the checkpoint above and the linked source map for current code presence.

Current state

  • G1.3 lever (issue #16) IMPLEMENTED + MEASURED + reviewer-approved: cross-turn KV prefix cache for retry phases — retry-phase prefill 63.7 s → 5.2 s (−91.8%, acceptance bar ≥ 50% MET; paired tasks −92.6%; 7/7 restore hits, 0 misses). Evidence: benchmarks/results/kv-cache-local-4b-20260824-083718.json + docs/g1.3-kv-prefix-cache.md.
  • Honest G1.3 projection: at 27B scale retry prefill is only 0.51% of wall, so this lever moves 2.12× → ~2.11× and CANNOT close the <2× bar alone — G1.3 (#7) remains open. Next levers: retry-storm reduction and native-arch llama builds (documented hypotheses, docs/parity-report-phase1.md Addendum 3).
  • Four-number proof draft landed: docs/four-number-proof.md (reconciled with Addendum 4; publication founder-gated per ENDGAME.md).
  • G2.4 capsule-v1 integration note landed: docs/g2.4-capsule-v1-integration.md; issue #15 CLOSED.
  • Phase 0 — Foundation: complete. Repo skeleton, governance docs, ADRs 0001–0003, Capsule v0 schema + round-trip tests, benchmark harness + 50-task suite, frontier-API baseline run, and first swap-baseline measurements all landed.
  • Hardware reality: development machine is an Apple M3, 8 GB unified memory (sysctl hw.memsize = 8.0 GB) — that is the T4 edge tier, not the T0 target. All Phase 0 swap numbers are measured on this machine as T4-class data; T0 (24 GB) targets remain the trajectory and must be re-measured on real T0 hardware when available.
  • Frontier baseline: deepseek-v4-pro via DeepSeek API (only provider available with an existing key; frontier-class, used strictly for baseline measurement — the core pipeline stays local per §4).
  • Backend: llama.cpp (Homebrew, Metal-capable) installed on the dev machine. Model candidates (official Qwen GGUFs): Qwen3-4B-Q4_K_M (2.5 GB), Qwen3-8B-Q4_K_M (5.0 GB), Qwen3-0.6B-Q8_0 (0.64 GB, router-class).

Phase 0 numbers (measured 2026-08-06, T4 tier: 8 GB Apple M3, llama.cpp Metal)

Metric Run 1 (v1 sampler) Run 2 (v2 RSS sampler) G0.2 T0 target
Cold swap (4B → 0.6B first token) 2.25 s 1.64 s ≤ 8 s ✓
Warm swap (page-cache reload) 2.70 s 4.46 s ≤ 3 s (met in run 1)
Eviction time 0.06–0.48 s 0.07–0.19 s —
Peak RSS (4B Q4_K_M) 1.74 GB 1.53 GB T4 ceiling 6.5 GB ✓
Peak RSS (0.6B Q8) 0.94 GB 0.92–0.94 GB ✓
8B Q4_K_M on 8 GB — FAILS (Metal alloc: failed to fit params to free device memory) T4 floor: 4B-class max

Findings: (1) eviction is nearly free — swap cost is dominated by weight loading; G2.1 (layer-priority streaming) is the right lever. (2) On 8 GB RAM, warm ≈ cold: the page cache cannot hold both models' weights, so the OS evicts the incoming model's cached pages during the outgoing load — warm advantage needs ≥ T0-size RAM. (3) Run-to-run variance 1.6–2.3 s cold / 2.7–4.5 s warm from OS cache state. (4) 8B Q4 does not fit the T4 tier (hardware floor). Raw data: benchmarks/results/swap_baseline-20260806-233738.json, -233855.json (4B↔0.6B), -234241.json (4B↔8B, failure recorded).

Frontier baseline (measured 2026-08-07, deepseek-v4-pro, suite swapos-v1)

  • 48/50 tasks passed (96.0%) — bugfix 17/17, feature 16/17, refactor 15/16
  • Mean 19.3 s/task, est cost $0.15 for the whole suite (~$0.003/task)
  • Fails: feature-08 (9/10), refactor-12 (8/9) — both single-test near-misses
  • Raw data: benchmarks/results/baseline-deepseek-v4-pro-20260807-001937.json
  • Phase 1 parity target derived: ≥ 80% of 96% → pipeline must pass ≥ 76.8% absolute (G1.2)
  • Phase 1 result (2026-08-19, Colab L4, 27B Q4 + 8B Q4, swap-per-phase): 40/50 = 80.0% — G1.2 MET (bar 76.8%). bugfix 12/17, feature 16/17 refactor 12/16; mean wall 40.0s = 2.07× API (bar 2× → G1.3 near-miss by 1.4s), mean load 1.96s, mean evict 0.17s. Report: docs/parity-report-phase1.md. CORRECTED (Addendum 4): this run was pass@1-only — retries rescued 0; 5/10 failures were a code defect.
  • Confirmation run (2026-08-20, Colab RTX PRO 6000 97.9GB, same config, temp 0.2): 39/50 = 78.0% — G1.2 CONFIRMED (two independent sessions/hardware at the operating point: 80.0% + 78.0%). bugfix 14/17, feature 14/17, refactor 11/16; mean wall 44.4s = 2.30× API, mean load 2.18s, mean evict 0.21s. CORRECTED (Addendum 4): 8/11 failures in this run were a code defect (NameError on server-start failure); the 0.6-run temperature conclusion is downgraded (11/15 failures were the same defect). G1.2 CLOSED. G1.3 near-miss stands (2.07× / 2.30× vs 2× bar). G1.5 pending 24 GB Air.

Broken / incomplete

  • G1.3 (#7) open: the <2× wall bar is not met — 2.12× (overlap) / 2.18× (sequential) vs 2× bar. The KV-prefix-cache lever is measured (−91.8% retry prefill) but retry prefill is only 0.51% of 27B wall, so the projected best case is ~2.11× — cannot close alone.
  • G1.5 (#9) open: T0 (24 GB Air) measurement pending — runbook prepared, not yet measured.
  • runtime/ — swap engine v0 exists (subprocess llama-server: load/generate/evict/measure); in-process engine (mmap, layer-priority, pre-fetch) is Phase 2.
  • pipeline/ and router/ — empty stubs by design (Phase 1).
  • Benchmark suite: 50/50 tasks authored + RED/GREEN verified; frontier baseline measured (48/50, 96.0%)
  • No T0 (24 GB) or T1 (48 GB) hardware available to the swarm yet — those numbers are open.

Phase 2 (active) — engineering down the swap

  • G2.3 CLOSED: verdict both halves TRUE at 27B scale, three runs consistent (capsule 8/8 @ 788-803 tok vs naive 7/8 @ 1663-1718 tok vs single 7-8/8 @ 999-2253 tok). Capsule = fastest arm.
  • G2.2 CLOSED (full-50 suite): load/phase 1.995 → 0.806 s (−60%), 41% of phases at zero load; wall 41.99 → 40.83 s (−2.8%; generation- dominated at 27B scale); quality 45/50 vs 47/50 within noise.
  • G2.1 CLOSED: met via overlap (mean paid load 0.806 s < 1.5 s bar; promoted swaps pay 0; local T4 direction 0.851 → 0.063 s).
  • G2.4 CLOSED (#15): compression done (8k budget, sub-linear growth, tests) and the capsule-v1 integration note (docs/g2.4-capsule-v1-integration.md) landed + approved — compression stays intentionally unwired from the running loop (by design, per the note); it is the designated overflow mechanism for long-horizon runs.
  • G1.2 RE-CONFIRMED at 94.0% (sequential 47/50, 8192 ctx + bounded feedback): bugfix 17/17, feature 17/17 — 97.9% of the frontier baseline. Overlap 45/50 (90.0%). The 8192 context + feedback bound unlocked the retry loop the 4096 config was choking.
  • G1.3 lever (issue #16) IMPLEMENTED + MEASURED + reviewer-approved: cross-turn KV prefix cache for retry phases — retry-phase prefill 63.7 s → 5.2 s (−91.8%; acceptance bar ≥ 50% MET; paired tasks −92.6%; 7/7 restore hits, 0 misses). Evidence: benchmarks/results/kv-cache-local-4b-20260824-083718.json + docs/g1.3-kv-prefix-cache.md.
  • Honest G1.3 projection: at 27B scale retry prefill is only 0.51% of wall, so this lever moves 2.12× → ~2.11× and CANNOT close the <2× bar alone — G1.3 (#7) remains open. Next levers: retry-storm reduction and native-arch llama builds (documented hypotheses, docs/parity-report-phase1.md Addendum 3).
  • Issues: G2.1 (#12), G2.2 (#13), G2.3 (#14), G2.4 (#15) CLOSED; G1.3 (#7) + G1.5 (#9) open.

Blockers

  • None. (T0/T1 hardware numbers are scheduling items, not blockers — mechanics are measured on T4 hardware now.)

2026-08-28 landing (wt/land-2026-08-28 → main)

  • Roadmap Step 0 landed (reviewer-approved, t_d24fd5e4): benchmarks/run_symmetric_baseline.py + evidence benchmarks/results/symmetric-baseline-20260825.json + docs/symmetric-baseline.md — falsification verdict: API-with-identical-loop 48/50 ≠ ~50/50; parity claim survives (47 vs 48, p≈1.0). pass@1 p-value amendment (t_fd73b8df) landed too (four-number-proof §2.1 + parity-report Addendum 4 item 3: exact McNemar p=0.016).
  • Roadmap gate 4 landed (review t_23f2732a fix applied in this landing): docs/methodology-standard.md + site/ benchmark page (deterministic build, zero external assets, both Step-0 runs reconciled "$0.25–$0.45 per suite"; build asserts pipeline failed-task ids against the parsed JSON). Required fix: pipeline third failure corrected to refactor-02 (was refactor-11) in PIPELINE_DOC.failed / LOOP_MCNEMAR.api_only_tasks / Addendum 5 cell.
  • Engine 1 templates landed (reviewer-approved, t_2e6095e4): templates/validation-program/ (one-pager + validation report + README; roadmap price bands only; EXAMPLE blocks with measured numbers).
  • Gates re-run after merge: 37 passed, 50 tasks OK, sacred diff 0 lines, build byte-identical across runs, serve 200.
  • G1.5 (T0 24 GB Air) remains an open item on the page (honest, no placeholders).

Test command

uv run --with pytest pytest capsule/tests benchmarks/harness/tests

Run command

# Frontier-API baseline (needs DEEPSEEK_API_KEY from ~/supplementary/.env)
uv run --with pytest benchmarks/harness/run_baseline.py --model deepseek-v4-pro

# Swap baseline on local hardware (needs llama.cpp + downloaded GGUFs in models/)
python3 runtime/swap_runner.py --model-a models/Qwen3-4B-Q4_K_M.gguf --model-b models/Qwen3-8B-Q4_K_M.gguf