draft(qwen4): add hardware-fit runtime and modular artifact path

docs/FREETOKEN_STAGE7F_STEP9B_EXECUTOR.md

@@ -0,0 +1,108 @@
+# FreeToken Qwen4 Stage 7F Step 9B executor
+
+Gate: `FREETOKEN-QWEN4-001 / STAGE7F-STEP9B-EXECUTOR-IMPLEMENTATION`
+
+This branch adds the missing execution controller around the accepted Stage 7E
+component writers. It does not change Q3, FTEXPERT1, active FTW v1, or the
+pinned PR257 runtime. Stage 7F itself is a zero-real-payload gate.
+
+## Safety model
+
+`Step9BExecutor` is dry-run by default. A network response body is unreachable
+unless both execution mode and explicit body permission are enabled. The CLI
+therefore requires `--execute --allow-network-body`; `--execute` alone fails
+before acquisition. The frozen manifest is the only source-file authority.
+
+The downloader:
+
+- resolves the immutable Hugging Face commit and checks file size, ETag, LFS
+  OID, Xet identity, and bounded Safetensors-header identity before transfer;
+- streams directly to `<filename>.partial` on Z:, with no Hugging Face cache;
+- binds every partial to a durable identity sidecar and resumes only with an
+  exact `Range` plus `If-Range` contract;
+- rejects an ignored range, malformed `Content-Range`, missing or changed body
+  ETag, oversized or undersized body, hash mismatch, and header mismatch;
+- persists actual response-body bytes before accepting each chunk, including
+  bytes received in a failed or over-cap attempt;
+- enforces no more than two active response bodies and provides cancellation;
+- promotes a source atomically only after all identities validate, then writes
+  an atomic source receipt.
+
+The controller rejects source payload without matching durable transfer-budget
+provenance. This prevents an orphan partial or final source from bypassing the
+upstream byte cap after a restart.
+
+## Stage controller
+
+The controller exposes explicit boundaries for:
+
+1. B1: nine metadata/config/tokenizer files and a bound receipt.
+2. B2: ten PLE source files, the accepted 128-segment Q3 writer, reader reopen,
+   exact 22,400,107,520-byte validation, precommit, and final receipt.
+3. B3: 48 ordered expert transactions, four source files per layer, accepted
+   FTEXPERT1 writer, exact 1,419,776,000-byte extent, FileExpertSource reopen,
+   and an independent layer receipt.
+4. B4: four BF16 source files, accepted active conversion, exact
+   4,804,403,200-byte FTW v1 contract, and receipt.
+5. B5: final modular-manifest validation and exact 95,353,758,720-byte known
+   component reconciliation.
+6. C6: a process-isolated static reopen using only the pinned PR257 worktree.
+
+Existing valid targets are never trusted by name or receipt alone. The
+controller rehashes and reopens them, then can recover a missing final receipt.
+Accepted component writers own atomic target promotion; the controller adds a
+validated precommit document followed by an atomic final receipt. Incomplete
+component partials are never promoted by the controller.
+
+Every component receipt binds the builder commit, runtime commit where
+relevant, source revision, source-inventory fingerprint, receipt-backed source
+file hashes, target length/hash, format, and validation results.
+
+## Capacity and environment gates
+
+The disk gate is restart-aware. It computes:
+
+`remaining verified source + remaining target + common conversion allowance + 64 GiB reserve`
+
+rather than requiring the original peak-free threshold after every completed
+file. With an empty source and target, the formula exactly reconciles to
+309,257,827,893 bytes. Physical host availability is measured independently
+and must remain at least 6,442,450,944 bytes; pagefile use is recorded but never
+counted as model capacity.
+
+The executor verifies the isolated `_pinned_tensor` extension hash and, in real
+execution preflight, runs a bounded HostBank pin/device-alias probe. Triton,
+TVM-FFI, Torch, compiler, and temporary caches are forced to the supplied Z:
+toolchain root.
+
+Source retirement is explicitly rejected by this controller. The real handoff
+uses `source_retirement_authorized=false` and retains every verified source.
+
+## C6 isolation
+
+C6 launches a fresh child process with `PYTHONPATH` beginning at exactly the
+pinned PR257 runtime source. It performs static-only checks of:
+
+- hardware-fit markers and active FTW verification;
+- Q3PLEFileTable construction;
+- all 48 expert metadata entries and the 12/36 placement policy;
+- QD4 FileExpertSource construction for each file-tier layer;
+- graph-disabled and prefill-overlap-disabled policy, including forced-graph
+  rejection;
+- a tiny resident HostBank pin/device-pointer probe.
+
+C6 does not instantiate the complete model, resident expert banks, production
+GPU cache, KV cache, a model layer, a forward pass, generation, or a server.
+
+## Validation
+
+The Stage 7F suite uses a local deterministic HTTP server for clean transfer,
+interruption, exact resume, ETag drift, ignored and malformed ranges,
+over/undersized bodies, hashes, Safetensors headers, cancellation, concurrency,
+failure isolation, receipt recovery, and byte-cap behavior. A hard kill-switch
+test proves dry run cannot issue a real Hugging Face body GET. The full frozen
+manifest test plans 9 metadata files, 206 weight files, 48 expert boundaries,
+135,195,303,851 weight bytes, and the complete B1-B5/C6 order.
+
+The production command is intentionally emitted only in the separately
+regenerated Step 9 handoff. Stage 7F does not execute it.

docs/FREETOKEN_STAGE7J_Q3_ACCELERATION.md

@@ -0,0 +1,51 @@
+# Stage 7J — Byte-Exact Q3 Acceleration
+
+Stage 7J replaces only the production Q3 encoder's scalar row loop.  The
+Q3_PLE_32 format, 128 logical segment boundaries, per-segment hashes, global
+weight scale, manifest schema, source inventory, and PR257 runtime remain
+unchanged.
+
+## Reason for remediation
+
+The first real conversion measured approximately 198 kB/s and projected about
+31 hours for the 22,400,107,520-byte Q3 extent while using roughly one CPU
+core.  The run was stopped before its first component receipt.  Its incomplete
+target partial is not a valid component; all ten verified PLE source files and
+the lifetime transfer ledger remain unchanged.
+
+## Selected implementation
+
+The production Safetensors path now reads bounded 131,072-row chunks and
+executes the existing two-pass codec as batched Torch operations.  CUDA is
+selected when available and the same batched arithmetic has a CPU fallback.
+The scalar `quantize_block()` and `quantize_row()` functions remain the
+reference authority.
+
+Exactness protections include:
+
+- the historical FP8-to-FP32 source conversion boundary;
+- float64 scale and refinement arithmetic;
+- sequential `cumsum` reductions matching the scalar left-fold order;
+- per-block early-convergence state;
+- the existing integer round-to-nearest-even BF16 scale conversion;
+- final requantization against the stored BF16 scale;
+- unchanged low-bit-first 3-bit packing;
+- ordered writes and unchanged logical segment/hash construction.
+
+## Evidence
+
+- 50,000 deterministic FP8-origin rows: scalar and CUDA output byte-identical;
+- SHA-256 of the 3,500,000-byte differential output:
+  `dfe54d8a9d122d356cb46b21a7fda6e2daea7a91b404bcd4a808a9ab9e012f39`;
+- measured differential speedup: 125.9x;
+- 1,048,576-row sustained synthetic stream: 425,479 rows/s including source
+  generation, host/device transfer, output materialization, and hashing;
+- sustained projected encoding time: 12.53 minutes;
+- 131,072-row warmed batch: 1,320,719 rows/s, projected 4.04 minutes for codec
+  execution alone;
+- peak measured GPU allocation: 1,329,070,080 bytes;
+- focused executor/Q3/resume regression matrix: 107 passed;
+- Q3 writer suite: 20 passed.
+
+The real Q3 conversion is not restarted until this isolated branch is committed
+and the resume handoff pins that exact commit.

docs/FREETOKEN_STAGE7K_ACTIVE_FTW_TERMINAL_PADDING.md

@@ -0,0 +1,46 @@
+# Stage 7K — active FTW terminal-padding remediation
+
+## Outcome
+
+Stage 7K preserves the frozen active FTW v1 extent of `4,804,403,200` bytes without changing a tensor or introducing artifact v2. The production tensor stream naturally ends at `4,804,399,104` bytes. The exact `4,096`-byte difference is represented as one zero-filled terminal compatibility page.
+
+## Root cause
+
+The accepted analytical layout included the two-byte global PLE scale in the active resident-byte calculation. FTW alignment turns that scalar into one 4-KiB page. Production correctly binds the global PLE scale once in `ple-q3.json`, so the active tensor stream contains no duplicate PLE-scale tensor and ends one aligned page earlier.
+
+The real candidate contains 1,698 tensor entries. Its final tensor ends exactly at byte `4,804,399,104`. Existing tensor payloads reconcile as:
+
+```text
+packed NVFP4 weights       1,772,748,800
+FP8 block scales             221,593,600
+FP16 row globals               4,162,176
+protected tensors          2,804,402,200
+raw tensor bytes           4,802,906,776
+inter-entry alignment          1,492,328
+tensor-stream extent       4,804,399,104
+terminal compatibility page       4,096
+frozen active extent       4,804,403,200
+```
+
+The page is not a tensor and does not represent a second PLE scale. FTW readers traverse indexed tensor entries and ignore the reserved tail.
+
+## Recovery contract
+
+`ensure_ftw_terminal_padding()` accepts only FTW v1 with 4-KiB alignment and only these two states:
+
+- the exact unpadded production extent, followed by appending and fsyncing one zero page;
+- the exact frozen extent with an already published or recoverable zero page.
+
+It rejects nonterminal shard disagreement, nonzero tail bytes, partial pages, oversize tails, unknown extents, malformed geometry, and incompatible format/alignment. Publication order is shard append and fsync, then bounded Windows-safe atomic index replacement. A crash after append but before index replacement is recovered by validating and adopting the exact zero tail. Repeated recovery is idempotent.
+
+Only the final shard `nbytes` and index `total_bytes` change. Tensor keys, order, offsets, lengths, dtypes, shapes, kinds, and all pre-tail bytes remain unchanged.
+
+## Gate boundaries
+
+- Source bodies remain retained and are not downloaded again.
+- The lifetime transfer ledger remains `135,252,480,565 / 135,252,480,565`.
+- Q3 and all 48 expert sidecars remain unchanged.
+- PR257 runtime commit `0307a6114c57b0efc61bc17688f3288fe0bf1dc7` remains unchanged.
+- B5 and isolated C6 may proceed only after B4 publishes a valid receipt at the frozen extent.
+- No full model, inference, generation, serving, or Step 10 work is authorized.
+

docs/plans/FREETOKEN-QWEN4-001-STAGE7F.md

@@ -0,0 +1,22 @@
+# FREETOKEN-QWEN4-001 Stage 7F execution plan
+
+Task: `STAGE7F-STEP9B-EXECUTOR-IMPLEMENTATION`
+
+Definition of done: a dry-run-default, restartable Step 9B controller coordinates immutable source acquisition, B1-B5 conversion, receipts, stop gates, retained-source policy, and isolated C6 validation without changing frozen component formats or the pinned PR257 runtime. Synthetic transport/controller tests and the real 206-row manifest dry run pass with zero upstream model payload bytes, then one local commit and a non-executed Step 9 handoff are produced.
+
+Dependencies: accepted builder `b64a342ea8e5ccac39a7619747b4a7b3e37466f3`; runtime `0307a6114c57b0efc61bc17688f3288fe0bf1dc7`; source manifest revision `7b719225242aacd3dbd3f9407468c2ee9a9d2594`.
+
+Validation: focused downloader, receipt/recovery, staged-controller, C6 subprocess, and accepted component-writer tests; full real-manifest dry run; `python -m compileall -q python/freetoken`; `git diff --check`; commit parent and clean-worktree checks.
+
+| Step | Status | Work |
+|---|---|---|
+| F0 | DONE | Verify authorities, PR257 head, zero-payload state, and create isolated branch. |
+| F1 | DONE | Map accepted writer/manifest/runtime seams and freeze executor contract. |
+| F2 | DONE | Implement immutable transport, downloader, partial identity, byte budget, receipts, and logging. |
+| F3 | DONE | Implement B1-B5 state controller, disk/RAM gates, retained-source policy, and CLI. |
+| F4 | DONE | Implement isolated C6 static-validation subprocess controller and environment check. |
+| F5 | DONE | Add synthetic HTTP, range/cap/cancellation, receipt crash-recovery, and controller tests. |
+| F6 | DONE | Run synthetic end-to-end and real 206-row no-body dry run; capture required evidence. |
+| F7 | DONE | Run accepted regressions, compileall, diff checks, and adversarial final review. |
+| F8 | DONE | Commit the executor branch and regenerate the non-executed Step 9 handoff. |
+| F9 | DONE | Verify clean worktrees, zero payload, evidence completeness, and final gate decision. |

docs/plans/FREETOKEN-QWEN4-001-STAGE7G.md

@@ -0,0 +1,35 @@
+# FREETOKEN-QWEN4-001 / Stage 7G Storage Identity Remediation
+
+## Definition of done
+
+Separate Git blob, LFS OID/metadata ETag, Xet hash, and transport-body ETag throughout the Step 9B manifest, downloader, partial/resume state, and receipts; preserve the lifetime transfer ledger at 14,137 bytes; migrate the three existing source receipts without network body transfer; pass a body-disabled real restart rehearsal and the complete 215-file dry run; commit one clean remediation commit without changing any binary artifact/runtime contract.
+
+## Dependencies and hard boundaries
+
+- Parent executor commit: `6b62c5057ae13deb76bf92a80e8f925e90d55929`.
+- Runtime remains byte-identical at `0307a6114c57b0efc61bc17688f3288fe0bf1dc7`.
+- Source inventory fingerprint remains `8572d200e31b344faff0fdaf0dc72aa4726c1f062443d4109531b62ca63f66eb`.
+- Lifetime transfer ledger remains `14,137 / 135,252,480,565` bytes.
+- HEAD/API metadata only; any new source response-body byte is a stop condition.
+- Q3, FTEXPERT1, active FTW, PR257 runtime, placement, conversion arithmetic, and target byte totals are out of scope.
+
+## Execution plan
+
+| Step | Status | Validation |
+|---|---|---|
+| Preserve and hash v1 manifest, receipts, transfer ledger, source files, and accepted worktrees | DONE | Exact SHA/length/state inventory |
+| Freeze all nine metadata identities and representative BF16/PLE/EXPERT identities using metadata-only queries | DONE | Commit/size/Git/LFS/Xet semantic checks; zero body bytes |
+| Implement manifest v2 identity fields and generation/migration | DONE | 9 metadata + 206 weights; totals/fingerprint unchanged |
+| Implement executor v2 metadata, body ETag, partial, resume, and receipt semantics | DONE | Focused Git/LFS/Xet and transport tests |
+| Revalidate three existing files and migrate receipts with predecessor hashes | DONE | Local full hashes passed; zero new body bytes; ledger unchanged |
+| Run body-disabled real restart rehearsal and full 215-file dry run | DONE | Reached tokenizer body boundary; no Xet mismatch; no body GET |
+| Run regressions, compileall, diff-check, source-delta audit, and adversarial review | DONE | 168 non-overlapping tests passed; compileall/diff-check passed; independent findings remediated |
+| Create evidence, regenerate handoff, commit, and verify clean authorities | DONE | One commit with exact parent; runtime/history unchanged |
+
+## Validation commands
+
+- Focused and complete pytest through the accepted Stage 7F Python environment.
+- `python -m compileall -q python/freetoken`
+- `git diff --check`
+- Stage 7G executor `--execute` without `--allow-network-body` against the actual restart state.
+- Stage 7G executor `--dry-run` against all 215 manifest rows.

docs/plans/FREETOKEN-QWEN4-001-STAGE7H.md

@@ -0,0 +1,31 @@
+# FREETOKEN-QWEN4-001 Stage 7H execution plan
+
+Task: `STAGE7H-STEP9B-WINDOWS-ATOMIC-SIDECAR-RECOVERY`
+
+Definition of done: the accepted Stage 7G executor is extended only with bounded Windows atomic-JSON replacement retries and fail-closed orphan checkpoint recovery; the existing first-PLE partial is preserved byte-for-byte and its exact orphan identity is adopted without any network response-body bytes; the lifetime ledger, manifest, runtime, and B1 state remain unchanged; the body-disabled real restart derives the exact future range plan; all focused regressions pass; and a clean local Stage 7H commit plus pinned resume handoff and evidence are produced.
+
+Dependencies:
+
+- Historical Stage 7G executor at `313c043861df5c57dfd7c2f98ec168dc7a631d28` remains clean.
+- Manifest v2 SHA-256 remains `8e4074cd1a8950bfb19ebdfdd4c5154b66db3ed538ba99fe41221d1be9361e74`.
+- PR257 runtime remains `0307a6114c57b0efc61bc17688f3288fe0bf1dc7` and clean.
+- Existing B1, partial body, sidecars, and transfer ledger remain available and unmodified until preservation evidence is captured.
+- No body GET, Range request, Q3 conversion, or subsequent source acquisition is permitted.
+
+Validation commands:
+
+- Focused Stage 7F/7G/7H executor tests under `tests/checkpoint`.
+- Z:-backed real Windows lock integration and repeated-publication stress tests.
+- Body-disabled execution against the actual retained source state.
+- `python -m compileall -q python/freetoken`.
+- `git diff --check`.
+- Post-recovery hashes/sizes for the partial, ledger, manifest, B1 receipt, runtime authority, and recovered sidecar.
+
+Steps:
+
+1. `DONE` — Freeze authorities and preserve pre-recovery evidence, including a full partial-body recovery fingerprint.
+2. `DONE` — Audit executor-owned handles and map every orphan-adoption invariant.
+3. `DONE` — Implement bounded Windows replace retry and fail-closed orphan recovery.
+4. `DONE` — Run synthetic retry, Windows lock, stress, recovery, identity/resume, receipt, and transfer-budget regressions.
+5. `IN PROGRESS` — Adopt the independently validated real orphan and run the body-disabled actual-state restart rehearsal.
+6. `PENDING` — Produce Stage 7H evidence, commit locally, regenerate the Step 9B resume handoff, and verify clean closeout.

docs/plans/FREETOKEN-QWEN4-001-STAGE7I.md

@@ -0,0 +1,37 @@
+# FREETOKEN-QWEN4-001 Stage 7I plan
+
+Task: `STAGE7I-STEP9B-PLE-LAYER-SELECTOR-REMEDIATION`
+
+## Definition of done
+
+- Preserve executor commit `d391b4a6e6a31ffa206f7c8339920e2130e174f3`, runtime commit `0307a6114c57b0efc61bc17688f3288fe0bf1dc7`, acquisition manifest v2, all 19 acquired source files, and lifetime transfer ledger `51,257,444,615 / 135,252,480,565`.
+- Make the Step 9B executor validate and select the frozen production PLE source layer represented by the immutable local source index instead of requesting nonexistent layer 2.
+- Fail closed unless the index contains exactly `shard_0..shard_127` plus the global `weight_scale` under the expected production layer.
+- Prove the correction with synthetic/index-only tests and a body-disabled real-state rehearsal. Do not construct the real Q3 artifact.
+- Create one local executor commit and regenerate the resume handoff. Do not resume network body acquisition or B3.
+
+## Dependencies
+
+- Parent executor commit: `d391b4a6e6a31ffa206f7c8339920e2130e174f3`.
+- Immutable source revision: `7b719225242aacd3dbd3f9407468c2ee9a9d2594`.
+- Acquisition manifest SHA-256: `8e4074cd1a8950bfb19ebdfdd4c5154b66db3ed538ba99fe41221d1be9361e74`.
+- Existing local B1/B2 source and receipts; no new response-body transfer.
+
+## Steps
+
+1. `DONE` — Verify authorities, manifest, ledger, and acquired source state; create isolated Stage 7I worktree.
+2. `DONE` — Implemented the smallest explicit PLE selector validation and focused regressions.
+3. `DONE` — Ran focused executor/Q3 tests, relevant regression suite, compileall, and diff-check.
+4. `DONE` — Completed independent read-only review and index/header-only real-state rehearsal.
+5. `DONE` — Prepared evidence and the local commit for the regenerated resume handoff and clean closeout.
+
+## Validation commands
+
+```text
+python -m pytest -q tests/checkpoint/test_step9b_ple_selector.py
+python -m pytest -q tests/checkpoint/test_step9b_executor.py tests/checkpoint/test_step9b_executor_contract.py tests/checkpoint/test_step9b_identity_v2.py tests/checkpoint/test_step9b_windows_atomic.py tests/checkpoint/test_q3_ple_writer.py
+python -m compileall -q python/freetoken
+git diff --check
+```
+
+The real-state rehearsal must omit `--allow-network-body` and must stop before Q3 construction.