[codex] Preserve SPEC-028 benchmark evidence

[Augustas11]

Summary

This is a handoff PR for SPEC-028 after local hardware testing showed that we need larger Apple Silicon infra before making an enablement call.

It intentionally does not enable speculative decoding by default. SPEC-028 remains operator-gated by draft_model; if no draft model is configured, macprovider-cli keeps using the existing target-only path.

What changed

• Added macprovider-cli spec028-benchmark to measure baseline vs speculative decoding on the actual macprovider/MLX/Metal runtime path.
• Added benchmark recommendation gates for:
  • median speculative/baseline TPS ratio
  • p95 total latency ratio
  • p95 TTFT ratio
  • speculative acceptance rate
  • sustained TPS ratio when a sustained window is requested
• Relaxed tokenizer artifact matching for same-vocabulary fast-tokenizer snapshots:
  • ignore top-level chat_template in tokenizer_config.json
  • when tokenizer.json exists, ignore redundant vocab.json / merges.txt packaging differences
  • when tokenizer.json is absent, keep vocab.json / merges.txt binding for slow-tokenizer layouts
• Added specs/SPEC-028-BENCHMARK-HANDOFF.md with findings, commands, test evidence, and the rented Mac validation plan.

Local findings

Local host: Apple Silicon M5, 32 GB unified memory, release macprovider-cli, context cap 8192 tokens.

Completed matrix:

Pair	Fixture	Baseline TPS	Spec TPS	TPS ratio	Sustained ratio	Acceptance	Result
Llama 3.2 3B / 1B	spec028-small-air-short-chat-v1	56.12	51.46	0.917x	0.871x	54.5%	Do not enable
Llama 3.2 3B / 1B	spec028-small-air-streaming-check-v1	52.15	44.84	0.860x	0.832x	54.1%	Do not enable
Qwen2.5-Coder 7B / 1.5B	spec028-code-iso8601-v1	24.74	31.42	1.270x	1.322x	96.5%	Throughput win, TTFT fail

Qwen2.5-Coder 7B / 1.5B showed real throughput lift and high acceptance, but failed the first-token latency gate:

• tpsRatio: 1.2698843002639708
• sustainedRatio: 1.3220061447492875
• acceptanceRate: 0.9645390070921985
• p95LatencyRatio: 0.7948502457691152
• ttftP95Ratio: 1.5052631578947369
• recommendEnable: false
• reason: p95 TTFT ratio 1.505 > 1.000

32B / 7B findings

The original draft_model_tokenizer_mismatch on Qwen2.5-Coder 32B / 7B was our policy being too strict, not a real tokenizer vocabulary mismatch. The snapshots matched on tokenizer.json, vocab.json, added_tokens.json, and special_tokens_map.json; the difference was redundant merges.txt packaging in the 7B snapshot.

After the tokenizer policy fix, 32B / 7B no longer failed immediately, but it did not complete on the 32 GB local host. A 16-token smoke was also stopped after several minutes with an empty output file.

Observed local footprint:

• Qwen2.5-Coder 32B 4-bit snapshot: about 17 GB
• Qwen2.5-Coder 7B 4-bit snapshot: about 4 GB
• Combined weights: about 21 GB before MLX runtime memory, KV cache, Metal buffers, allocator overhead, and OS pressure

Conclusion: 32B / 7B is not a measured loss. It is not locally benchmarkable on this 32 GB host without narrower diagnostics or more unified memory.

Rented infra plan

Generic VPS/Linux/NVIDIA cloud is not sufficient for the main product decision because this implementation is specifically mlx-swift + Metal on Apple Silicon. It can validate a different algorithmic stack, but not the live macprovider runtime path.

Use rented Apple Silicon instead:

• Minimum: 64 GB unified memory
• Preferred: 128 GB unified memory
• Ideal: 192 GB or 256 GB unified memory
• Candidate provider categories researched:
  • MacStadium dedicated Apple Silicon Mac cloud: https://macstadium.com/bare-metal-mac
  • MacWeb Mac Studio rental listings: https://macweb.com/macstudio
  • MacinCloud dedicated/on-demand Mac servers: https://www.macincloud.com/
  • Apple Mac Studio hardware tiers: https://www.apple.com/mac-studio/

Required rented-host matrix:

• Qwen2.5-Coder 7B target / 1.5B draft
• Qwen2.5-Coder 32B target / 7B draft
• Llama 3.2 3B target / 1B draft as known negative control

Suggested command shape is in specs/SPEC-028-BENCHMARK-HANDOFF.md.

Tests run

swift test --package-path phase3-binary \
  --filter Spec028PlumbingTests/testTokenizerArtifactFingerprintBindsTokenizerFiles \
  --filter Spec028PlumbingTests/testSpec028BenchmarkEvaluationGatesTTFTRegression

Result: passed, 2 tests, 0 failures.

swift build --package-path phase3-binary -c release --product macprovider-cli

Result: passed. Existing Swift concurrency/deprecation warnings remain.

Follow-up recommendation

Keep SPEC-028 disabled by default until rented Apple Silicon testing proves a positive matrix by model pair and RAM tier. If Qwen2.5-Coder 7B / 1.5B keeps the throughput win but TTFT still fails, test first-token mitigation next: warm draft/target prefill, defer speculative decoding until after token 1, and cache/reuse prompt preparation where safe.