feat(cache): add exact semantic session anchors

[PhilipJohnBasile]

Scope

Exact semantic SessionBank anchors, extracted from #336: complete-message prefix planning with strict token-prefix validation, bounded anchor selection, prefill-boundary propagation across cold/exact/near-prefix paths, and request-local planning behind MTPLX_SEMANTIC_ANCHORS=1.

Current-baseline blocker resolved

Head dfe28bf94fb62d35ba305d3b764c044c880f0905 now includes exact upstream 21be78b3f51820eecef020e5e4855c0715eaf9a5 through a normal merge. The feature and replay harness can now be evaluated on the current correctness baseline, rather than comparing against the older pre-turn-boundary implementation.

Readiness run 34069355260, macOS ARM64 / Python 3.11, real MLX imports:

• 562 tests passed, zero failures/errors/skips across semantic-anchor planning/replay, SessionBank selections, OpenAI serving, dashboard endpoints and import boundaries.
• Compilation and diff checks passed.
• The exact tested merge was fast-forwarded to this branch. Audit helpers/workflows remain outside the PR.

Real-model acceptance gate remains explicit

The requested same-build real-model multi-turn OFF/ON performance comparison is not yet measured. These tests do not establish improved cached-token recovery or TTFT for the 27B/128K tool-turn case. That distinction is retained rather than substituting small-model or mocked numbers for the requested hardware receipt.

The tested harness is scripts/bench_semantic_anchor_replay.py, with fifteen dedicated tests and protocol in docs/validation/semantic-anchor-ab.md. It replays identical frozen complete requests, requires uncached initial state and explicit final cached-token usage, distinguishes first generated delta from first visible content, rejects incomplete/error streams, verifies request/prompt-count consistency and output parity, and writes atomic non-overwriting receipts. Manifests are labeled operator attestations, not server-verified flag claims.

Required experiment: the same current feature build with only the anchor flag changing, identical pinned weights/tokenizer/runtime settings, fresh separate server/SSD banks, repeated balanced arm order, and both plain-append and long tool-result turns over the same base. Report every warm turn's cached tokens and TTFT, including zero gains, regressions and parity failures. Existing block/generation/turn-anchor benefits must not be attributed to this feature.

The PR remains non-draft and ready for source review. The current-main prerequisite and regression-test work are complete; the maintainer's empirical acceptance gate is not represented as cleared.

Historical planner-only measurements

The prior 1,000-iteration fixtures at 32/128/512 messages measured p50 planner cost 0.751/3.263/13.206 ms. These are historical CPU/control-path costs, not inference or cache-hit evidence. No memory governor, expert telemetry, policy hooks, dashboard code or generated assets are included.

[snagnever]

A boundary case for the semantic-anchor path, with a repro it should cover.

On stock 2.10.0 (no semantic anchors), a tool turn at 128K re-prefills the whole
prompt, while a plain append over the same base restores 99%. Same base prefix, same
~1k suffix length:

shape	cached/prompt
append (base + ~1k plain suffix)	0.99
tool_turn (base + tool-result turn, ~1k suffix)	0.00

The tool turn's commit fails, so no clean anchor for the base survives:

store-on-prefill: len=126688 cached=0 restore=cold
session_commit: committed=false,
  reason=retokenized_prefix_older_than_session,
  unsafe_reason=stop_token_boundary_mismatch

The re-serialized tool-call markup shifts the token boundary, so exact-restore and
near-prefix both fall through to cold. The append case keeps its boundary and restores
via block_prefix_boundary_clone.

Does this PR's complete-message prefix planning make the message boundary before a tool
turn a valid anchor, so the base restores across the tool turn? That is the case #121
closed on block-overlap salvage, but at 128K the tool turn gets zero salvage
(cached=0), so a client that calls a tool every turn pays a full ~870s re-prefill.
The #383 cross-session preemption is a separate factor. I ruled it out as the sole
cause here: MTPLX_POSTCOMMIT_WAIT_TIMEOUT_S=30 leaves the tool_turn miss
byte-identical.

I can build this branch and run the repro with MTPLX_SEMANTIC_ANCHORS=1 to confirm
whether it restores the base across the tool turn. Happy to share the full debug log or
a minimal mtplx serve repro.

Environment: MTPLX 2.10.0, M4 Max 128 GB, macOS 26.5,
Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed, --context-window 131072 --ssd-session-cache on.

[youssofal]

Update: handled on our side, nothing for you to do. The commits authored under the misspelled basilcom.com domain were rewritten to your real address, 14 commits now carry your linked identity. The graph will reflect it as GitHub reindexes. Note tonight's history rewrite changed hashes, so open branches need a rebase onto current main.

[PhilipJohnBasile]

thanks for the headsup. I ended up buying the domain and doing an alias on that address.

…

On Tue, Sep 1, 2026 at 3:56 AM Youssof Altoukhi ***@***.***> wrote: *youssofal* left a comment (youssofal/MTPLX#350) <#350 (comment)> Attribution note while these are in review. Main carries 8 commits authored as ***@***.*** that are not connected to your GitHub account, so the contributors graph undercounts you. Add ***@***.*** under Settings, Emails and verify it, and GitHub rebuilds the graph automatically, usually within a day. Docs: https://docs.github.com/en/account-and-profile/how-tos/contribution-settings/troubleshooting-missing-contributions — Reply to this email directly, view it on GitHub <#350?email_source=notifications&email_token=AAK2UR7T3IVWFSODND3T4U35MZ6J7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBZGA3TQMZWGU32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5490783657>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK2UR3APXC2ZCVNIHR2G5L5MZ6J7AVCNFSNUABGKJSXA33TNF2G64TZHMYTEMRWHEYTCMRUGU5US43TOVSTWNJSGUZTIMJSGM2TNILWAI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[youssofal]

Thanks for keeping the claim honest: the planner cost is measured, the cache-hit effect is not, and the cache-hit effect is the number that decides this. The bank already anchors on block-prefix boundaries, 512-token intervals, generation boundaries and, since 2.11.2, the turn boundaries a session generated from (#446). So the question for semantic anchors is what they restore that those do not.

When you have the paired real-model multi-turn run (warm-turn cached tokens and time to first token, with and without MTPLX_SEMANTIC_ANCHORS=1, on the same transcript), I will run the same transcript here and compare the two anchor sets side by side. On hold until then.

[PhilipJohnBasile]

@youssofal Agreed that the deciding measurement is incremental restoration beyond the current block/generation/turn anchors, not planner cost. I have not produced the real-model A/B yet, so the benchmark hold remains.

I added a reproducible frozen-transcript replay/compare harness and protocol at 892a0d3:

• scripts/bench_semantic_anchor_replay.py
• tests/test_semantic_anchor_replay.py
• docs/validation/semantic-anchor-ab.md

It sends identical complete request bodies in both arms, requires an uncached first turn and explicit final cached-token usage, separates first generated-delta TTFT from visible-content TTFT, rejects truncated/error streams, and compares output hashes and prompt-token counts before interpreting timing deltas. Role-only events do not start TTFT. It records hashes and metrics rather than raw transcript/output text and atomically refuses to overwrite existing receipts.

The protocol requires one feature-capable build containing the current #446 correctness baseline, separate fresh server/SSD banks, the same pinned model/tokenizer and runtime settings, and repeated balanced OFF/ON ordering. The plain-append versus long tool-turn case reported by @snagnever is included in the required workload, not claimed as reproduced here.

Validation performed: all 15 no-model harness tests pass, including a loopback fake-server replay; both published Python blobs exactly match the tested local files, and byte compilation passes. These validate measurement plumbing only. No 27B/128K model, cache-hit gain, or real-model TTFT improvement is claimed. The source is available for review, but this does not satisfy or remove your real-hardware gate.

[PhilipJohnBasile]

Final harness verification: bdb763e is now pushed. It normalizes imports/formatting and sets the executable bit without changing the measurement contract.

Hosted run https://github.com/PhilipJohnBasile/MTPLX/actions/runs/34065860863 passed all 15 replay tests, Ruff, format checking, byte compilation, and diff checks before producing the exact source-only candidate promoted to this branch. It also checked the committed Codex capture receipt. The helper workflow remains fork-only and is not in this PR.

This completes the harness validation, not the real-model anchor comparison. The same-build/current-baseline 27B/128K A/B remains outstanding as described above.

[PhilipJohnBasile]

@youssofal The current-baseline prerequisite is now addressed in dfe28bf: the feature and frozen-transcript harness are integrated with 21be78b, so the comparison no longer starts from the older pre-turn-boundary code. The published source passed 562 semantic-anchor, replay, SessionBank, server, dashboard and import-boundary tests with no failures/skips on macOS ARM64: https://github.com/PhilipJohnBasile/MTPLX/actions/runs/34069355260 .

The empirical acceptance question is still deliberately unanswered: no 27B/128K OFF/ON timing or cache-hit gain was fabricated from unit tests or the 7 GiB hosted runner. The same-build paired hardware protocol remains in the PR. Source integration and regressions are ready for review; the real-model receipt remains the specific outstanding gate.