SLM-418: publish DSH5-10 event-memory disposition (falsification close)

[Tyler-R-Kendrick]

SLM-418 (DSH5-10): final disposition — falsification close

Closes SLM-418 ("Use undo, redo, and fork history as replay-grounded preference signal for ambiguous follow-up instructions").

Decision

no_held_out_benefit_at_fixture_scale_retain_dag_only — the issue's own falsification/stop rule, invoked verbatim:

If replay-grounded history yields no held-out benefit, retain the event DAG for runtime/evaluation only and do not add preference training complexity.

This is the designed, legitimate negative close. The conversation event DAG (ConversationTraceV1) is retained as sole state authority for runtime/evaluation; no SFT/preference training complexity is added (no training run, no checkpoint, no TypedOperatorPolicyScorer wiring, no schema change to frozen OperatorPolicyInputV1).

What this PR adds

• docs/design/iter-slm418-dsh5-10-disposition-20260727.{json,md} — final OperatorEventMemoryReportV1-style report (operator_event_memory_report/v1, real version_stamp via build_version_stamp): pattern coverage, ablation numbers, acceptance-criteria status, non-goals honored, adapter-gap summary, successor conditions.
• versions.json: new component harness.experiments.slm418_dsh5_10_disposition v1 (covers the two docs); harness.experiments v139 → v140 history note. No watched code file changed.

Evidence (all on merged main)

Pattern coverage: 7/7 named extraction patterns in src/slm_training/dsl/operators/replay_preference.py (v7) — edit_then_undo, undo_then_redo, partial_rollback, checkout_another_state, fork_then_choose_one_branch, merge_success, pronoun_focus_followup — each replay-verified against the exact legal set at its input state. Merge conflict excluded from every ranking denominator by construction (test_merge_conflict_never_yields_a_preference_row).

Ablation (PR #1129, 57f5bdbb; re-run this session):

Metric	Value
Corpus	8 sessions (6 train / 2 held-out), 40 rows, group-stable split
Grid	5 context views × depths {1,2,4,8,16} = 25 cells
undo_family_rate	0.9
Held-out pairwise accuracy, all 25 cells incl. current_state_only	1.0 (ceiling effect: is_history_control alone separates the 4-pair held-out split)
Verdict	no_benefit_fixture_scale (baseline 1.0, best state_plus_recent_receipts depth 1 = 1.0, tied)
Calibration proxy (Brier)	0.00011 baseline vs 0.00059 recent-receipts d1, → 0.00007 by d16
Structural grid	recency vs ancestry windows genuinely differ on rollback_chain_8 (undo vs operator)

Adapter gap (docs/design/dsh5-10-policy-scorer-adapter-gap-20260727.md): 6/7 relations choose history-control tokens not expressible as OperatorActionViewV1 rows; training not warranted by evidence. Successor condition recorded: a real argument-bound corpus build (VAR3-04/05 pattern) showing held-out benefit over state-only/derived-only baselines on a powered split.

Verification (this branch, from origin/main 0fd83214)

• python -m scripts.run_replay_preference_context_view_ablation → no_benefit_fixture_scale, 8 sessions / 40 rows, undo_family_rate=0.9, all 25 cells 1.0.
• pytest tests/test_dsl/test_replay_preference.py tests/test_harnesses/preference/test_operator_history_pairs.py tests/test_evals/test_ambiguous_operator_followups.py -q → 50 passed.
• python -m scripts.verify_version_stamps --check → ok (3 changed files, 1 component touched).
• python -m scripts.repo_policy → ok (tracked + untracked).
• git diff --check → clean. Pre-commit hook also ran verify_version_stamps --staged + tests/test_versioning (15 passed).

Note: the demo pairs builder (scripts/build_replay_preference_pairs.py, PRs #1125/#1127/#1128) is not on origin/main (unmerged branch origin/claude/great-dirac-ni43oh); the disposition does not depend on it.

Summary by CodeRabbit

• Documentation
  • Added a design report and disposition documenting replay-history evaluation results.
  • Recorded that event history remains the runtime and evaluation authority.
  • Documented fixture-scale testing, reproducibility results, acceptance criteria, and future conditions for revisiting preference training.
• Chores
  • Updated internal version and experiment registry records to reflect recent component history and the new evaluation report.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
slm-training	Ready	Preview, Comment	Jul 27, 2026 4:59pm

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 23243028-148e-4344-9b9b-666a47e0e905

📥 Commits

Reviewing files that changed from the base of the PR and between 0fd8321 and cfd7db2.

📒 Files selected for processing (3)

• docs/design/iter-slm418-dsh5-10-disposition-20260727.json
• docs/design/iter-slm418-dsh5-10-disposition-20260727.md
• src/slm_training/resources/versions.json

---

📝 Walkthrough

Walkthrough

Adds SLM-418 disposition documents and version-registry entries. The report records fixture-scale replay-preference ablation results showing no held-out benefit, retains the conversation event DAG as runtime and evaluation authority, and defines conditions for reconsidering preference training.

Changes

SLM-418 disposition

Layer / File(s)	Summary
Disposition evidence and follow-up conditions docs/design/iter-slm418-dsh5-10-disposition-20260727.json, docs/design/iter-slm418-dsh5-10-disposition-20260727.md	Documents replay extraction coverage, fixture-scale ablation metrics, acceptance-criteria outcomes, non-goals, the OperatorActionViewV1 adapter gap, successor conditions, and reproducibility results.
Version registry integration src/slm_training/resources/versions.json	Updates component history notes, bumps harness.experiments to v140, and registers the new SLM-418 disposition component with its design artifact paths.

Estimated code review effort: 2 (Simple) | ~15 minutes

Possibly related PRs

• Tyler-R-Kendrick/slm-training#1129: Adds the context-view and turn-depth ablation harness and evaluation tooling referenced by this disposition.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch slm-418-dsh5-10-disposition

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}