SLM-418: add partial-rollback and fork-choose demo scenarios (DSH5-10) - #1128
Merged
Tyler-R-Kendrick merged 1 commit intoJul 27, 2026
Conversation
Stacked on #1127. Tenth slice of SLM-418's replay-grounded preference work: extends scripts/build_replay_preference_pairs.py with two more standalone scratch traces (build_demo_partial_rollback_scenario, build_demo_fork_choose_scenario) covering the two named patterns (partial_rollback, fork_then_choose_one_branch) extraction has supported since the fifth slice but the demo TwoTower corpus never exercised. 6 of 7 named patterns are now in this corpus; only pronoun_focus_followup remains absent, deliberately, since the ninth slice already covers it on the separate typed_operator_policy path. Reran the full SFT -> build-pairs -> preference-training chain against the richer 6-pair corpus. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hzz1DzercsjW7yBQaddo4Z
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Tyler-R-Kendrick
merged commit Jul 27, 2026
a08851e
into
claude/great-dirac-ni43oh-pronoun-focus-policy
6 checks passed
Tyler-R-Kendrick
added a commit
that referenced
this pull request
Jul 27, 2026
* SLM-418: build a demo replay-preference pairs corpus + run it (DSH5-10) Seventh slice, stacked on #1124 (the sixth slice's row->PreferencePair renderer). Adds scripts/build_replay_preference_pairs.py: builds one small, deterministic, honestly-labeled fixture_or_scratch conversation exercising three of the seven named patterns (edit-then-undo, undo-then-redo, checkout-another-state), extracts rows, renders them, and writes a real pairs.jsonl -- the first real, on-disk pairs corpus this feature line has ever produced. 2 of 3 rows render; undo_then_redo never does, for a structural reason, not a bug: for any deterministic zero-argument operator, redo and reapplying the same operator at the same input state are, by construction, identical text, so the renderer's own dedup guard correctly declines rather than emitting a self-contradictory pair. Confirmed (again) while scoping this slice: no real captured ConversationTraceV1 corpus exists anywhere in this repo, and no harness ingests one -- build_symbolic_operator_corpus synthesizes traces combinatorially from gold DSL records, it doesn't read captured usage. Also ran the first real end-to-end chain using DSH5-10 rows: a scratch SFT checkpoint (wf_smoke_v2, seed 0, 8 steps -- last_loss identical to the 16+ prior independently-verified rows in the smoke-loop ledger, confirming this is the same deterministic artifact those rows already verified), the new demo pairs corpus, and one bounded scripts/train_preference.py train call against it (6 steps, 2 pairs, last_loss=1.077). Both well under MAX_RUN_MINUTES=3. Still not a training or held-out-benefit claim: n_pairs=2 on a scratch fixture says the pipeline runs end to end for real, nothing about whether the signal helps the model or should train the DSH3-selected policy head the issue actually asks about. harness.preference.replay_pairs bumped v1 -> v2 (adds the new script+test to its watched paths). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDctCUvBtGrdMvgNd34oHR * SLM-418: add merge_success to the demo pairs corpus + rerun (DSH5-10) Eighth slice, stacked on #1125 (the seventh slice's demo trace + first real training run). Adds build_demo_merge_scenario to scripts/build_replay_preference_pairs.py: a second scratch fixture (two branches forked from a shared base, editing disjoint node refs), mirroring the exact disjoint-target shape tests/test_dsl/test_operator_merge.py already verifies merges cleanly, replayably, and order-invariantly. Reaches merge_success, the one named pattern the seventh slice's single-trace corpus structurally cannot (extract_merge_preference_row never operates on a shared ConversationTraceV1). main() now combines both sources into one report and one pairs.jsonl: 4 rows (3 trace-scan + 1 merge_success), 3 render -- undo_then_redo is still the only drop, for the same structural reason the seventh slice documented. Reran the full training chain against the richer corpus: same deterministic SFT checkpoint (last_loss=32.610084533691406, matching every prior wf_smoke_v2/seed-0/8-step row), preference training now over 3 pairs instead of 2 (9 steps, last_loss=0.531). Still not a training or held-out-benefit claim -- a larger, more structurally diverse scratch corpus, not evidence the signal helps the model. harness.preference.replay_pairs bumped v2 -> v3. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDctCUvBtGrdMvgNd34oHR * SLM-418: first step onto the DSH3 typed policy target (DSH5-10) Ninth slice, stacked on #1126. Every prior slice since the sixth has flagged the same gap: the TwoTower PreferencePair path (#1124-#1126) exists only for tooling compatibility -- the disposition doc's own remaining-scope note names typed_operator_policy.py's TypedOperatorPolicyScorer, not TwoTower, as the issue's actual training target. This slice takes the first real step onto it, and honestly narrows what's reachable there. Adds src/slm_training/harnesses/experiments/argument_preference.py: build_argument_preference_example renders only pronoun_focus_followup rows into a TypedOperatorArgumentPreferenceExampleV1 -- the one named pattern whose chosen/rejected actions are the same operator with a differing argument for the same slot. The other six patterns are history controls (undo/redo/checkout) or merge, none of which has a row in OperatorPolicyInputV1.action_rows (built only from legal_set.entries); wiring them needs a real scope decision this slice leaves open rather than guesses at. typed_operator_argument_preference_loss is a Bradley-Terry pairwise margin over CandidateScoringHead logits (surrogate, not textbook DPO, same honesty class as train_preference.py's own dpo_loss). Real, structural finding, not a bug: training on the actual pronoun-focus-followup fixture provably cannot move the loss. Both sibling refs share every field ReferenceModelViewV1 exposes -- they differ only by semantic_fingerprint, which is deliberately stripped from every model input for anti-identity-leakage. Two feature-identical candidates get byte-identical embeddings through the shared-weight CandidateScoringHead regardless of any parameter update, so the loss sits at -log_sigmoid(0)=ln(2) structurally. Proven exact (pytest.approx) and distinct from a mechanism failure via a second test with a synthetic, feature-distinguishable pair, where training does reduce the loss. Consequence for future corpus-building: only pairs whose candidates differ in an allowed feature carry learnable signal for this scorer -- worth knowing before anyone builds a real corpus and is puzzled why it plateaus. harness.experiments.argument_preference registered fresh (v1). No change to typed_operator_policy.py itself (only consumed). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDctCUvBtGrdMvgNd34oHR * SLM-418: add partial-rollback and fork-choose demo scenarios (DSH5-10) (#1128) Stacked on #1127. Tenth slice of SLM-418's replay-grounded preference work: extends scripts/build_replay_preference_pairs.py with two more standalone scratch traces (build_demo_partial_rollback_scenario, build_demo_fork_choose_scenario) covering the two named patterns (partial_rollback, fork_then_choose_one_branch) extraction has supported since the fifth slice but the demo TwoTower corpus never exercised. 6 of 7 named patterns are now in this corpus; only pronoun_focus_followup remains absent, deliberately, since the ninth slice already covers it on the separate typed_operator_policy path. Reran the full SFT -> build-pairs -> preference-training chain against the richer 6-pair corpus. Claude-Session: https://claude.ai/code/session_01Hzz1DzercsjW7yBQaddo4Z Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #1127 (which is stacked on #1126, #1125, #1124). This is the tenth slice of SLM-418's replay-grounded preference work (DSH5-10), run as part of a scheduled autotrain-loop session continuing this stack.
typed_operator_policy.py's action space) remains genuinely open and is not attempted here. This slice closes a smaller, independent gap instead: two of the seven named DSH5-10 patterns —partial_rollbackandfork_then_choose_one_branch— have been supported byextract_replay_preference_rowssince the fifth slice, butscripts/build_replay_preference_pairs.py's demo TwoTower corpus never exercised them.build_demo_partial_rollback_scenario(root→edit1→edit2→undo→undo) andbuild_demo_fork_choose_scenario(root→edit→fork→checkout(pre-fork branch)), mirroring the exact shapestests/test_dsl/test_replay_preference.py's owntest_partial_rollback_yields_a_row_for_the_second_consecutive_undoandtest_fork_then_return_to_original_branch_yields_a_distinct_relationalready verify, using the script's existing toy 4-state cycling operator._new_toy_trace/_apply_cyclehelpers rather than copying it again;build_demo_traceitself is behavior-unchanged (confirmed by its unmodified test still passing).main()now combines four sources: 7 rows (2edit_then_undo+ 1 each ofundo_then_redo/checkout_another_state/partial_rollback/fork_then_choose_one_branch/merge_success), 6 render —undo_then_redois still the only drop, for the same structural reason documented since the seventh slice.pronoun_focus_followupremains absent, deliberately — the ninth slice already covers it on the separatetyped_operator_policyargument-preference path, so duplicating it here would just repeat that slice's already-documented structural null-training finding.last_loss=32.610084533691406, matching every priorwf_smoke_v2/seed-0/8-step row in the ledger), then preference training over 6 pairs instead of 3 (last_loss=1.122). Both commands well underMAX_RUN_MINUTES=3.harness.preference.replay_pairsbumpedv3→v4.Full disposition in
docs/design/dsh5-10-replay-preference-rows.md's new "Tenth slice" section.Test plan
env -u NODE_OPTIONS pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_harnesses/preference/test_replay_pairs.py tests/test_scripts/test_build_replay_preference_pairs.py tests/test_harnesses/experiments/test_argument_preference.py tests/test_harnesses/experiments/test_typed_operator_policy.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py—94 passed(92 from SLM-418: first step onto the DSH3 typed policy target (DSH5-10) #1127 + 2 new)ruff checkclean on both changed filespython -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-ni43oh-pronoun-focus-policy—ok (1 component(s) touched)python -m scripts.repo_policy—okpython -m scripts.verify_decode_invariants— cleanscripts.train_model(SFT) →scripts.build_replay_preference_pairs(now 7 rows/6 pairs) →scripts.train_preference train(6 pairs, 12 steps), outputs inspected (not committed —outputs/is gitignored)Generated by Claude Code