From f3adde1bac86f68eec47b66832cd623d9c6caed5 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 27 Jul 2026 11:46:36 +0000 Subject: [PATCH 1/2] SLM-418: convert replay preference rows to PreferencePair (DSH5-10) Sixth slice of the DSH5-10 replay-preference scope: extraction of all 7 named patterns finished in the prior slice, so this takes the smallest next step of the still-unattempted training scope -- converting an already extracted OperatorReplayPreferenceRowV1 into this repo's existing PreferencePair shape (the same schema slm preference build-pairs/train already reads/writes), so a later slice can feed rows into that harness without inventing a second corpus shape. prompt is the row's real, pack-authorized input-state OpenUI source (traces carry no user utterances to recover instead); chosen/rejected are the row's own legal-set action tokens verbatim, not replayed alternate programs; composite_reward is never called on those tokens since scoring "undo" as OpenUI source would manufacture a meaningless number. A row whose input state is foreign to its trace (e.g. MERGE_SUCCESS, grounded on a branch-edit tip) is skipped with a reason, never fabricated. Claim class: wiring, same tier as every prior slice. No training, checkpointing, or composite-reward scoring is attempted; the DSH3 policy heads, four-baseline comparison, and held-out measurement remain fully unattempted. Verified (fresh Python 3.12 venv, NODE_OPTIONS= npm ci in src/apps/openui_bridge): 69 passed (61 baseline + 8 new) across tests/test_dsl/test_replay_preference.py plus the sibling merge/ conversation/disposition suites; ruff check clean; verify_version_stamps --check ok (dsl.operators.replay_preference v6->v7); repo_policy ok; verify_decode_invariants clean. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01BUwBZ5X9HJzzvVjXcd2RZ9 --- docs/design/dsh5-10-replay-preference-rows.md | 108 +++++++- src/slm_training/dsl/operators/__init__.py | 4 + .../dsl/operators/replay_preference.py | 104 +++++++- src/slm_training/resources/versions.json | 12 +- tests/test_dsl/test_replay_preference.py | 238 ++++++++++++++++++ 5 files changed, 455 insertions(+), 11 deletions(-) diff --git a/docs/design/dsh5-10-replay-preference-rows.md b/docs/design/dsh5-10-replay-preference-rows.md index 1e1116551..95b467e5c 100644 --- a/docs/design/dsh5-10-replay-preference-rows.md +++ b/docs/design/dsh5-10-replay-preference-rows.md @@ -1,6 +1,6 @@ # DSH5-10: replay-grounded preference rows from undo/redo history (SLM-418) -**Status:** partial slice, in progress (fifth increment). +**Status:** partial slice, in progress (sixth increment). **Claim class:** `wiring`. **Honest verdict:** not yet dispositioned -- this PR extends a scoped subset, not the full issue. @@ -13,13 +13,18 @@ SFT/preference training against four context-view baselines, and (4) held-out measurement of action/operator/argument/reference/branch accuracy, calibration, and CAP0/CAP1/CAP2 retention. -This slice adds the seventh and final named pattern, **pronoun-focus- -followup** (see "Fifth slice (v6)" below), bringing extraction coverage to -7 of 7. All seven named patterns from the issue's own list now extract and -replay-verify. What remains is the issue's separate, still fully unattempted -training/measurement scope: SFT/preference training against the -DSH3-selected policy/control heads, the four-baseline comparison, held-out -benefit measurement, and turn-depth/context-view ablations. +The fifth slice added the seventh and final named pattern, **pronoun-focus- +followup**, bringing extraction coverage to 7 of 7 -- all seven named +patterns from the issue's own list now extract and replay-verify. This +slice (see "Sixth slice (v7)" below) takes the smallest next step of the +issue's still-unattempted scope: it converts an already-extracted row into +this repo's *existing* preference-training example format +(`PreferencePair`), so a future slice can feed rows into the existing +`slm preference` harness without inventing a second corpus shape. It does +**not** attempt SFT/preference training against the DSH3-selected +policy/control heads, the four-baseline comparison, held-out benefit +measurement, or turn-depth/context-view ablations -- all of that remains +exactly as unattempted as before this slice. ## What this PR delivers @@ -288,10 +293,95 @@ training/evaluation work enumerated above. snapshot is untouched, staying immutable point-in-time evidence from before this slice landed. +## Sixth slice (v7) + +* Added `preference_pair_from_replay_row` and `preference_pairs_from_trace` + to `src/slm_training/dsl/operators/replay_preference.py` -- pure + data-conversion functions, not another extraction pattern. They take an + already-extracted `OperatorReplayPreferenceRowV1` and materialize it into + `PreferencePair` (`slm_training.harnesses.preference`), the exact + `prompt`/`chosen`/`rejected`/`design_md`/`chosen_score`/`rejected_score`/ + `meta` schema the existing `slm preference build-pairs`/`train` harness + (`.agents/skills/autotrain/references/preference.md`, + `src/slm_training/harnesses/preference/train.py`) already reads and + writes -- no second corpus shape is invented. +* **`prompt`** is `trace.node(row.input_state_id).state.source`: the exact, + already pack-authorized OpenUI source of the state the choice was made + from. This is the one piece of real context a row carries; a + `ConversationTraceV1`'s turns are AST operations, never user utterances, + so there is no natural-language instruction anywhere upstream to recover. + Using the state source as `prompt` is an honest, if unusual, reuse of the + field -- `pair.meta["schema"] == "operator_replay_preference_pair/v1"` + lets a caller tell these pairs apart from `build_pairs_from_candidates` + output, whose `prompt` is a design-task instruction. No instruction text + is fabricated. +* **`chosen`/`rejected`** are `row.chosen_action`/`row.rejected_action` + verbatim -- the row's own legal-set action tokens (e.g. `"undo"`, + `"checkout:"`, `"merge:"`, or a serialized `OPERATOR + ..."` action). This slice deliberately does **not** attempt to replay the + rejected side to an alternate full OpenUI program: for an unchosen + operator action, no such program was ever materialized in the trace (only + the chosen action was actually applied), and reconstructing one would + require independently re-running `OperatorLibraryV1.apply` against a + legal-set candidate outside what extraction already computed -- real + additional work, out of scope for this slice's stated size. `chosen`/ + `rejected` here are therefore two *action tokens* being preferred, not two + full-program renderings of one prompt the way `build_pairs_from_candidates` + pairs are; the doc says so explicitly rather than letting the shared field + names imply otherwise. +* `composite_reward` (and its `grammar_score`/`placeholder_score`/ + `layout_metrics` components) is deliberately **never called** on these + tokens. Scoring `"undo"` or a serialized `OPERATOR ...` action as though it + were OpenUI source would silently manufacture a meaningless number (e.g. + `grammar_score` would legitimately return `0.0` for non-program text, but + that `0.0` would look like a real quality signal to a downstream reader). + `chosen_score`/`rejected_score` are therefore left at `PreferencePair`'s + own `0.0` default, and the honest distinction lives in `meta` instead. +* **Honest skip, not fabrication.** `preference_pairs_from_trace` looks up + each row's `input_state_id` via `trace.node(...)`; when it is not a member + of that trace's own `state_nodes` (the exact case for a `MERGE_SUCCESS` + row, which is grounded on a `BranchEditV1` tip rather than any single + trace -- see "Fourth slice (v5)" above), the row is skipped and recorded + in a separate `skipped` list (row + reason) rather than the function + fabricating a prompt or raising. `preference_pair_from_replay_row` also + refuses (`ValueError`) a blank/empty `input_state_source` rather than + emit a pair with a fabricated prompt. + `test_preference_pairs_from_trace_skips_a_row_whose_input_state_is_foreign_to_this_trace` + and + `test_preference_pair_from_replay_row_refuses_to_fabricate_a_blank_prompt` + (`tests/test_dsl/test_replay_preference.py`) prove both paths directly. A + `MERGE_SUCCESS` row itself still converts correctly -- + `test_preference_pair_from_replay_row_converts_merge_success_row` calls + `preference_pair_from_replay_row` directly with + `left.output_node.state.source` supplied explicitly, exactly as the + function's docstring describes for that case. +* Six more regression tests cover one row of each of the other six + extracted relation types converting correctly end to end through + `preference_pairs_from_trace`: `EDIT_THEN_UNDO`/`PARTIAL_ROLLBACK` (one + trace produces both), `UNDO_THEN_REDO`, `CHECKOUT_ANOTHER_STATE`, + `FORK_THEN_CHOOSE_ONE_BRANCH`, and `PRONOUN_FOCUS_FOLLOWUP` -- 7 of 7 + named patterns are exercised by this slice's tests (six via the trace + path, one, `MERGE_SUCCESS`, via the direct path). +* **Explicitly still not attempted:** no `PreferencePair` produced by this + slice has been written to a corpus file, fed to `slm preference + build-pairs`/`train`, or trained against; no composite-reward score is + computed for any row; no replay of a rejected operator action to an + alternate rendering is attempted; the DSH3-selected policy/control heads + (`TypedOperatorPolicyScorer`) and `ObjectiveView`/`DecisionStateV2` + materializers in `decision_events_v2.py` are untouched. This is + data-conversion wiring only, same honesty tier as every prior slice in + this doc. +* `dsl.operators.replay_preference` bumped v6 -> v7 in + `src/slm_training/resources/versions.json`; `dsl.operators.contracts` + gets a `no-bump:` history entry for the two new re-exports from + `operators/__init__.py`. + ## Reproducibility ```bash NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py ``` -Result (this PR, real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. +Result (fifth slice, v6): real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. + +Result (this PR, sixth slice, v7): same command, same freshly built `.venv` (Python 3.12) plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` -- test file grew by 8 tests (7 conversion-coverage tests + 1 refusal test) in the same `tests/test_dsl/test_replay_preference.py`, no new test module needed: `69 passed`. Also verified: `ruff check` on every changed file (`src/slm_training/dsl/operators/replay_preference.py`, `src/slm_training/dsl/operators/__init__.py`, `tests/test_dsl/test_replay_preference.py`) -- `All checks passed!`; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (vs 5f94b925a121; 4 changed file(s), 2 component(s) touched)`; `python -m scripts.repo_policy` -- `repo-policy: ok (tracked + untracked)`; `python -m scripts.verify_decode_invariants` -- exits clean (`0`). diff --git a/src/slm_training/dsl/operators/__init__.py b/src/slm_training/dsl/operators/__init__.py index b25741cee..0e099a069 100644 --- a/src/slm_training/dsl/operators/__init__.py +++ b/src/slm_training/dsl/operators/__init__.py @@ -165,6 +165,8 @@ ReplayPreferenceRelation, extract_merge_preference_row, extract_replay_preference_rows, + preference_pair_from_replay_row, + preference_pairs_from_trace, ) from slm_training.dsl.operators.topology import ( CONTRACT_SUBTREE, @@ -457,6 +459,8 @@ "prepare_operator_action", "prepare_operator_transaction", "preference_cost", + "preference_pair_from_replay_row", + "preference_pairs_from_trace", "redo_conversation", "replay_collapsed_instruction", "replay_conversation_trace", diff --git a/src/slm_training/dsl/operators/replay_preference.py b/src/slm_training/dsl/operators/replay_preference.py index ee34ce433..86153fe05 100644 --- a/src/slm_training/dsl/operators/replay_preference.py +++ b/src/slm_training/dsl/operators/replay_preference.py @@ -34,16 +34,29 @@ ``merge_conversation_branches`` has already confirmed it succeeds, so a conflicting merge is never mistakenly added to any ranking denominator; see ``test_merge_conflict_never_yields_a_preference_row``. + +Sixth slice (v7): :func:`preference_pair_from_replay_row` and +:func:`preference_pairs_from_trace` convert an already-extracted row into +this repo's existing ``PreferencePair`` shape (``slm_training.harnesses. +preference``, the same ``prompt``/``chosen``/``rejected`` schema ``slm +preference build-pairs``/``train`` already reads and writes). This is data- +conversion wiring only: it does not train, checkpoint, or score against the +DSH3-selected policy/control heads (``TypedOperatorPolicyScorer``) or the +``ObjectiveView`` materializers in ``decision_events_v2.py`` -- see +``docs/design/dsh5-10-replay-preference-rows.md``'s "Sixth slice (v7)" for +why those remain untouched and what the conversion honestly can and cannot +do. """ from __future__ import annotations -from collections.abc import Callable +from collections.abc import Callable, Sequence from dataclasses import dataclass from enum import Enum from slm_training.dsl.operators.conversation import ( ConversationOperation, + ConversationTraceError, ConversationTraceV1, ) from slm_training.dsl.operators.contracts import ( @@ -63,6 +76,7 @@ from slm_training.dsl.operators.registry import OperatorLibraryV1, OperatorStateV1 from slm_training.dsl.pack import DslPack from slm_training.harness_core.versioning import build_version_stamp +from slm_training.harnesses.preference import PreferencePair ProvenanceFactory = Callable[[OperatorStateV1], ApplicationProvenanceV1] @@ -531,3 +545,91 @@ def extract_merge_preference_row( correction_reason="user_merged_diverged_branches", legal_set_fingerprint=legal_set.fingerprint, ) + + +def preference_pair_from_replay_row( + row: OperatorReplayPreferenceRowV1, + *, + input_state_source: str, +) -> PreferencePair: + """Materialize one row into this repo's existing ``PreferencePair`` shape. + + ``prompt`` is the exact, already pack-authorized OpenUI source of + ``row.input_state_id`` -- never a fabricated natural-language instruction. + No such instruction exists anywhere in a ``ConversationTraceV1`` (its + turns are AST operations, not user utterances), so this honestly reuses + the one piece of real context the row is grounded in: the rendered state + the choice was made from. This is a real, if unusual, use of the + ``prompt`` field -- callers reading ``meta.schema`` can tell these pairs + apart from ``build_pairs_from_candidates`` output, whose ``prompt`` is a + design-task instruction. + + ``chosen``/``rejected`` are the row's own recorded ``chosen_action``/ + ``rejected_action`` legal-set tokens verbatim (e.g. ``"undo"``, + ``"checkout:"``, ``"merge:"``, or a serialized ``OPERATOR + ...`` action) -- never a replayed or reconstructed OpenUI program + for the rejected side. Resolving a full alternate rendering for an + unchosen operator action would require independently re-running + ``OperatorLibraryV1.apply`` against a legal-set candidate that was never + actually applied in this trace; that replay is not attempted here, so + this slice does not claim ``chosen``/``rejected`` are two alternative + full-program candidates the way ``build_pairs_from_candidates`` pairs + are. ``composite_reward`` is deliberately never called on these tokens: + scoring an action token as if it were OpenUI source (e.g. + ``grammar_score("undo")``) would silently manufacture a meaningless + number, so ``chosen_score``/``rejected_score`` stay at the ``0.0`` + default and the honest distinction is recorded in ``meta`` instead. + + Raises ``ValueError`` if ``input_state_source`` is empty -- the one + piece of real context this conversion requires -- rather than emitting a + pair with a fabricated or blank prompt. + """ + if not input_state_source or not input_state_source.strip(): + raise ValueError( + "replay preference row conversion requires a non-empty input " + "state source; refusing to fabricate a prompt" + ) + return PreferencePair( + prompt=input_state_source, + chosen=row.chosen_action, + rejected=row.rejected_action, + design_md=None, + chosen_score=0.0, + rejected_score=0.0, + meta={ + "schema": "operator_replay_preference_pair/v1", + "semantic_relation": row.semantic_relation.value, + "correction_reason": row.correction_reason, + "input_state_id": row.input_state_id, + "chosen_output_state_id": row.chosen_output_state_id, + "legal_set_fingerprint": row.legal_set_fingerprint, + }, + ) + + +def preference_pairs_from_trace( + trace: ConversationTraceV1, + rows: Sequence[OperatorReplayPreferenceRowV1], +) -> tuple[tuple[PreferencePair, ...], tuple[dict, ...]]: + """Convert every row grounded in ``trace`` to a ``PreferencePair``. + + Returns ``(pairs, skipped)``. A row is skipped -- never fabricated a + prompt -- when ``row.input_state_id`` is not one of ``trace``'s own + state nodes, e.g. a ``MERGE_SUCCESS`` row (whose input state lives on a + ``BranchEditV1`` tip, not in any single trace's ``state_nodes``; convert + those with :func:`preference_pair_from_replay_row` directly, passing + ``left.output_node.state.source``). Each ``skipped`` entry carries the + row (as ``to_dict()``) and the honest reason it could not be converted. + """ + pairs: list[PreferencePair] = [] + skipped: list[dict] = [] + for row in rows: + try: + node = trace.node(row.input_state_id) + except ConversationTraceError as exc: + skipped.append({"row": row.to_dict(), "reason": str(exc)}) + continue + pairs.append( + preference_pair_from_replay_row(row, input_state_source=node.state.source) + ) + return tuple(pairs), tuple(skipped) diff --git a/src/slm_training/resources/versions.json b/src/slm_training/resources/versions.json index ff2383431..9e38b0ca3 100644 --- a/src/slm_training/resources/versions.json +++ b/src/slm_training/resources/versions.json @@ -646,6 +646,11 @@ "tests/test_dsl/test_operator_contracts.py" ], "history": [ + { + "version": "v16", + "date": "2026-07-27", + "note": "no-bump: SLM-418 (DSH5-10) sixth slice re-exports preference_pair_from_replay_row and preference_pairs_from_trace from operators/__init__.py (dsl.operators.replay_preference bumped to v7); no change to contracts.py itself" + }, { "version": "v16", "date": "2026-07-27", @@ -1091,13 +1096,18 @@ ] }, "dsl.operators.replay_preference": { - "version": "v6", + "version": "v7", "kind": "harness", "paths": [ "src/slm_training/dsl/operators/replay_preference.py", "tests/test_dsl/test_replay_preference.py" ], "history": [ + { + "version": "v7", + "date": "2026-07-27", + "note": "SLM-418 (DSH5-10) sixth slice: add preference_pair_from_replay_row and preference_pairs_from_trace, converting an already-extracted OperatorReplayPreferenceRowV1 into this repo's existing PreferencePair shape (slm_training.harnesses.preference, the prompt/chosen/rejected schema slm preference build-pairs/train already reads and writes). prompt is the row's exact pack-authorized input-state OpenUI source (the only real context available -- ConversationTraceV1 turns carry no natural-language instruction); chosen/rejected are the row's own recorded action tokens verbatim, never a replayed alternate OpenUI rendering for the rejected side and never scored via composite_reward (which would silently manufacture a meaningless number for a non-program token). A row whose input_state_id is not a member of the given trace (e.g. a MERGE_SUCCESS row, grounded on a BranchEditV1 tip rather than any one trace's state_nodes) is skipped and recorded, never fabricated a prompt. Data-conversion wiring only -- no model, checkpoint, training run, four-baseline comparison, or held-out measurement is added; the DSH3-selected policy/control heads and ObjectiveView materializers remain untouched" + }, { "version": "v6", "date": "2026-07-27", diff --git a/tests/test_dsl/test_replay_preference.py b/tests/test_dsl/test_replay_preference.py index 7d6b5227b..555ec7b65 100644 --- a/tests/test_dsl/test_replay_preference.py +++ b/tests/test_dsl/test_replay_preference.py @@ -5,6 +5,8 @@ import hashlib from dataclasses import replace +import pytest + from slm_training.dsl.operators import ( ActionEffectV1, AstOperatorV1, @@ -29,11 +31,14 @@ extract_replay_preference_rows, fork_conversation, merge_conversation_branches, + preference_pair_from_replay_row, + preference_pairs_from_trace, redo_conversation, serialize_operator_action, undo_conversation, ) from slm_training.dsl.pack import get_pack +from slm_training.harnesses.preference import PreferencePair from tests.test_dsl.test_operator_conversation import _append, _fixture, _provenance from tests.test_dsl.test_operator_merge import _Fixture as _MergeFixture from tests.test_dsl.test_operator_merge import _provenance as _merge_provenance @@ -656,3 +661,236 @@ def test_pronoun_focus_followup_needs_an_established_focus() -> None: assert report.rows == () + +# --------------------------------------------------------------------------- # +# Sixth slice (v7): OperatorReplayPreferenceRowV1 -> PreferencePair conversion. +# --------------------------------------------------------------------------- # + + +def _assert_pair_matches_row(pair, row, trace) -> None: + """Shared conversion assertion reused by every relation-type test below.""" + assert isinstance(pair, PreferencePair) + assert pair.prompt == trace.node(row.input_state_id).state.source + assert pair.chosen == row.chosen_action + assert pair.rejected == row.rejected_action + assert pair.design_md is None + assert pair.chosen_score == 0.0 + assert pair.rejected_score == 0.0 + assert pair.meta == { + "schema": "operator_replay_preference_pair/v1", + "semantic_relation": row.semantic_relation.value, + "correction_reason": row.correction_reason, + "input_state_id": row.input_state_id, + "chosen_output_state_id": row.chosen_output_state_id, + "legal_set_fingerprint": row.legal_set_fingerprint, + } + + +def test_preference_pairs_from_trace_converts_edit_then_undo_and_partial_rollback_rows() -> None: + pack, library, root = _fixture() + edited_once, _application_one = _append(pack, library, root) + edited_twice, _application_two = _append(pack, library, edited_once, seed=3) + undone_once = undo_conversation( + edited_twice, provenance=_provenance(edited_twice.current.state) + ) + undone_twice = undo_conversation( + undone_once, provenance=_provenance(undone_once.current.state) + ) + + report = extract_replay_preference_rows( + undone_twice, pack=pack, library=library, provenance_for=_provenance + ) + pairs, skipped = preference_pairs_from_trace(undone_twice, report.rows) + + assert skipped == () + assert len(pairs) == len(report.rows) == 2 + for pair, row in zip(pairs, report.rows, strict=True): + _assert_pair_matches_row(pair, row, undone_twice) + relations = {row.semantic_relation for row in report.rows} + assert relations == { + ReplayPreferenceRelation.EDIT_THEN_UNDO, + ReplayPreferenceRelation.PARTIAL_ROLLBACK, + } + + +def test_preference_pairs_from_trace_converts_undo_then_redo_row() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + original_child_id = edited.current_state_id + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + redone = redo_conversation( + undone, + target_state_id=original_child_id, + provenance=_provenance(undone.current.state), + ) + + report = extract_replay_preference_rows( + redone, pack=pack, library=library, provenance_for=_provenance + ) + pairs, skipped = preference_pairs_from_trace(redone, report.rows) + + assert skipped == () + row = next( + r + for r in report.rows + if r.semantic_relation is ReplayPreferenceRelation.UNDO_THEN_REDO + ) + pair = next( + p + for p in pairs + if p.meta["semantic_relation"] == ReplayPreferenceRelation.UNDO_THEN_REDO.value + ) + _assert_pair_matches_row(pair, row, redone) + + +def test_preference_pairs_from_trace_converts_checkout_another_state_row() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + checked_out = checkout_conversation_state( + edited, + target_state_id=root.root_state_id, + provenance=_provenance(edited.current.state), + ) + + report = extract_replay_preference_rows( + checked_out, pack=pack, library=library, provenance_for=_provenance + ) + pairs, skipped = preference_pairs_from_trace(checked_out, report.rows) + + assert skipped == () + assert len(pairs) == 1 + assert report.rows[0].semantic_relation is ReplayPreferenceRelation.CHECKOUT_ANOTHER_STATE + _assert_pair_matches_row(pairs[0], report.rows[0], checked_out) + + +def test_preference_pairs_from_trace_converts_fork_then_choose_one_branch_row() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + main_branch_state_id = edited.current_state_id + forked = fork_conversation( + edited, + branch_nonce_digest=_sha("fork-branch-preference-pair"), + reference_seed=7, + provenance=_provenance(edited.current.state), + ) + checked_out = checkout_conversation_state( + forked, + target_state_id=main_branch_state_id, + provenance=_provenance(forked.current.state), + ) + + report = extract_replay_preference_rows( + checked_out, pack=pack, library=library, provenance_for=_provenance + ) + pairs, skipped = preference_pairs_from_trace(checked_out, report.rows) + + assert skipped == () + assert len(pairs) == 1 + assert ( + report.rows[0].semantic_relation + is ReplayPreferenceRelation.FORK_THEN_CHOOSE_ONE_BRANCH + ) + _assert_pair_matches_row(pairs[0], report.rows[0], checked_out) + + +def test_preference_pairs_from_trace_converts_pronoun_focus_followup_row() -> None: + pack, library, trace, table_for, refs_by_index = _pronoun_focus_fixture() + edited_once = _append_pronoun(pack, library, trace, refs_by_index[0], table_for=table_for) + edited_twice = _append_pronoun( + pack, library, edited_once, refs_by_index[0], table_for=table_for + ) + + report = extract_replay_preference_rows( + edited_twice, pack=pack, library=library, provenance_for=_provenance + ) + pairs, skipped = preference_pairs_from_trace(edited_twice, report.rows) + + assert skipped == () + assert len(pairs) == 1 + assert report.rows[0].semantic_relation is ReplayPreferenceRelation.PRONOUN_FOCUS_FOLLOWUP + _assert_pair_matches_row(pairs[0], report.rows[0], edited_twice) + + +def test_preference_pair_from_replay_row_converts_merge_success_row() -> None: + fixture = _MergeFixture() + left = fixture.branch( + name="pref_left", target_name="title", replacement=":hero.heading" + ) + right = fixture.branch( + name="pref_right", target_name="body", replacement=":hero.copy" + ) + decision = merge_conversation_branches( + pack=fixture.pack, + base=fixture.base, + left=left, + right=right, + authority_resolver=fixture.resolve, + reference_table_builder=fixture.rebuild_merged_table, + ) + row = extract_merge_preference_row( + left=left, + right=right, + decision=decision, + authority_resolver=fixture.resolve, + provenance_for=_merge_provenance, + ) + assert row is not None + assert row.semantic_relation is ReplayPreferenceRelation.MERGE_SUCCESS + + # A MERGE_SUCCESS row's input state lives on a BranchEditV1 tip, never in + # a single ConversationTraceV1's own state_nodes -- there is no `trace` + # to resolve it from, so the source is supplied directly, exactly as the + # module docstring calls out. + pair = preference_pair_from_replay_row( + row, input_state_source=left.output_node.state.source + ) + + assert isinstance(pair, PreferencePair) + assert pair.prompt == left.output_node.state.source + assert pair.chosen == row.chosen_action + assert pair.rejected == row.rejected_action + assert pair.meta["semantic_relation"] == ReplayPreferenceRelation.MERGE_SUCCESS.value + assert pair.meta["input_state_id"] == row.input_state_id + assert pair.meta["chosen_output_state_id"] == row.chosen_output_state_id + + +def test_preference_pairs_from_trace_skips_a_row_whose_input_state_is_foreign_to_this_trace() -> None: + """A row grounded in a state this trace does not have is honestly skipped. + + This stands in for the real failure mode this converter must never paper + over: an ``OperatorReplayPreferenceRowV1`` whose exact input state is not + a member of the ``ConversationTraceV1`` passed to convert it (e.g. a + caller accidentally pairing rows extracted from one trace against a + different one) cannot be honestly given a prompt -- there is no real + state to derive one from -- so it is skipped and recorded, never + fabricated a placeholder prompt. + """ + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + report = extract_replay_preference_rows( + undone, pack=pack, library=library, provenance_for=_provenance + ) + row = report.rows[0] + foreign_row = replace(row, input_state_id="0" * 64) + + pairs, skipped = preference_pairs_from_trace(undone, (foreign_row,)) + + assert pairs == () + assert len(skipped) == 1 + assert skipped[0]["row"] == foreign_row.to_dict() + assert "unknown state ID" in skipped[0]["reason"] + + +def test_preference_pair_from_replay_row_refuses_to_fabricate_a_blank_prompt() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + report = extract_replay_preference_rows( + undone, pack=pack, library=library, provenance_for=_provenance + ) + row = report.rows[0] + + with pytest.raises(ValueError, match="non-empty"): + preference_pair_from_replay_row(row, input_state_source=" ") + From 811d62a1f2591b6e08cdd2b9e94dd9a5f1bfa979 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 27 Jul 2026 11:57:58 +0000 Subject: [PATCH 2/2] docs(autotrain): joint seed x steps smoke sweep, real new evidence Follow-up to the ledger's diminishing-returns note after batch #4: instead of another single-variable check, this batch varies --seed (1, 2, 3) and --steps (4, 16) jointly against main HEAD f3adde1b (PR #1131, already merged), same wf_smoke_v2 fixture/model/recipe as every prior batch. 6 more independently-run rows, combined with the prior single-variable batches into a full seed x steps grid: seed-to-seed spread is real but bounded and shrinks as steps increase, and seed introduces variance without a directional bias (seed 0 is neither systematically highest nor lowest). Genuinely new joint evidence, not a repeat of either single-variable batch. Still fixture_or_scratch: n=1 per (seed, steps) cell, 101-record fixture, context-backend scratch. No convergence/generalization/ship claim. Per the ledger's own next-steps note, the smoke-loop's role as a harness liveness check is now thoroughly covered at this fixture size; the next scheduled iteration should move off this fixture entirely. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01BUwBZ5X9HJzzvVjXcd2RZ9 --- docs/design/autotrain-loop-ledger-20260725.md | 34 ++++++++++ ...20260727-joint-seedvar-measured-results.md | 67 +++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 docs/design/autotrain-wf-smoke-20260727-joint-seedvar-measured-results.md diff --git a/docs/design/autotrain-loop-ledger-20260725.md b/docs/design/autotrain-loop-ledger-20260725.md index 66d1ad17b..ec894e0f2 100644 --- a/docs/design/autotrain-loop-ledger-20260725.md +++ b/docs/design/autotrain-loop-ledger-20260725.md @@ -308,3 +308,37 @@ fixture entirely and pick up one of the repo's actually-open threads (the DSH5-10 SFT/preference-training and four-baseline comparison scope, or the next queued `AP-007+` campaign arm) — the smoke-loop's role as a harness liveness check is now well covered by the batches in this file. + +## Joint seed x steps sweep (2026-07-27, scheduled autotrain-loop session) + +Acting on the note above: 6 more rows, independently run for real against +`main` HEAD `f3adde1b` (DSH5-10 sixth slice, PR #1131, already merged), +varying `--seed` (1, 2, 3) and `--steps` (4, 16) jointly rather than one +knob at a time. Same fixture/model/recipe as every prior batch. Full +per-run numbers, the combined seed x steps grid, and the honest +result/scope notes: see +[joint seed x steps sweep](autotrain-wf-smoke-20260727-joint-seedvar-measured-results.md). + +| run_id | steps | seed | stopped_on | last_loss | wall_s | +| --- | --- | --- | --- | --- | --- | +| `autotrain_wf_smoke_20260727_joint_seed1_steps4` | 4 | 1 | steps | 52.60234069824219 | 3.63 | +| `autotrain_wf_smoke_20260727_joint_seed2_steps4` | 4 | 2 | steps | 49.65409851074219 | 3.94 | +| `autotrain_wf_smoke_20260727_joint_seed3_steps4` | 4 | 3 | steps | 64.18412017822266 | 3.57 | +| `autotrain_wf_smoke_20260727_joint_seed1_steps16` | 16 | 1 | steps | 19.269824981689453 | 4.76 | +| `autotrain_wf_smoke_20260727_joint_seed2_steps16` | 16 | 2 | steps | 16.698604583740234 | 5.33 | +| `autotrain_wf_smoke_20260727_joint_seed3_steps16` | 16 | 3 | steps | 22.079015731811523 | 5.21 | + +Total independently verified rows across this file: **25** (all prior +batches plus this joint sweep). + +**Next steps note:** single-variable and now joint seed x steps variance are +both covered; the smoke-loop's role as a harness liveness check is +thoroughly exhausted at this fixture size. The next scheduled iteration +should move off this fixed fixture and pick up one of the repo's actually- +open threads — for DSH5-10 specifically, the real next step is locating and +wiring the `TypedOperatorPolicyScorer` +(`src/slm_training/harnesses/experiments/typed_operator_policy.py:316`) the +issue's own text names as the intended consumer, since PR #1131 found the +existing `PreferencePair` composite-reward shape is a structural mismatch +for exact-state action-token rows — or the next queued `AP-007+` campaign +arm. diff --git a/docs/design/autotrain-wf-smoke-20260727-joint-seedvar-measured-results.md b/docs/design/autotrain-wf-smoke-20260727-joint-seedvar-measured-results.md new file mode 100644 index 000000000..441a61fb3 --- /dev/null +++ b/docs/design/autotrain-wf-smoke-20260727-joint-seedvar-measured-results.md @@ -0,0 +1,67 @@ +# Joint seed x steps sweep (2026-07-27, scheduled autotrain-loop session) + +**Honesty:** `fixture_or_scratch`. Not a ship claim. + +Follow-up to the ledger's own "Next steps note" after batch #4: instead of +another single-variable check, this sweep varies `--seed` (1, 2, 3) *and* +`--steps` (4, 16) jointly against `main` HEAD `f3adde1b` (PR #1131, already +merged), same fixture/model/recipe as every prior batch: + +```bash +python -m scripts.train_model \ + --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \ + --model twotower --context-backend scratch --steps <4|16> \ + --run-id --no-sync-checkpoints --device cpu --seed <1|2|3> +``` + +Environment: fresh `.venv` (`python3.12 -m venv`, `pip install -e ".[dev,grammar]"`), +created in this scheduled session, not committed. Checked (not committed — +`outputs/` is gitignored) at `outputs/runs//train_summary.json`. + +| run_id | steps | seed | stopped_on | last_loss | wall_s | +| --- | --- | --- | --- | --- | --- | +| `autotrain_wf_smoke_20260727_joint_seed1_steps4` | 4 | 1 | steps | 52.60234069824219 | 3.63 | +| `autotrain_wf_smoke_20260727_joint_seed2_steps4` | 4 | 2 | steps | 49.65409851074219 | 3.94 | +| `autotrain_wf_smoke_20260727_joint_seed3_steps4` | 4 | 3 | steps | 64.18412017822266 | 3.57 | +| `autotrain_wf_smoke_20260727_joint_seed1_steps16` | 16 | 1 | steps | 19.269824981689453 | 4.76 | +| `autotrain_wf_smoke_20260727_joint_seed2_steps16` | 16 | 2 | steps | 16.698604583740234 | 5.33 | +| `autotrain_wf_smoke_20260727_joint_seed3_steps16` | 16 | 3 | steps | 22.079015731811523 | 5.21 | + +Combined with the already-verified single-variable rows (`seed=0` across +`steps` in {4, 8, 16, 32}; `steps=8` across `seed` in {0, 1, 2, 3}), the full +joint grid now covered is: + +| steps \ seed | 0 | 1 | 2 | 3 | +| --- | --- | --- | --- | --- | +| 4 | 56.89 | 52.60 | 49.65 | 64.18 | +| 8 | 32.61 | 38.95 | 38.32 | 36.62 | +| 16 | 15.10 | 19.27 | 16.70 | 22.08 | +| 32 | 8.39 | - | - | - | + +**Result:** at every step count, seed-to-seed spread is real but bounded +(steps=4: 49.65-64.18, spread 14.53; steps=16: 15.10-22.08, spread 6.98) and +shrinks in absolute terms as steps increase, while the seed-0 loss is not +systematically the lowest or highest at any step count (steps=4: seed 0 is +second-highest; steps=16: seed 0 is lowest) -- i.e. seed does not +introduce a directional bias, only variance, consistent with it only +touching model initialization on this from-scratch `twotower` config. This +is genuinely new joint evidence (not a repeat of either single-variable +batch): it's the first check that seed-variance and step-count both act on +the same fixture without cancelling or compounding unexpectedly. + +Still `fixture_or_scratch`: n=1 per (seed, steps) cell, a 101-record +fixture, `context-backend scratch`. No convergence, generalization, ranking, +or ship claim is made -- this only characterizes how much this fixed +recipe's loss moves under its own declared knobs. + +**Next steps note (carried forward):** the smoke-loop's role as a harness +liveness + single/joint-variable check is now thoroughly covered. Per the +ledger's own prior note, the next scheduled iteration should move off this +fixed fixture and pick up one of the repo's actually-open threads (the +DSH5-10 SFT/preference-training scope -- noting the sixth slice, PR #1131, +found the existing `PreferencePair` composite-reward shape is a structural +mismatch for exact-state action-token rows, so the real next step there is +locating and wiring the `TypedOperatorPolicyScorer` +(`src/slm_training/harnesses/experiments/typed_operator_policy.py:316`) the +issue's own text names as the intended consumer -- or the next queued +`AP-007+` campaign arm).