diff --git a/docs/design/dsh5-10-replay-preference-rows.md b/docs/design/dsh5-10-replay-preference-rows.md index 1e1116551..d175a3af0 100644 --- a/docs/design/dsh5-10-replay-preference-rows.md +++ b/docs/design/dsh5-10-replay-preference-rows.md @@ -1,6 +1,6 @@ # DSH5-10: replay-grounded preference rows from undo/redo history (SLM-418) -**Status:** partial slice, in progress (fifth increment). +**Status:** partial slice, in progress (tenth increment). **Claim class:** `wiring`. **Honest verdict:** not yet dispositioned -- this PR extends a scoped subset, not the full issue. @@ -13,13 +13,35 @@ SFT/preference training against four context-view baselines, and (4) held-out measurement of action/operator/argument/reference/branch accuracy, calibration, and CAP0/CAP1/CAP2 retention. -This slice adds the seventh and final named pattern, **pronoun-focus- -followup** (see "Fifth slice (v6)" below), bringing extraction coverage to -7 of 7. All seven named patterns from the issue's own list now extract and -replay-verify. What remains is the issue's separate, still fully unattempted -training/measurement scope: SFT/preference training against the -DSH3-selected policy/control heads, the four-baseline comparison, held-out -benefit measurement, and turn-depth/context-view ablations. +The fifth slice added the seventh and final named pattern, bringing +extraction coverage to 7 of 7 (see "Fifth slice (v6)" below). The sixth +slice added the first (and, until then, entirely missing) converter from an +extracted row to the `PreferencePair` shape `scripts/train_preference.py` +actually consumes. The seventh slice ran the first real +(`fixture_or_scratch`) end-to-end pass: a scratch SFT checkpoint, a demo +replay-preference pairs corpus covering 3 of the 7 named patterns, and one +bounded `scripts/train_preference.py train` call against it. This slice +extends the demo corpus to a fourth pattern, `merge_success` -- the one +pattern the seventh slice's trace-scan corpus could not reach -- and reruns +the same training chain against the now-richer 3-pair corpus; see "Eighth +slice" below. This slice takes the first real step onto the issue's actual +named training target: `typed_operator_policy.py`'s +`TypedOperatorPolicyScorer`, not the generic TwoTower pair format the +sixth/seventh/eighth slices used for tooling compatibility. Only one of the +seven named patterns (`pronoun_focus_followup`) is honestly representable +there today -- see "Ninth slice" below for why, and for a real, structural +(not a bug) null-training finding this exposed. What remains is a *real* +pairs corpus (no captured conversation-trace data exists anywhere in this +repo -- see below), a scope decision for the other six patterns' history- +control/merge actions (none has a row in the typed policy's action space), +the four-baseline comparison, held-out benefit measurement, and turn-depth/ +context-view ablations. This slice does not attempt that scope decision; +instead it closes a smaller, independent gap on the generic TwoTower demo +corpus side: two of the seven named patterns (`partial_rollback`, +`fork_then_choose_one_branch`) were still absent from +`scripts/build_replay_preference_pairs.py`'s demo corpus even though +extraction has covered them since the fifth slice -- see "Tenth slice" +below. ## What this PR delivers @@ -288,10 +310,336 @@ training/evaluation work enumerated above. snapshot is untouched, staying immutable point-in-time evidence from before this slice landed. +## Sixth slice + +* New module `src/slm_training/harnesses/preference/replay_pairs.py`: + `render_replay_preference_pair` renders one `OperatorReplayPreferenceRowV1` + into a `PreferencePair` (`prompt`/`chosen`/`rejected`, the shape + `scripts/train_preference.py`'s `build-pairs`/`train` path consumes), and + `render_replay_preference_pairs` batches a report's rows. Before this + slice **nothing in the repo converted a row into any trainable shape** -- + confirmed by grepping the whole tree for `ReplayPreferenceRelation`, + `extract_replay_preference_rows`, and `extract_merge_preference_row`: only + the operators module itself, its own test file, the version-stamp + registry, and this doc referenced them. +* **Never fabricates a state.** `chosen_output_state_id` is always an + already-materialized node by construction (true of every row every prior + slice has produced). The rejected side is resolved the same + replay-grounded way, with no shortcut: + * `undo` -> the input state's own parent (an existing node -- no + computation). + * `redo:` / `checkout:` -> that state is already + materialized; the id is read directly out of the action string. + * an operator action's serialized form (the `pronoun_focus_followup` + sibling case) -> recomputes the exact legal set at the input state + (`enumerate_operator_legal_set`, the identical call + `extract_replay_preference_rows` itself makes), matches the row's + `rejected_action` string against it, and actually applies it through + the pack-authorized `OperatorLibraryV1.apply` -- the same executor + every other application in this module goes through. This is a real, + independently-reproducible state, not a guess. + * `merge:` as a *rejected* action is deliberately left unrendered + (returns `None`). It cannot occur under today's single-merge-candidate + extraction (`extract_merge_preference_row` only ever offers one + `merge:` candidate, and it is always the *chosen* side of a + `MERGE_SUCCESS` row -- see `test_rejected_merge_action_is_never_rendered` + for the defensive-branch proof), but the renderer refuses to guess a + merged state instead of honestly declining if that ever changes. +* **Two call shapes, one renderer.** Trace-turn-scan rows + (`extract_replay_preference_rows`) resolve nodes via + `resolve_node=trace.node` directly. `MERGE_SUCCESS` rows + (`extract_merge_preference_row`) never live on a shared + `ConversationTraceV1` -- `left`/`right` are independently-verified + `BranchEditV1` edges and `decision.continuation.merged_node` is a fresh + node -- so the new `merge_node_resolver(left, right, decision)` builds the + equivalent `NodeResolver` over exactly those nodes instead. +* **Open, explicitly-flagged modeling choice:** the pair's `prompt` is set + to the input state's own DSL source (score chosen/rejected + *continuations* of the current AST against it). This is a first, + documented cut for compatibility with the existing generic TwoTower pair + format, not a validated training-objective decision -- the disposition's + own remaining-scope note names the DSH3-selected policy/control heads + (`typed_operator_policy.py`) as the actual training target, and whoever + wires a real training run should treat the prompt shape as open rather + than inherited from this slice. +* **Still wiring only.** This slice adds the converter and its tests; it + does not build a pairs corpus from real conversation traces, does not run + `scripts/train_preference.py`, and makes no training or held-out-benefit + claim. `harness.preference.replay_pairs` registered fresh (`v1`, initial + registration) in `src/slm_training/resources/versions.json`; no existing + component's behavior changed (`dsl.operators.replay_preference` stays at + `v6` -- this slice only *consumes* its existing public API). + +## Seventh slice + +* New script `scripts/build_replay_preference_pairs.py`: builds one small, + deterministic, honestly-labeled scratch conversation (`build_demo_trace`) + exercising three of the seven named patterns (edit-then-undo, + undo-then-redo, checkout-another-state) with a toy zero-argument cycling + operator, extracts rows, renders them via the sixth slice's + `render_replay_preference_pairs`, and writes a real `pairs.jsonl` via the + existing `write_pairs`. This is the **first real, on-disk pairs corpus + this feature line has ever produced** -- everything before this slice was + either an in-memory row/pair in a unit test, or a function that could + render one but had never been run outside `pytest`. +* **No real corpus exists to build from.** Confirmed again this slice (grep + for `"conversation_trace"` / `"schema": "conversation_trace` across + `src/slm_training/resources/`): zero persisted `ConversationTraceV1` + records anywhere in this repo, and no harness ingests captured + conversation history (`build_symbolic_operator_corpus` in + `harnesses/train_data/operator_corpus.py` *synthesizes* traces + combinatorially from existing gold DSL records; it does not read + real/captured usage). The demo trace here is explicitly scratch, not a + stand-in for that missing corpus. +* **2 of 3 rows render, honestly.** `edit_then_undo` and + `checkout_another_state` render real, non-degenerate pairs. + `undo_then_redo` never does, for a structural reason, not a bug: for + *any* deterministic, zero-argument operator, `redo` and "reapply the same + operator at the same input state" are, by construction, the identical + resulting text, so the renderer's own dedup guard + (`render_replay_preference_pair`) correctly declines rather than emitting + a self-contradictory pair. The script's own printed report says so + (`pairs_dropped: 1`) rather than silently hiding it. +* **First real training run using DSH5-10 rows, full pipeline, one command + chain:** + ```bash + python -m scripts.train_model --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \ + --model twotower --context-backend scratch --steps 8 \ + --run-id replay_pref_sft_ckpt --no-sync-checkpoints --device cpu --seed 0 + python -m scripts.build_replay_preference_pairs \ + --out outputs/data/preference/replay_demo_pairs.jsonl + python -m scripts.train_preference train \ + --checkpoint outputs/runs/replay_pref_sft_ckpt/checkpoints/last.pt \ + --pairs outputs/data/preference/replay_demo_pairs.jsonl \ + --out-dir outputs/runs/replay_pref_dpo --steps 6 --device cpu + ``` + SFT step: `last_loss=32.610084533691406` -- identical to every prior + `wf_smoke_v2`/seed-0/8-step row in + `docs/design/autotrain-loop-ledger-20260725.md` (16+ prior independent + reproductions), confirming this checkpoint is the same deterministic + artifact those rows already verified, not a new unverified path. + Preference step: `{"steps": 6, "last_loss": 1.0767018795013428, + "mean_loss": 0.9917331635951996, "n_pairs": 2, "reference_free": true}` + (`outputs/runs/replay_pref_dpo/preference_summary.json`, not committed -- + `outputs/` is gitignored). Both commands completed in well under + `MAX_RUN_MINUTES=3` (SFT ~10s per the ledger's own prior timings for this + exact recipe; the 6-step preference pass over 2 pairs on CPU is + comparably fast). +* **Still not a training or held-out-benefit claim.** `n_pairs=2` on a + scratch fixture with no held-out split is `fixture_or_scratch` wiring + evidence that the pipeline *runs end to end for real* -- SFT checkpoint + in, DSH5-10-extracted-and-rendered pairs in, a real + `train_preference.py train` loss trajectory out. It says nothing about + whether this signal helps the model, generalizes, or should train the + DSH3-selected policy head the issue actually asks about. +* `harness.preference.replay_pairs` bumped `v1` -> `v2` in + `src/slm_training/resources/versions.json` (adds the new script + test to + its watched paths). + +## Eighth slice + +* Adds `build_demo_merge_scenario` to `scripts/build_replay_preference_pairs.py`: + a second scratch fixture -- two branches forked from a shared base editing + disjoint node refs (title vs body) -- mirroring the exact disjoint-target + shape `tests/test_dsl/test_operator_merge.py` already verifies merges + cleanly, replayably, and order-invariantly. This reaches `merge_success`, + the one named pattern the seventh slice's single-trace corpus structurally + cannot: `extract_merge_preference_row` never operates on a shared + `ConversationTraceV1` (see the sixth slice's `merge_node_resolver`), so it + needs its own two-branch construction rather than another turn in the + same trace. +* `main()` now combines both sources into one report and one `pairs.jsonl`: + 4 rows total (3 from the trace-scan corpus + 1 `merge_success`), 3 render. + `undo_then_redo` is still the only drop, for the same structural reason + the seventh slice documented (never a bug to fix on this fixture family). +* Reran the full training chain against the now-richer corpus: + ```bash + python -m scripts.train_model --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \ + --model twotower --context-backend scratch --steps 8 \ + --run-id replay_pref_sft_ckpt2 --no-sync-checkpoints --device cpu --seed 0 + python -m scripts.build_replay_preference_pairs \ + --out outputs/data/preference/replay_demo_pairs_v2.jsonl + python -m scripts.train_preference train \ + --checkpoint outputs/runs/replay_pref_sft_ckpt2/checkpoints/last.pt \ + --pairs outputs/data/preference/replay_demo_pairs_v2.jsonl \ + --out-dir outputs/runs/replay_pref_dpo2 --steps 9 --device cpu + ``` + SFT step: `last_loss=32.610084533691406` again -- the same deterministic + artifact every prior `wf_smoke_v2`/seed-0/8-step row in the smoke-loop + ledger reproduces. Preference step, now over 3 pairs instead of 2: + `{"steps": 9, "last_loss": 0.5314897894859314, "mean_loss": + 0.7201318964362144, "n_pairs": 3, "reference_free": true}` + (`outputs/runs/replay_pref_dpo2/preference_summary.json`, not committed). + Both commands again well under `MAX_RUN_MINUTES=3`. +* **Still not a training or held-out-benefit claim** -- `n_pairs=3` on a + scratch fixture is a larger, more structurally diverse smoke corpus (now + covering 4 of the 7 named patterns instead of 3), not evidence the signal + helps the model or generalizes. The three remaining un-exercised-in-a-script + patterns (`partial_rollback`, `fork_then_choose_one_branch`, and + `pronoun_focus_followup`) are left for a future slice rather than piling + more scratch fixtures onto this one; see the doc's still-open + training/measurement scope above. +* `harness.preference.replay_pairs` bumped `v2` -> `v3` in + `src/slm_training/resources/versions.json`. + +## Ninth slice + +Every slice from the sixth onward has flagged the same gap: the sixth +through eighth slices' `PreferencePair`/TwoTower path exists only for +tooling compatibility, since the disposition's own remaining-scope note +names `typed_operator_policy.py`'s `TypedOperatorPolicyScorer` -- not +TwoTower -- as the issue's actual training target. This slice takes the +first real step onto that target, and honestly narrows what's reachable +there. + +* New module `src/slm_training/harnesses/experiments/argument_preference.py`: + * `build_argument_preference_example` renders one row into a + `TypedOperatorArgumentPreferenceExampleV1` -- but **only for + `pronoun_focus_followup`**. `OperatorPolicyInputV1.action_rows` is built + only from `legal_set.entries` (operator-registry actions; + `build_operator_policy_input` in + `slm_training.models.operator_policy_view`). The other six named + patterns' `chosen_action`/`rejected_action` are history controls + (`undo`, `redo:`, `checkout:`) or `merge:` -- none of + which has a row in that space at all. `pronoun_focus_followup` is the + one pattern whose chosen and rejected actions are the *same* operator + with a different bound argument for the *same* slot: a genuine + argument-selection preference the typed policy's + `argument_head` (`CandidateScoringHead`) can score. Wiring the other + six patterns would require a real scope decision about what an "action + row" even means for a control action -- left open here, not guessed at. + * `typed_operator_argument_preference_loss` is a Bradley-Terry pairwise + margin, `-log_sigmoid(chosen_logit - rejected_logit)`, over the two + candidates' `CandidateScoringHead` logits for the differing slot. + Surrogate preference loss, not textbook DPO -- the same honesty note + `scripts/train_preference.py`'s own `dpo_loss` already carries for the + generic TwoTower path. + * `train_typed_operator_argument_preference` mirrors + `train_typed_operator_policy`'s own matched full-batch schedule exactly, + over this new loss. +* **Real, structural finding, not a bug:** training on the actual + `pronoun_focus_followup` fixture + (`tests/test_dsl/test_replay_preference.py`'s `_pronoun_focus_fixture`) + provably cannot move the loss. Both sibling refs share every field + `ReferenceModelViewV1` exposes (`ref_kind=VALUE`, `value_type= + openui.string`, no parent, no position) -- they differ only by + `semantic_fingerprint`, which `FORBIDDEN_FIELD_NAMES` in + `slm_training.models.operator_policy_view` deliberately strips from every + model input as anti-identity-leakage. `OperatorFeatureEncoder. + _reference_embeddings` is a pure function of exactly those allowed + fields, with no row-index feature, so two feature-identical candidates + get byte-identical embeddings through the shared-weight + `CandidateScoringHead` regardless of any parameter update: the loss sits + at `-log_sigmoid(0) = ln(2)` structurally, provably, for as many steps as + you run it. `test_training_cannot_move_the_loss_when_candidates_are_feature_identical` + proves this is exact (`pytest.approx`, not "roughly unchanged"); a + second test with a synthetic, feature-*distinguishable* pair (differing + `relative_position`) proves the loss function and gradient flow + themselves work correctly (`test_training_reduces_the_pairwise_loss_when_candidates_differ`) + -- this is a property of *this fixture's* candidates, not of the + mechanism. + * Consequence for the open training/measurement scope above: any real + corpus of `pronoun_focus_followup` rows will only carry a learnable + argument-preference signal for pairs whose candidates differ in + `ref_kind`/`value_type`/`compiler_facts`/`has_parent`/ + `relative_position`/selector fields -- feature-identical siblings + (plausibly common for repeated same-type VALUE refs, exactly the + minimal case this fixture represents) are structurally unlearnable + signal by this scorer's own anti-leakage design, not a data-quantity + problem. Worth surfacing before anyone builds a real corpus and is + puzzled why training on it plateaus. +* No harness code outside the new module changed; `typed_operator_policy.py` + itself is unchanged (only consumed, not modified). +* `harness.experiments.argument_preference` registered fresh (`v1`, initial + registration) in `src/slm_training/resources/versions.json`. + +## Tenth slice + +This slice does not attempt the ninth slice's flagged scope decision +(what an "action row" means for a history-control or merge action in the +typed policy's action space) -- that remains genuinely open. Instead it +closes a smaller, independent gap on the generic TwoTower demo-corpus side +that the eighth and ninth slices both left explicitly unaddressed: two of +the seven named patterns were still absent from +`scripts/build_replay_preference_pairs.py`'s demo corpus even though +`extract_replay_preference_rows` has covered them since the fifth slice. + +* Two new standalone scratch traces, `build_demo_partial_rollback_scenario` + and `build_demo_fork_choose_scenario`, mirror the exact shapes + `tests/test_dsl/test_replay_preference.py`'s own + `test_partial_rollback_yields_a_row_for_the_second_consecutive_undo` and + `test_fork_then_return_to_original_branch_yields_a_distinct_relation` + already verify -- root->edit1->edit2->undo->undo for `partial_rollback` + (a second consecutive undo, no intervening edit), and + root->edit->fork->checkout(pre-fork branch) for + `fork_then_choose_one_branch` -- using the script's existing toy 4-state + cycling operator rather than inventing a new one. +* The three trace-building scenarios' near-duplicated pack/library/root + setup was factored into two shared helpers, `_new_toy_trace` and + `_apply_cycle`, rather than copied a third time; `build_demo_trace` + itself is behavior-unchanged (same rows, same pairs) after the refactor, + which the unmodified `test_build_demo_trace_exercises_three_named_patterns` + test confirms. +* `main()` now combines four sources instead of two: **7 rows total** (2 + `edit_then_undo` + 1 each of `undo_then_redo` / `checkout_another_state` / + `partial_rollback` / `fork_then_choose_one_branch` / `merge_success`), **6 + render**. `undo_then_redo` is still the only drop, for the same + structural reason the seventh slice documented (never a bug on this + fixture family). The `partial_rollback` scenario's own extra + `edit_then_undo` row is kept and counted honestly rather than + cherry-picked out, matching every prior slice's convention of reporting + everything a scenario extracts. +* **6 of the 7 named patterns are now exercised in this demo TwoTower + corpus.** Only `pronoun_focus_followup` remains absent here -- and + deliberately so: the ninth slice already covers it on the separate + `typed_operator_policy` argument-preference path (a different pair + format from this script's generic TwoTower one), and wiring it into this + corpus too would just duplicate that slice's already-documented + structural (feature-identical-siblings) null-training finding rather + than add new information. +* Reran the full training chain against the richer 6-pair corpus: + ```bash + python -m scripts.train_model --train-dir src/slm_training/resources/data/train/wf_smoke_v2 \ + --model twotower --context-backend scratch --steps 8 \ + --run-id replay_pref_sft_ckpt3 --no-sync-checkpoints --device cpu --seed 0 + python -m scripts.build_replay_preference_pairs \ + --out outputs/data/preference/replay_demo_pairs_v4.jsonl + python -m scripts.train_preference train \ + --checkpoint outputs/runs/replay_pref_sft_ckpt3/checkpoints/last.pt \ + --pairs outputs/data/preference/replay_demo_pairs_v4.jsonl \ + --out-dir outputs/runs/replay_pref_dpo3 --steps 12 --device cpu + ``` + SFT step: `last_loss=32.610084533691406` again -- the same deterministic + artifact every prior `wf_smoke_v2`/seed-0/8-step row in this ledger + reproduces. Preference step, now over 6 pairs instead of 3: + `{"steps": 12, "last_loss": 1.1219162940979004, "mean_loss": + 1.0559024934967358, "n_pairs": 6, "reference_free": true}` + (`outputs/runs/replay_pref_dpo3/preference_summary.json`, not committed). + Both commands again well under `MAX_RUN_MINUTES=3`. +* **Still not a training or held-out-benefit claim.** `n_pairs=6` on a + scratch fixture with no held-out split is a broader, more structurally + diverse smoke corpus (now covering 6 of 7 named patterns instead of 4), + not evidence the signal helps the model or generalizes. The ninth + slice's flagged scope decision for wiring history-control/merge actions + into the typed policy's own action space remains the actual open item + for the issue's real training target. +* `harness.preference.replay_pairs` bumped `v3` -> `v4` in + `src/slm_training/resources/versions.json`. + ## Reproducibility ```bash -NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py +NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_harnesses/preference/test_replay_pairs.py tests/test_scripts/test_build_replay_preference_pairs.py tests/test_harnesses/experiments/test_argument_preference.py tests/test_harnesses/experiments/test_typed_operator_policy.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py ``` -Result (this PR, real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. +Result (fifth-slice PR, real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. + +Result (sixth-slice PR #1124, real run in a fresh `.venv-dsh510` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `env -u NODE_OPTIONS npm ci` in `src/apps/openui_bridge` -- the ambient `NODE_OPTIONS="--import tsx" --max-old-space-size=8192` is rejected outright by Node for both `npm ci` and `pytest` in this environment, so it has to be unset, not just locally overridden, unlike the fifth slice's note above): `69 passed` against `main` HEAD `5f94b92` (includes the fifth slice, already merged). Also verified: `ruff check` clean on both new files; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. No training run in this slice; `outputs/` untouched. + +Result (seventh-slice PR #1125, real run in a fresh `.venv-dsh510`, same environment recipe as the sixth slice above, stacked on top of PR #1124 which was still unmerged when this slice started): `71 passed` (69 from the sixth slice + 2 new). Also verified: `ruff check` clean on both new files; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`. Plus the real training run described above (SFT checkpoint + demo pairs + preference-training pass, both commands well under `MAX_RUN_MINUTES=3`); its `outputs/runs/replay_pref_sft_ckpt/` and `outputs/runs/replay_pref_dpo/` are not committed (`outputs/` is gitignored) per this repo's checked-not-committed convention for scratch run artifacts. + +Result (eighth-slice PR #1126, real run in a fresh `.venv-dsh510`, same environment recipe as above, stacked on top of PR #1125 which was still unmerged when this slice started): `72 passed` (71 from the seventh slice + 1 new). Also verified: `ruff check` clean; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. Plus the reran training chain described above; `outputs/runs/replay_pref_sft_ckpt2/` and `outputs/runs/replay_pref_dpo2/` are not committed (`outputs/` is gitignored). + +Result (this PR, ninth slice, real run in the same `.venv-dsh510`, stacked on top of PR #1126 which was still unmerged when this slice started): `92 passed` (72 from the prior slices + 6 new in `test_argument_preference.py`, plus `test_typed_operator_policy.py`'s own 14 pre-existing tests now included in this suite's reproduction command for the first time since this slice touches that module's consumer surface). Also verified: `ruff check` clean on both new files; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. No end-to-end training run in this slice (the `train_typed_operator_argument_preference` calls are inside the test suite itself, proving the mechanism works on a synthetic distinguishable pair and correctly plateaus on the real fixture's feature-identical pair -- not a separate `outputs/`-writing run). + +Result (this PR, tenth slice, real run in a fresh `.venv-autotrain` -- Python 3.12.3, `torch==2.5.1+cu124` (CPU-only invocation via `--device cpu`), `pip install -e ".[dev,grammar]"`, plus `env -u NODE_OPTIONS npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gate, same recipe as every prior slice's note above -- stacked on top of PR #1127 which was still unmerged when this slice started, base commit `9be1464`): `94 passed` (92 from the ninth slice + 2 new in `tests/test_scripts/test_build_replay_preference_pairs.py`). Also verified: `ruff check` clean on both changed files; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-ni43oh-pronoun-focus-policy` -- `ok (1 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean. Plus the reran training chain described above (SFT checkpoint + 6-pair demo corpus + preference-training pass, both commands well under `MAX_RUN_MINUTES=3`); `outputs/runs/replay_pref_sft_ckpt3/` and `outputs/runs/replay_pref_dpo3/` are not committed (`outputs/` is gitignored). diff --git a/scripts/build_replay_preference_pairs.py b/scripts/build_replay_preference_pairs.py new file mode 100644 index 000000000..b4448523a --- /dev/null +++ b/scripts/build_replay_preference_pairs.py @@ -0,0 +1,519 @@ +#!/usr/bin/env python3 +"""Build a first replay-preference-pairs corpus and report it honestly. + +SLM-418 (DSH5-10)'s extraction (7/7 patterns) and rendering +(``slm_training.harnesses.preference.replay_pairs``, sixth slice) both +exist, but no real captured multi-turn ``ConversationTraceV1`` corpus does +-- confirmed while scoping this slice: `resources/data/` has no persisted +``conversation_trace/v1`` records anywhere, and no harness ingests one (see +``docs/design/dsh5-10-replay-preference-rows.md``). This script does not +invent one. It builds a single small, deterministic, self-contained +conversation exercising several of the seven named patterns (edit-then- +undo, undo-then-redo, checkout-another-state) with the same toy +zero-argument cycling operator this repo's own DSH5-10 test suite uses, so +the pipeline (extract -> render -> write pairs.jsonl) can be run and +measured end to end for real. This is ``fixture_or_scratch`` corpus, the +same honesty class as the smoke-loop recipes in +``docs/design/autotrain-loop-ledger-20260725.md`` -- not a claim about real +usage patterns, model quality, or held-out benefit. + +One of the four rows this produces (``undo_then_redo``) never renders: for +any deterministic, zero-argument operator, ``redo`` and "reapply the same +operator at the same input state" are, by definition, the identical text -- +no cycle-length trick avoids this; only a genuinely non-deterministic or +argument-bearing operator would. The renderer's own dedup guard +(``render_replay_preference_pair`` in +``slm_training.harnesses.preference.replay_pairs``) correctly declines +rather than emitting a self-contradictory pair -- see this script's own +``pairs_dropped`` in its printed report, and +``test_undo_then_redo_row_declines_a_degenerate_collapse`` in +``tests/test_harnesses/preference/test_replay_pairs.py``. + +A second, separate scenario (``build_demo_merge_scenario``) covers +``merge_success``, the one named pattern ``build_demo_trace`` cannot reach: +``extract_merge_preference_row`` never operates on a shared +``ConversationTraceV1`` the way the other six patterns do (see +``merge_node_resolver`` in ``slm_training.harnesses.preference.replay_pairs``), +so it needs its own two-branch fixture with disjoint-target edits, mirroring +the shape ``tests/test_dsl/test_operator_merge.py`` already verifies is +valid, replayable, and order-invariant. + +Two more standalone scratch traces -- ``build_demo_partial_rollback_scenario`` +and ``build_demo_fork_choose_scenario`` (tenth slice) -- cover +``partial_rollback`` (a second consecutive undo, not preceded by an +intervening edit) and ``fork_then_choose_one_branch`` (a checkout back +across a fork boundary), mirroring the exact shapes +``tests/test_dsl/test_replay_preference.py``'s own +``test_partial_rollback_yields_a_row_for_the_second_consecutive_undo`` and +``test_fork_then_return_to_original_branch_yields_a_distinct_relation`` +already verify, using this script's own toy cycling operator rather than +inventing a new one. That leaves ``pronoun_focus_followup`` as the only +named pattern absent from this demo TwoTower corpus -- the ninth slice +covers it on the separate `typed_operator_policy` argument-preference path +instead (see the disposition doc's "Ninth slice"); wiring it here too would +just duplicate that slice's already-documented structural finding on the +generic pair format this script produces. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from dataclasses import replace +from pathlib import Path + +from slm_training.dsl.operators import ( + ActionEffectV1, + ApplicationProvenanceV1, + AstOperatorV1, + BranchEditV1, + CompilerCoverage, + ConversationStateNodeV1, + EffectDeltaKind, + EffectDeltaV1, + OperatorLibraryV1, + OperatorMutationV1, + OperatorReplayPreferenceRowV1, + OperatorStateV1, + RefKind, + ReferenceDescriptorV1, + RegisteredOperatorV1, + append_operator_turn, + branch_fingerprint, + build_reference_table, + checkout_conversation_state, + clone_reference_table_for_branch, + create_conversation_trace, + extract_merge_preference_row, + extract_replay_preference_rows, + fork_conversation, + merge_conversation_branches, + redo_conversation, + undo_conversation, +) +from slm_training.dsl.operators.conversation import ConversationTraceV1 +from slm_training.dsl.pack import DslPack, get_pack +from slm_training.harnesses.preference import PreferencePair, write_pairs +from slm_training.harnesses.preference.replay_pairs import ( + merge_node_resolver, + render_replay_preference_pair, + render_replay_preference_pairs, +) + +_MERGE_SOURCE = 'root = Card([TextContent(":hero.title"), TextContent(":hero.body")], "clear")' + +_SOURCE = 'root = TextContent(":hero.title")' +_OPERATOR_ID = "openui.demo_cycle_text" +_REQUEST_ID = "replay-pairs-demo" + + +def _sha(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +def _provenance(state: OperatorStateV1) -> ApplicationProvenanceV1: + return ApplicationProvenanceV1( + pack_id="openui", + compiler_id="scripts.build_replay_preference_pairs", + compiler_version="v1", + source_artifact_digest=_sha(state.source), + request_id=_REQUEST_ID, + ) + + +def _table(state: OperatorStateV1, branch: str, *, seed: int): + return build_reference_table( + request_id=_REQUEST_ID, + state_digest=state.state_digest, + branch_digest=branch, + descriptors=( + ReferenceDescriptorV1( + ref_kind=RefKind.VALUE, + semantic_fingerprint=_sha("demo-value"), + value_type="openui.string", + ), + ), + seed=seed, + ) + + +def _execute(state: OperatorStateV1, _arguments) -> OperatorMutationV1: + # A 4-state cycle, not 3: long enough that the checkout-back-to-root + # target in build_demo_trace() below doesn't coincidentally equal what + # one more deterministic reapplication of this same operator would + # produce at the checkout decision state (which would make the + # rendered pair degenerate -- see the module's own + # undo_then_redo note for why that's unavoidable for a *different* + # reason on that pattern, but avoidable here by construction). + replacements = ( + (":hero.title", ":hero.body"), + (":hero.body", ":hero.caption"), + (":hero.caption", ":hero.footer"), + (":hero.footer", ":hero.title"), + ) + for before, after in replacements: + if before in state.source: + return OperatorMutationV1( + source=state.source.replace(before, after), + effect=ActionEffectV1(compiler_coverage=CompilerCoverage.EXACT), + ) + raise ValueError("demo.no_transition") + + +def _new_toy_trace( + *, branch_label: str +) -> tuple[DslPack, OperatorLibraryV1, ConversationTraceV1]: + """One fresh, standalone scratch conversation using this script's own + toy 4-state cycling operator (see ``_execute``). Every demo scenario + below starts from its own call to this helper so scenarios never share + trace/branch state -- each is independently replayable and its rows + are independently attributable to that one scenario.""" + base_pack = get_pack("openui") + root_state = OperatorStateV1.from_source(base_pack, _SOURCE) + declaration = AstOperatorV1( + operator_id=_OPERATOR_ID, + version="v1", + domain="openui.ast", + codomain="openui.ast", + argument_slots=(), + preconditions=(), + effect_signature=(), + locality="node", + cost=1.0, + ) + library = OperatorLibraryV1((RegisteredOperatorV1(declaration, _execute),)) + pack = replace(base_pack, operator_library=library) + branch = branch_fingerprint(root_state.state_digest, _sha(f"demo-branch-{branch_label}")) + root_table = _table(root_state, branch, seed=1) + trace = create_conversation_trace( + pack=pack, + root_state=root_state, + root_reference_table=root_table, + provenance=_provenance(root_state), + ) + return pack, library, trace + + +def _apply_cycle( + pack: DslPack, library: OperatorLibraryV1, trace: ConversationTraceV1, *, seed: int +) -> ConversationTraceV1: + result = library.apply( + pack, trace.current.state, _OPERATOR_ID, (), _provenance(trace.current.state) + ) + assert result.succeeded and result.state is not None + return append_operator_turn( + trace, + pack=pack, + library=library, + application=result.application, + output_reference_table=_table(result.state, trace.current.branch_digest, seed=seed), + ) + + +def build_demo_trace() -> tuple[DslPack, OperatorLibraryV1, ConversationTraceV1]: + """A single scratch trace exercising three of the seven named patterns. + + root -> edit -> undo (edit_then_undo) -> redo (undo_then_redo) -> edit + -> checkout(root) (checkout_another_state). Deterministic (fixed seeds, + no randomness); see the module docstring for why this is not real + captured conversation data. + """ + pack, library, trace = _new_toy_trace(branch_label="main") + edited = _apply_cycle(pack, library, trace, seed=2) + original_child_id = edited.current_state_id + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + redone = redo_conversation( + undone, target_state_id=original_child_id, provenance=_provenance(undone.current.state) + ) + edited_again = _apply_cycle(pack, library, redone, seed=3) + checked_out = checkout_conversation_state( + edited_again, + target_state_id=edited_again.root_state_id, + provenance=_provenance(edited_again.current.state), + ) + return pack, library, checked_out + + +def build_demo_partial_rollback_scenario() -> ( + tuple[DslPack, OperatorLibraryV1, ConversationTraceV1] +): + """A second, standalone scratch trace reaching ``partial_rollback``. + + root -> edit1 -> edit2 -> undo (edit_then_undo, edit2's state) -> undo + (partial_rollback, edit1's state) -- mirrors + ``test_partial_rollback_yields_a_row_for_the_second_consecutive_undo``. + Both rows this trace yields are kept and reported honestly (this + scenario is not cherry-picked down to only the new relation). + """ + pack, library, trace = _new_toy_trace(branch_label="rollback") + edited_once = _apply_cycle(pack, library, trace, seed=2) + edited_twice = _apply_cycle(pack, library, edited_once, seed=3) + undone_once = undo_conversation( + edited_twice, provenance=_provenance(edited_twice.current.state) + ) + undone_twice = undo_conversation( + undone_once, provenance=_provenance(undone_once.current.state) + ) + return pack, library, undone_twice + + +def build_demo_fork_choose_scenario() -> ( + tuple[DslPack, OperatorLibraryV1, ConversationTraceV1] +): + """A third, standalone scratch trace reaching ``fork_then_choose_one_branch``. + + root -> edit (main branch) -> fork (new branch) -> checkout(main) -- + mirrors + ``test_fork_then_return_to_original_branch_yields_a_distinct_relation``. + """ + pack, library, trace = _new_toy_trace(branch_label="fork") + edited = _apply_cycle(pack, library, trace, seed=2) + main_branch_state_id = edited.current_state_id + forked = fork_conversation( + edited, + branch_nonce_digest=_sha("demo-fork-target"), + reference_seed=8, + provenance=_provenance(edited.current.state), + ) + checked_out = checkout_conversation_state( + forked, + target_state_id=main_branch_state_id, + provenance=_provenance(forked.current.state), + ) + return pack, library, checked_out + + +def _merge_branch_edit( + *, + base_pack: DslPack, + base_state: OperatorStateV1, + base_node: ConversationStateNodeV1, + root_table, + name: str, + target_name: str, + replacement: str, + seed: int, + authorities: dict, +) -> BranchEditV1: + branch = branch_fingerprint(base_state.state_digest, _sha(f"demo-merge-{name}")) + table = clone_reference_table_for_branch(root_table, branch_digest=branch, seed=seed) + input_node = ConversationStateNodeV1( + parent_state_id=base_node.state_id, + branch_digest=branch, + state=base_state, + reference_table=table, + ) + target = next( + entry.ref for entry in table.entries if entry.descriptor.value_type == f"openui.{target_name}" + ) + operator_id = f"openui.demo_merge_{name}" + declaration = AstOperatorV1( + operator_id=operator_id, + version="v1", + domain="openui.ast", + codomain="openui.ast", + argument_slots=(), + preconditions=(), + effect_signature=(EffectDeltaKind("property"),), + locality="node", + cost=1.0, + ) + before = f":hero.{target_name}" + + def execute(state: OperatorStateV1, _arguments) -> OperatorMutationV1: + if before not in state.source: + raise ValueError("demo.no_transition") + return OperatorMutationV1( + source=state.source.replace(before, replacement), + effect=ActionEffectV1( + property_deltas=( + EffectDeltaV1( + kind=EffectDeltaKind("property"), target=target, before="before", after="after" + ), + ), + compiler_coverage=CompilerCoverage.EXACT, + ), + ) + + branch_library = OperatorLibraryV1((RegisteredOperatorV1(declaration, execute),)) + branch_pack = replace(base_pack, operator_library=branch_library) + applied = branch_library.apply( + branch_pack, input_node.state, operator_id, (), _provenance(input_node.state) + ) + assert applied.succeeded and applied.state is not None + output_table = build_reference_table( + request_id=table.request_id, + state_digest=applied.state.state_digest, + branch_digest=branch, + descriptors=tuple(entry.descriptor for entry in table.entries), + seed=seed + 10, + ) + output_node = ConversationStateNodeV1( + parent_state_id=input_node.state_id, + branch_digest=branch, + state=applied.state, + reference_table=output_table, + ) + authorities[input_node.state_id] = (branch_pack, branch_library) + return BranchEditV1(input_node, output_node, applied.application) + + +def _rebuild_merged_table(_pack, merged_state: OperatorStateV1, branch_digest: str, seed: int): + """Fixture-only compiler facts, derived anew from the canonical merged source.""" + descriptors = tuple( + ReferenceDescriptorV1( + ref_kind=RefKind.NODE, + semantic_fingerprint=_sha(f"canonical:{marker}"), + value_type=f"openui.{marker.removeprefix(':hero.')}", + ) + for marker in (":hero.heading", ":hero.copy") + if marker in merged_state.source + ) + return build_reference_table( + request_id=_REQUEST_ID, + state_digest=merged_state.state_digest, + branch_digest=branch_digest, + descriptors=descriptors, + seed=seed, + ) + + +def build_demo_merge_scenario() -> tuple[list[OperatorReplayPreferenceRowV1], list[PreferencePair]]: + """A second scratch scenario covering ``merge_success``, the one pattern + ``build_demo_trace`` cannot reach (see module docstring). + + Two branches fork from a shared base and edit disjoint node refs + (title vs body) -- the same disjoint-target shape + ``tests/test_dsl/test_operator_merge.py`` verifies merges cleanly, + replayably, and order-invariantly. + """ + base_pack = get_pack("openui") + base_state = OperatorStateV1.from_source(base_pack, _MERGE_SOURCE) + root_branch = branch_fingerprint(base_state.state_digest, _sha("demo-merge-root")) + descriptors = tuple( + ReferenceDescriptorV1( + ref_kind=RefKind.NODE, semantic_fingerprint=_sha(name), value_type=f"openui.{name}" + ) + for name in ("title", "body") + ) + root_table = build_reference_table( + request_id=_REQUEST_ID, + state_digest=base_state.state_digest, + branch_digest=root_branch, + descriptors=descriptors, + seed=21, + ) + base_node = ConversationStateNodeV1( + parent_state_id=None, branch_digest=root_branch, state=base_state, reference_table=root_table + ) + + authorities: dict = {} + left = _merge_branch_edit( + base_pack=base_pack, + base_state=base_state, + base_node=base_node, + root_table=root_table, + name="left", + target_name="title", + replacement=":hero.heading", + seed=22, + authorities=authorities, + ) + right = _merge_branch_edit( + base_pack=base_pack, + base_state=base_state, + base_node=base_node, + root_table=root_table, + name="right", + target_name="body", + replacement=":hero.copy", + seed=23, + authorities=authorities, + ) + + def resolve(node: ConversationStateNodeV1): + return authorities[node.state_id] + + decision = merge_conversation_branches( + pack=base_pack, + base=base_node, + left=left, + right=right, + authority_resolver=resolve, + reference_table_builder=_rebuild_merged_table, + ) + if not decision.succeeded or decision.continuation is None: + raise RuntimeError("demo merge scenario did not produce a mergeable pair") + + row = extract_merge_preference_row( + left=left, right=right, decision=decision, authority_resolver=resolve, provenance_for=_provenance + ) + if row is None: + return [], [] + + left_pack, left_library = resolve(left.input_node) + pair = render_replay_preference_pair( + row, + resolve_node=merge_node_resolver(left, right, decision), + pack=left_pack, + library=left_library, + provenance_for=_provenance, + ) + return [row], ([pair] if pair is not None else []) + + +def _extract_and_render( + pack: DslPack, library: OperatorLibraryV1, trace: ConversationTraceV1 +) -> tuple[list[OperatorReplayPreferenceRowV1], list[PreferencePair]]: + report = extract_replay_preference_rows( + trace, pack=pack, library=library, provenance_for=_provenance + ) + pairs = render_replay_preference_pairs( + report.rows, + resolve_node=trace.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + return list(report.rows), pairs + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--out", type=Path, default=Path("outputs/data/preference/replay_demo_pairs.jsonl") + ) + args = parser.parse_args(argv) + + trace_rows, trace_pairs = _extract_and_render(*build_demo_trace()) + rollback_rows, rollback_pairs = _extract_and_render(*build_demo_partial_rollback_scenario()) + fork_rows, fork_pairs = _extract_and_render(*build_demo_fork_choose_scenario()) + merge_rows, merge_pairs = build_demo_merge_scenario() + + all_rows = trace_rows + rollback_rows + fork_rows + merge_rows + all_pairs = trace_pairs + rollback_pairs + fork_pairs + merge_pairs + counts_by_relation: dict[str, int] = {} + for row in all_rows: + key = row.semantic_relation.value + counts_by_relation[key] = counts_by_relation.get(key, 0) + 1 + + n = write_pairs(args.out, all_pairs) + print( + json.dumps( + { + "rows": len(all_rows), + "counts_by_relation": counts_by_relation, + "pairs_rendered": n, + "pairs_dropped": len(all_rows) - n, + "out": str(args.out), + "corpus_kind": "fixture_or_scratch", + }, + indent=2, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/slm_training/harnesses/experiments/argument_preference.py b/src/slm_training/harnesses/experiments/argument_preference.py new file mode 100644 index 000000000..978b4de16 --- /dev/null +++ b/src/slm_training/harnesses/experiments/argument_preference.py @@ -0,0 +1,208 @@ +"""SLM-418 (DSH5-10) ninth slice: argument-level preference for the typed policy. + +The disposition doc (``docs/design/dsh5-10-replay-preference-rows.md``) has +named ``typed_operator_policy.py``'s ``TypedOperatorPolicyScorer`` -- not the +generic TwoTower pair format the sixth/seventh/eighth slices used -- as the +issue's actual training target since the sixth slice. This module is the +first real step onto that target, deliberately scoped to what is honestly +representable there today. + +Only ``pronoun_focus_followup`` rows qualify. The other six named patterns' +``chosen_action``/``rejected_action`` are history controls (``undo``, +``redo:``, ``checkout:``) or ``merge:`` -- none of which has a +row in ``OperatorPolicyInputV1.action_rows`` (built only from +``legal_set.entries``, i.e. operator-registry actions; see +``build_operator_policy_input`` in +``slm_training.models.operator_policy_view``). ``pronoun_focus_followup`` is +the one pattern whose chosen and rejected actions are both the *same* +operator with different bound arguments for the *same* slot -- a genuine +argument-selection preference, not an action-selection or control-selection +one. Wiring the other six patterns into this head requires a real scope +decision about what "action row" would even mean for a control, which is +left open rather than guessed at here. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import dataclass + +import torch +import torch.nn.functional as F + +from slm_training.dsl.operators import ( + LegalSetCoverage, + OperatorLegalSetV1, + OperatorLibraryV1, + OperatorReplayPreferenceRowV1, + ReferenceTableV1, + ReplayPreferenceRelation, + deserialize_operator_action, +) +from slm_training.harnesses.experiments.typed_operator_policy import TypedOperatorPolicyScorer +from slm_training.models.operator_policy_view import ( + OperatorPolicyInputV1, + build_operator_policy_input, +) + +__all__ = [ + "TypedOperatorArgumentPreferenceExampleV1", + "build_argument_preference_example", + "train_typed_operator_argument_preference", + "typed_operator_argument_preference_loss", +] + + +@dataclass(frozen=True) +class TypedOperatorArgumentPreferenceExampleV1: + """One replay-grounded argument-selection preference in the sanitized policy-input space. + + ``chosen_reference_row``/``rejected_reference_row`` are row indices into + ``view.reference_rows`` -- the same opaque-free space + ``TypedOperatorPolicyScorer.forward`` scores over, never the runtime's + own ``OperatorRef`` identities. + """ + + row_id: str + view: OperatorPolicyInputV1 + action_row: int + slot_id: str + chosen_reference_row: int + rejected_reference_row: int + + def __post_init__(self) -> None: + if not self.row_id: + raise ValueError("row_id is required") + if not 0 <= self.action_row < len(self.view.action_rows): + raise ValueError("action_row is outside the policy view") + slots = { + slot.slot_id: slot for slot in self.view.action_rows[self.action_row].argument_slots + } + if self.slot_id not in slots: + raise ValueError("slot_id names an unknown action slot") + candidates = slots[self.slot_id].candidate_rows + if self.chosen_reference_row not in candidates or self.rejected_reference_row not in candidates: + raise ValueError("chosen/rejected reference row is outside the live slot domain") + if self.chosen_reference_row == self.rejected_reference_row: + raise ValueError("chosen and rejected reference rows must differ") + + +def build_argument_preference_example( + row: OperatorReplayPreferenceRowV1, + *, + reference_table: ReferenceTableV1, + legal_set: OperatorLegalSetV1, + library: OperatorLibraryV1, +) -> TypedOperatorArgumentPreferenceExampleV1 | None: + """Render one row into an argument-preference example, or ``None``. + + Returns ``None`` -- never fabricates a slot/row pairing -- whenever + ``row`` is not ``pronoun_focus_followup``, its chosen/rejected actions + are not the same operator with exactly one differing slot (never true by + construction for a well-formed row, but checked rather than assumed), or + either bound ref cannot be resolved against ``reference_table`` (would + indicate a stale/inconsistent row). + """ + if row.semantic_relation is not ReplayPreferenceRelation.PRONOUN_FOCUS_FOLLOWUP: + return None + chosen_operator_id, chosen_arguments = deserialize_operator_action(row.chosen_action) + rejected_operator_id, rejected_arguments = deserialize_operator_action(row.rejected_action) + if chosen_operator_id != rejected_operator_id: + return None + chosen_by_slot = {argument.slot_id: argument.value for argument in chosen_arguments} + rejected_by_slot = {argument.slot_id: argument.value for argument in rejected_arguments} + if set(chosen_by_slot) != set(rejected_by_slot): + return None + differing_slots = [ + slot_id + for slot_id, chosen_ref in chosen_by_slot.items() + if chosen_ref != rejected_by_slot[slot_id] + ] + if len(differing_slots) != 1: + return None + slot_id = differing_slots[0] + + view = build_operator_policy_input(reference_table, legal_set, library) + action_row = next( + (action.row for action in view.action_rows if action.operator_id == chosen_operator_id), None + ) + if action_row is None: + return None + + table_entries = (*reference_table.entries, *reference_table.selectors) + row_by_ref = { + (entry.ref.KIND, entry.ref.opaque_id): index for index, entry in enumerate(table_entries) + } + chosen_ref = chosen_by_slot[slot_id] + rejected_ref = rejected_by_slot[slot_id] + chosen_row = row_by_ref.get((chosen_ref.KIND, chosen_ref.opaque_id)) + rejected_row = row_by_ref.get((rejected_ref.KIND, rejected_ref.opaque_id)) + if chosen_row is None or rejected_row is None: + return None + + try: + return TypedOperatorArgumentPreferenceExampleV1( + row_id=row.input_state_id, + view=view, + action_row=action_row, + slot_id=slot_id, + chosen_reference_row=chosen_row, + rejected_reference_row=rejected_row, + ) + except ValueError: + return None + + +def typed_operator_argument_preference_loss( + scorer: TypedOperatorPolicyScorer, example: TypedOperatorArgumentPreferenceExampleV1 +) -> torch.Tensor: + """Bradley-Terry pairwise margin over ``CandidateScoringHead`` logits. + + Surrogate preference loss on argument-selection logits for one slot -- + not textbook DPO (mirrors the same honesty note + ``scripts/train_preference.py``'s own ``dpo_loss`` carries for the + generic TwoTower path). Defers PARTIAL/UNKNOWN-coverage rows exactly + like ``typed_operator_policy_loss`` does, via a zero-valued, + gradient-connected loss. + """ + if example.view.coverage is not LegalSetCoverage.COMPLETE: + return next(scorer.parameters()).sum() * 0.0 + _, argument_logits = scorer(example.view) + logits, candidate_rows = argument_logits[(example.action_row, example.slot_id)] + chosen_index = candidate_rows.index(example.chosen_reference_row) + rejected_index = candidate_rows.index(example.rejected_reference_row) + return -F.logsigmoid(logits[chosen_index] - logits[rejected_index]) + + +def train_typed_operator_argument_preference( + scorer: TypedOperatorPolicyScorer, + examples: Sequence[TypedOperatorArgumentPreferenceExampleV1], + *, + steps: int, + learning_rate: float, +) -> list[float]: + """Train only COMPLETE examples with one matched full-batch schedule. + + Mirrors ``train_typed_operator_policy``'s own schedule shape exactly, + over this module's pairwise loss instead of single-label cross-entropy. + """ + if steps <= 0: + raise ValueError("steps must be positive") + if learning_rate <= 0: + raise ValueError("learning_rate must be positive") + trainable = [ + example for example in examples if example.view.coverage is LegalSetCoverage.COMPLETE + ] + if not trainable: + raise ValueError("typed operator argument preference has no COMPLETE training rows") + optimizer = torch.optim.Adam(scorer.parameters(), lr=learning_rate) + history = [] + for _ in range(steps): + optimizer.zero_grad() + loss = torch.stack( + [typed_operator_argument_preference_loss(scorer, example) for example in trainable] + ).mean() + loss.backward() + optimizer.step() + history.append(float(loss.detach())) + return history diff --git a/src/slm_training/harnesses/preference/replay_pairs.py b/src/slm_training/harnesses/preference/replay_pairs.py new file mode 100644 index 000000000..d0e5102e0 --- /dev/null +++ b/src/slm_training/harnesses/preference/replay_pairs.py @@ -0,0 +1,200 @@ +"""SLM-418 (DSH5-10): render replay-grounded rows into trainable preference pairs. + +``extract_replay_preference_rows`` / ``extract_merge_preference_row`` +(``slm_training.dsl.operators.replay_preference``) already ground every +``chosen_action``/``rejected_action`` in the exact legal set available at +one real conversation state. This module is the first slice of the +remaining SLM-418 gap those functions themselves document: nothing in the +repo converts a row into the ``PreferencePair`` shape +``scripts/train_preference.py``'s ``build-pairs``/``train`` path consumes. +It does not run any training and makes no held-out claim -- see +``docs/design/dsh5-10-replay-preference-rows.md`` for the full disposition. + +Rendering never fabricates a state. ``chosen_output_state_id`` is always an +already-materialized state node by construction. The rejected side is +resolved the same way, using only information the caller's own node +resolver (or one real, pack-authorized ``OperatorLibraryV1.apply`` call) +already provides: + +* ``undo`` -> the input state's own parent (an existing node). +* ``redo:`` / ``checkout:`` -> that state_id is already + a materialized node -- no computation needed. +* an operator action's serialized form -> re-derived by recomputing the + exact legal set at the input state (``enumerate_operator_legal_set``, + the same call ``extract_replay_preference_rows`` itself makes) and + actually applying the matching legal action through the pack-authorized + ``library.apply`` -- the same executor every other application in this + module goes through, not a shortcut. +* ``merge:`` -> deliberately unrendered. It is never the rejected + side under the current single-pair extraction (only ever the *chosen* + side of a ``MERGE_SUCCESS`` row, whose state is always resolvable via + ``chosen_output_state_id`` like any other row), but is left honestly + unhandled here rather than guessed at if that ever changes. + +Two call shapes share this rendering, since replay-preference rows come +from two different sources with two different notions of "the trace": +``render_replay_preference_pair(row, resolve_node=trace.node, ...)`` for +rows from ``extract_replay_preference_rows`` (a real +``ConversationTraceV1``), and ``merge_node_resolver(left, right, +decision)`` to build the equivalent resolver for a ``MERGE_SUCCESS`` row +from ``extract_merge_preference_row``, whose ``left``/``right`` +``BranchEditV1`` edges and ``BranchMergeDecisionV1`` are never appended to +a shared trace. + +The pair's ``prompt`` is the input state's own source: this is a first, +documented modeling choice (score chosen/rejected *continuations* of the +current AST against it), not a validated training-objective decision -- +whoever wires an actual training run should treat it as open, especially +against the DSH3 policy head (``typed_operator_policy.py``) the disposition +doc's remaining-scope note names, not just the generic TwoTower pair +format used here for compatibility with existing tooling. +""" + +from __future__ import annotations + +from collections.abc import Callable + +from slm_training.dsl.operators.conversation import ConversationStateNodeV1 +from slm_training.dsl.operators.legal_set import enumerate_operator_legal_set +from slm_training.dsl.operators.merge import BranchEditV1, BranchMergeDecisionV1 +from slm_training.dsl.operators.registry import OperatorLibraryV1 +from slm_training.dsl.operators.replay_preference import ( + OperatorReplayPreferenceRowV1, + ProvenanceFactory, +) +from slm_training.dsl.pack import DslPack +from slm_training.harnesses.preference import PreferencePair + +NodeResolver = Callable[[str], ConversationStateNodeV1] + + +def merge_node_resolver( + left: BranchEditV1, right: BranchEditV1, decision: BranchMergeDecisionV1 +) -> NodeResolver: + """A ``NodeResolver`` over exactly the nodes one merge attempt exposes.""" + nodes: dict[str, ConversationStateNodeV1] = { + left.input_node.state_id: left.input_node, + left.output_node.state_id: left.output_node, + right.input_node.state_id: right.input_node, + right.output_node.state_id: right.output_node, + } + if decision.continuation is not None: + merged = decision.continuation.merged_node + nodes[merged.state_id] = merged + + def resolve(state_id: str) -> ConversationStateNodeV1: + try: + return nodes[state_id] + except KeyError as exc: + raise KeyError(f"unknown merge state ID {state_id!r}") from exc + + return resolve + + +def render_replay_preference_pair( + row: OperatorReplayPreferenceRowV1, + *, + resolve_node: NodeResolver, + pack: DslPack, + library: OperatorLibraryV1, + provenance_for: ProvenanceFactory, +) -> PreferencePair | None: + """Render one replay-grounded row into a ``PreferencePair``, or ``None``. + + Returns ``None`` rather than fabricating a rejected state whenever the + rejected action is ``merge:`` (out of scope; see module + docstring), its target legal action cannot be re-derived from the exact + legal set at ``row.input_state_id`` (would indicate a stale/inconsistent + row), or the re-applied rejected state is identical to the chosen state + (degenerate; not a real preference). + """ + input_node = resolve_node(row.input_state_id) + chosen_text = resolve_node(row.chosen_output_state_id).state.source + + action = row.rejected_action + if action == "undo": + if input_node.parent_state_id is None: + return None + rejected_text = resolve_node(input_node.parent_state_id).state.source + elif action.startswith("redo:") or action.startswith("checkout:"): + target_state_id = action.split(":", 1)[1] + rejected_text = resolve_node(target_state_id).state.source + elif action.startswith("merge:"): + return None + else: + legal_set = enumerate_operator_legal_set( + pack=pack, + library=library, + state=input_node.state, + reference_table=input_node.reference_table, + provenance=provenance_for(input_node.state), + ordinary_nonoperator_actions=(), + ) + match = next( + ( + candidate + for candidate in legal_set.operator_actions + if candidate.serialized == action + ), + None, + ) + if match is None: + return None + result = library.apply( + pack, + input_node.state, + match.operator_id, + match.arguments, + provenance_for(input_node.state), + ) + if not result.succeeded or result.state is None: + return None + rejected_text = result.state.source + + if rejected_text == chosen_text: + return None + + return PreferencePair( + prompt=input_node.state.source, + chosen=chosen_text, + rejected=rejected_text, + meta={ + "pair_corpus": "replay_preference", + "schema": "replay_preference_pair/v1", + "semantic_relation": row.semantic_relation.value, + "correction_reason": row.correction_reason, + "input_state_id": row.input_state_id, + "chosen_action": row.chosen_action, + "rejected_action": row.rejected_action, + "legal_set_fingerprint": row.legal_set_fingerprint, + }, + ) + + +def render_replay_preference_pairs( + rows: tuple[OperatorReplayPreferenceRowV1, ...], + *, + resolve_node: NodeResolver, + pack: DslPack, + library: OperatorLibraryV1, + provenance_for: ProvenanceFactory, +) -> list[PreferencePair]: + """Render every row that can honestly be rendered; silently drops the rest. + + "Silently" here means no exception -- callers that need to know how + many rows were dropped (and why) should call + ``render_replay_preference_pair`` per-row themselves, since this + function intentionally mirrors the shape ``write_pairs`` expects. + """ + pairs: list[PreferencePair] = [] + for row in rows: + pair = render_replay_preference_pair( + row, + resolve_node=resolve_node, + pack=pack, + library=library, + provenance_for=provenance_for, + ) + if pair is not None: + pairs.append(pair) + return pairs diff --git a/src/slm_training/resources/versions.json b/src/slm_training/resources/versions.json index ff2383431..c8226fae6 100644 --- a/src/slm_training/resources/versions.json +++ b/src/slm_training/resources/versions.json @@ -7079,6 +7079,21 @@ } ] }, + "harness.experiments.argument_preference": { + "version": "v1", + "kind": "harness", + "paths": [ + "src/slm_training/harnesses/experiments/argument_preference.py", + "tests/test_harnesses/experiments/test_argument_preference.py" + ], + "history": [ + { + "version": "v1", + "date": "2026-07-27", + "note": "SLM-418 (DSH5-10) ninth slice: first real step onto the DSH3 typed_operator_policy.py target the disposition doc has named as the actual training target since the sixth slice, instead of the generic TwoTower pair format the sixth/seventh/eighth slices used. build_argument_preference_example renders only pronoun_focus_followup rows (the one pattern whose chosen/rejected actions are the same operator with a differing argument for the same slot -- the other six patterns are history controls or merge, none representable in OperatorPolicyInputV1.action_rows). typed_operator_argument_preference_loss is a Bradley-Terry pairwise margin over CandidateScoringHead logits (surrogate, not textbook DPO, same honesty class as scripts/train_preference.py's dpo_loss). Honest finding: the real pronoun-focus fixture's two sibling refs are feature-identical in the sanitized OperatorPolicyInputV1 view (differ only by semantic_fingerprint, which is deliberately stripped for anti-identity-leakage), so training provably cannot move the loss for that exact case -- a structural null result, not a bug, proven distinct from a synthetic fixture where candidates differ in an allowed feature (relative_position) and training does reduce the loss. No harness code outside this new module changed; typed_operator_policy.py itself is unchanged (only consumed)" + } + ] + }, "harness.experiments.typed_operator_policy": { "version": "v11", "kind": "harness", @@ -7941,6 +7956,38 @@ } ] }, + "harness.preference.replay_pairs": { + "version": "v4", + "kind": "harness", + "paths": [ + "src/slm_training/harnesses/preference/replay_pairs.py", + "tests/test_harnesses/preference/test_replay_pairs.py", + "scripts/build_replay_preference_pairs.py", + "tests/test_scripts/test_build_replay_preference_pairs.py" + ], + "history": [ + { + "version": "v4", + "date": "2026-07-27", + "note": "SLM-418 (DSH5-10) tenth slice: adds build_demo_partial_rollback_scenario and build_demo_fork_choose_scenario, two more standalone scratch traces mirroring test_replay_preference.py's own partial_rollback and fork_then_choose_one_branch fixtures, using the script's existing toy cycling operator (refactored the trace-building preamble into shared _new_toy_trace/_apply_cycle helpers rather than duplicating it a third time). main() now combines four sources: 7 rows (2 edit_then_undo + 1 each of undo_then_redo/checkout_another_state/partial_rollback/fork_then_choose_one_branch/merge_success), 6 render (still only undo_then_redo drops, same structural reason as before). 6 of 7 named patterns are now exercised in this demo TwoTower corpus; only pronoun_focus_followup remains absent here (covered instead on the separate typed_operator_policy argument-preference path added in the ninth slice). No change to the sixth slice's renderer itself" + }, + { + "version": "v3", + "date": "2026-07-27", + "note": "SLM-418 (DSH5-10) eighth slice: adds build_demo_merge_scenario, a second scratch fixture (two branches with disjoint-target edits, mirroring tests/test_dsl/test_operator_merge.py's own verified-mergeable shape) covering merge_success -- the one named pattern the original build_demo_trace could not reach, since extract_merge_preference_row never operates on a shared ConversationTraceV1. main() now combines both sources: 4 rows (3 from the trace + 1 merge), 3 render (still only undo_then_redo drops, for the structural reason documented in the seventh slice), 1 combined pairs.jsonl. No change to the sixth slice's renderer itself" + }, + { + "version": "v2", + "date": "2026-07-27", + "note": "SLM-418 (DSH5-10) seventh slice: scripts/build_replay_preference_pairs.py builds a single small, deterministic, honestly-labeled fixture_or_scratch conversation (no real captured ConversationTraceV1 corpus exists anywhere in the repo -- confirmed this slice) exercising three of the seven named patterns, extracts rows, renders them via render_replay_preference_pairs, and writes a real pairs.jsonl. 2 of 3 rows render (undo_then_redo never does -- for any deterministic zero-argument operator, redo and reapplying the same operator at the same input state are, by definition, textually identical, so the renderer's dedup guard correctly declines it rather than emitting a self-contradictory pair). First real, on-disk pairs corpus this feature line has ever produced; still wiring/demo, no training run in this component" + }, + { + "version": "v1", + "date": "2026-07-27", + "note": "initial registration; SLM-418 (DSH5-10) sixth slice: render OperatorReplayPreferenceRowV1 rows (both extract_replay_preference_rows turn-scan rows and extract_merge_preference_row's standalone MERGE_SUCCESS rows) into PreferencePair, the shape scripts/train_preference.py's build-pairs/train path consumes. No prior converter existed. Wiring only -- no training run, no held-out claim; see docs/design/dsh5-10-replay-preference-rows.md's sixth-slice note for the honesty scope and the still-open prompt-shape modeling question" + } + ] + }, "harness.preference.remine_campaign": { "version": "v2", "kind": "harness", diff --git a/tests/test_harnesses/experiments/test_argument_preference.py b/tests/test_harnesses/experiments/test_argument_preference.py new file mode 100644 index 000000000..06b328bc1 --- /dev/null +++ b/tests/test_harnesses/experiments/test_argument_preference.py @@ -0,0 +1,250 @@ +"""SLM-418 (DSH5-10) ninth slice: argument-level preference over the typed policy head.""" + +from __future__ import annotations + +import math + +import pytest + +from slm_training.dsl.operators import ( + BindingPhase, + BoundArgumentV1, + LegalSetCoverage, + OperatorSupportVerdict, + RefKind, + ReplayPreferenceRelation, + enumerate_operator_legal_set, + extract_replay_preference_rows, + serialize_operator_action, + undo_conversation, +) +from slm_training.harnesses.experiments.argument_preference import ( + TypedOperatorArgumentPreferenceExampleV1, + build_argument_preference_example, + train_typed_operator_argument_preference, + typed_operator_argument_preference_loss, +) +from slm_training.harnesses.experiments.typed_operator_policy import TypedOperatorPolicyScorer +from slm_training.models.operator_policy_view import ( + OperatorActionViewV1, + OperatorArgumentSlotViewV1, + OperatorPolicyInputV1, + ReferenceModelViewV1, +) +from tests.test_dsl.test_operator_conversation import _append, _fixture, _provenance +from tests.test_dsl.test_replay_preference import ( + _PRONOUN_OPERATOR_ID, + _append_pronoun, + _pronoun_focus_fixture, +) + + +def _legal_set_and_table(pack, library, trace, state_id): + node = trace.node(state_id) + legal_set = enumerate_operator_legal_set( + pack=pack, + library=library, + state=node.state, + reference_table=node.reference_table, + provenance=_provenance(node.state), + ordinary_nonoperator_actions=(), + ) + return node.reference_table, legal_set + + +def _pronoun_focus_row_and_context(): + pack, library, trace, table_for, refs_by_index = _pronoun_focus_fixture() + edited_once = _append_pronoun(pack, library, trace, refs_by_index[0], table_for=table_for) + edited_twice = _append_pronoun( + pack, library, edited_once, refs_by_index[0], table_for=table_for + ) + report = extract_replay_preference_rows( + edited_twice, pack=pack, library=library, provenance_for=_provenance + ) + row = report.rows[0] + assert row.semantic_relation is ReplayPreferenceRelation.PRONOUN_FOCUS_FOLLOWUP + reference_table, legal_set = _legal_set_and_table( + pack, library, edited_twice, row.input_state_id + ) + return row, pack, library, reference_table, legal_set, refs_by_index + + +def test_build_argument_preference_example_from_a_real_pronoun_focus_row() -> None: + row, pack, library, reference_table, legal_set, refs_by_index = _pronoun_focus_row_and_context() + + example = build_argument_preference_example( + row, reference_table=reference_table, legal_set=legal_set, library=library + ) + + assert example is not None + assert example.row_id == row.input_state_id + assert example.slot_id == "value" + action = example.view.action_rows[example.action_row] + assert action.operator_id == _PRONOUN_OPERATOR_ID + candidates = next( + slot.candidate_rows for slot in action.argument_slots if slot.slot_id == "value" + ) + assert example.chosen_reference_row in candidates + assert example.rejected_reference_row in candidates + assert example.chosen_reference_row != example.rejected_reference_row + + +def test_build_argument_preference_example_declines_non_pronoun_focus_relations() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + report = extract_replay_preference_rows( + undone, pack=pack, library=library, provenance_for=_provenance + ) + row = report.rows[0] + assert row.semantic_relation is ReplayPreferenceRelation.EDIT_THEN_UNDO + reference_table, legal_set = _legal_set_and_table(pack, library, undone, row.input_state_id) + + example = build_argument_preference_example( + row, reference_table=reference_table, legal_set=legal_set, library=library + ) + + assert example is None + + +def test_build_argument_preference_example_declines_a_row_with_no_differing_slot() -> None: + """Defensive branch: a hand-built row whose chosen/rejected bind the same ref renders None.""" + from dataclasses import replace as dataclass_replace + + row, pack, library, reference_table, legal_set, refs_by_index = _pronoun_focus_row_and_context() + same_action = serialize_operator_action( + _PRONOUN_OPERATOR_ID, (BoundArgumentV1("value", refs_by_index[0]),) + ) + degenerate_row = dataclass_replace(row, chosen_action=same_action, rejected_action=same_action) + + example = build_argument_preference_example( + degenerate_row, reference_table=reference_table, legal_set=legal_set, library=library + ) + + assert example is None + + +def test_loss_is_finite_and_gradient_connected() -> None: + row, pack, library, reference_table, legal_set, refs_by_index = _pronoun_focus_row_and_context() + example = build_argument_preference_example( + row, reference_table=reference_table, legal_set=legal_set, library=library + ) + assert example is not None + scorer = TypedOperatorPolicyScorer.from_examples((example,), dim=8) + + loss = typed_operator_argument_preference_loss(scorer, example) + + assert loss.requires_grad + assert loss.isfinite() + + +def test_training_cannot_move_the_loss_when_candidates_are_feature_identical() -> None: + """Honest null result, not a bug: this fixture's two sibling refs are + provably indistinguishable to the scorer. + + ``ReferenceModelViewV1`` deliberately strips ``semantic_fingerprint`` + (the only thing distinguishing ``_PRONOUN_TARGETS[0]`` from ``[1]`` -- + see ``FORBIDDEN_FIELD_NAMES`` in + ``slm_training.models.operator_policy_view``, anti-identity-leakage by + design) and ``OperatorFeatureEncoder._reference_embeddings`` is a pure + function of ``ref_kind``/``value_type``/``compiler_facts``/``has_parent``/ + ``relative_position`` only -- no row-index feature. Both candidate rows + here share every one of those (same ``VALUE`` kind, same + ``openui.string`` type, no parent, no position), so they get + byte-identical embeddings through the shared-weight + ``CandidateScoringHead`` regardless of any parameter update: the loss is + stuck at ``-log_sigmoid(0) = ln(2)`` structurally, not from + under-training. See ``test_training_reduces_the_pairwise_loss_when_candidates_differ`` + below for proof the loss function and gradient flow work correctly when + the signal *is* representable. + """ + row, pack, library, reference_table, legal_set, refs_by_index = _pronoun_focus_row_and_context() + example = build_argument_preference_example( + row, reference_table=reference_table, legal_set=legal_set, library=library + ) + assert example is not None + scorer = TypedOperatorPolicyScorer.from_examples((example,), dim=8) + + before = float(typed_operator_argument_preference_loss(scorer, example).detach()) + history = train_typed_operator_argument_preference( + scorer, (example,), steps=20, learning_rate=0.05 + ) + after = float(typed_operator_argument_preference_loss(scorer, example).detach()) + + assert len(history) == 20 + assert after == pytest.approx(before, abs=1e-6) + assert before == pytest.approx(math.log(2.0), abs=1e-4) + + +def test_training_reduces_the_pairwise_loss_when_candidates_differ() -> None: + """When the two candidates differ in an allowed feature (here: + ``relative_position``), the signal is representable and training + actually reduces the loss -- proving the mechanism itself works; the + null result above is specific to feature-identical candidates, not a + defect in the loss or the training loop. + """ + references = ( + ReferenceModelViewV1( + row=0, + ref_kind=RefKind.INDEX, + value_type="openui.string", + compiler_facts=(), + has_parent=False, + parent_row=None, + relative_position=0, + ), + ReferenceModelViewV1( + row=1, + ref_kind=RefKind.INDEX, + value_type="openui.string", + compiler_facts=(), + has_parent=False, + parent_row=None, + relative_position=1, + ), + ) + action = OperatorActionViewV1( + row=0, + operator_id="openui.fixture_argpref", + operator_version="v1", + locality="node", + cost=1.0, + effect_signature=(), + argument_slots=( + OperatorArgumentSlotViewV1( + slot_id="value", + ref_kind=RefKind.INDEX, + binding_phase=BindingPhase.APPLICATION, + required=True, + repeated=False, + candidate_rows=(0, 1), + domain_complete=True, + ), + ), + verdict=OperatorSupportVerdict.SUPPORTED, + coverage=LegalSetCoverage.COMPLETE, + ) + view = OperatorPolicyInputV1( + reference_rows=references, + action_rows=(action,), + ordinary_action_count=0, + coverage=LegalSetCoverage.COMPLETE, + ) + example = TypedOperatorArgumentPreferenceExampleV1( + row_id="synthetic-distinguishable", + view=view, + action_row=0, + slot_id="value", + chosen_reference_row=0, + rejected_reference_row=1, + ) + scorer = TypedOperatorPolicyScorer.from_examples((example,), dim=8) + + before = float(typed_operator_argument_preference_loss(scorer, example).detach()) + history = train_typed_operator_argument_preference( + scorer, (example,), steps=30, learning_rate=0.1 + ) + after = float(typed_operator_argument_preference_loss(scorer, example).detach()) + + assert len(history) == 30 + assert after < before diff --git a/tests/test_harnesses/preference/test_replay_pairs.py b/tests/test_harnesses/preference/test_replay_pairs.py new file mode 100644 index 000000000..0396727b1 --- /dev/null +++ b/tests/test_harnesses/preference/test_replay_pairs.py @@ -0,0 +1,306 @@ +"""Tests for SLM-418 (DSH5-10) replay-preference-row -> PreferencePair rendering.""" + +from __future__ import annotations + +from slm_training.dsl.operators import ( + BoundArgumentV1, + OperatorReplayPreferenceRowV1, + ReplayPreferenceRelation, + checkout_conversation_state, + extract_merge_preference_row, + extract_replay_preference_rows, + merge_conversation_branches, + redo_conversation, + serialize_operator_action, + undo_conversation, +) +from slm_training.harnesses.preference import PreferencePair +from slm_training.harnesses.preference.replay_pairs import ( + merge_node_resolver, + render_replay_preference_pair, + render_replay_preference_pairs, +) +from tests.test_dsl.test_operator_conversation import _append, _fixture, _provenance +from tests.test_dsl.test_operator_merge import _Fixture as _MergeFixture +from tests.test_dsl.test_operator_merge import _provenance as _merge_provenance +from tests.test_dsl.test_replay_preference import ( + _PRONOUN_OPERATOR_ID, + _append_pronoun, + _pronoun_focus_fixture, +) + + +def test_edit_then_undo_row_renders_a_pair_matching_the_trace_states() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + + report = extract_replay_preference_rows( + undone, pack=pack, library=library, provenance_for=_provenance + ) + row = report.rows[0] + assert row.semantic_relation is ReplayPreferenceRelation.EDIT_THEN_UNDO + + pair = render_replay_preference_pair( + row, + resolve_node=undone.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + assert isinstance(pair, PreferencePair) + assert pair.prompt == undone.node(row.input_state_id).state.source + assert pair.chosen == undone.node(row.chosen_output_state_id).state.source + # "undo" chosen -> the rejected side must be some other legal candidate's + # own resulting text, never the same text as the chosen undo target. + assert pair.chosen == root.node(root.root_state_id).state.source + assert pair.rejected != pair.chosen + assert pair.meta["pair_corpus"] == "replay_preference" + assert pair.meta["semantic_relation"] == "edit_then_undo" + assert pair.meta["chosen_action"] == "undo" + assert pair.meta["rejected_action"] == row.rejected_action + + +def test_undo_then_redo_row_declines_a_degenerate_collapse() -> None: + """The fixture's rejected alternative here happens to reproduce ``chosen`` verbatim. + + ``_fixture()``'s operator is deterministic and zero-argument, so at the + root state its only candidate (the alphabetically-first rejected + action, since ``"OPERATOR ..."`` sorts before ``"checkout:..."``/ + ``"undo"``) reapplies the exact same transition ``redo`` already + reached -- title -> body either way. Rendering that as a preference + pair would tell the model the identical text is simultaneously + preferred and rejected, so the renderer's dedup guard must decline it + rather than emit a self-contradictory pair. + """ + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + original_child_id = edited.current_state_id + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + redone = redo_conversation( + undone, + target_state_id=original_child_id, + provenance=_provenance(undone.current.state), + ) + + report = extract_replay_preference_rows( + redone, pack=pack, library=library, provenance_for=_provenance + ) + row = next( + r + for r in report.rows + if r.semantic_relation is ReplayPreferenceRelation.UNDO_THEN_REDO + ) + + pair = render_replay_preference_pair( + row, + resolve_node=redone.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + assert pair is None + + +def test_checkout_another_state_row_renders_via_the_operator_reapplication_branch() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + checked_out = checkout_conversation_state( + edited, + target_state_id=root.root_state_id, + provenance=_provenance(edited.current.state), + ) + + report = extract_replay_preference_rows( + checked_out, pack=pack, library=library, provenance_for=_provenance + ) + row = next( + r + for r in report.rows + if r.semantic_relation is ReplayPreferenceRelation.CHECKOUT_ANOTHER_STATE + ) + + pair = render_replay_preference_pair( + row, + resolve_node=checked_out.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + assert pair is not None + assert pair.chosen == checked_out.node(row.chosen_output_state_id).state.source + assert pair.rejected != pair.chosen + + +def test_rejected_undo_action_resolves_to_the_input_states_own_parent() -> None: + """Directly exercises the ``"undo"``-rejected branch (a control lookup, no apply). + + Constructed directly rather than relying on the fixture's own + alphabetical tie-break (its ``"OPERATOR ..."`` candidate always sorts + first), so this proves the parent-lookup path itself, independent of + which branch a real extracted row happens to land on. root -> edit1 -> + edit2, so ``edit2``'s parent (edit1, "body") is textually distinct from + the row's chosen destination (root, "title"). + """ + pack, library, root = _fixture() + edited_once, _application1 = _append(pack, library, root) + edited_twice, _application2 = _append(pack, library, edited_once, seed=3) + + contrived = OperatorReplayPreferenceRowV1( + input_state_id=edited_twice.current_state_id, + chosen_action=f"checkout:{root.root_state_id}", + rejected_action="undo", + chosen_output_state_id=root.root_state_id, + semantic_relation=ReplayPreferenceRelation.CHECKOUT_ANOTHER_STATE, + correction_reason="user_checked_out_alternate_state", + legal_set_fingerprint="0" * 64, + ) + + pair = render_replay_preference_pair( + contrived, + resolve_node=edited_twice.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + assert pair is not None + assert pair.chosen == root.node(root.root_state_id).state.source + assert pair.rejected == edited_once.node(edited_once.current_state_id).state.source + assert pair.rejected != pair.chosen + + +def test_pronoun_focus_followup_row_renders_rejected_by_reapplying_the_sibling() -> None: + pack, library, trace, table_for, refs_by_index = _pronoun_focus_fixture() + edited_once = _append_pronoun(pack, library, trace, refs_by_index[0], table_for=table_for) + edited_twice = _append_pronoun( + pack, library, edited_once, refs_by_index[0], table_for=table_for + ) + + report = extract_replay_preference_rows( + edited_twice, pack=pack, library=library, provenance_for=_provenance + ) + row = report.rows[0] + assert row.semantic_relation is ReplayPreferenceRelation.PRONOUN_FOCUS_FOLLOWUP + assert row.rejected_action == serialize_operator_action( + _PRONOUN_OPERATOR_ID, (BoundArgumentV1("value", refs_by_index[1]),) + ) + + pair = render_replay_preference_pair( + row, + resolve_node=edited_twice.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + assert pair is not None + assert pair.chosen == edited_twice.node(row.chosen_output_state_id).state.source + # The rejected side was never applied to any trace node -- it is only + # derivable by re-running the sibling legal action through the same + # pack-authorized executor, which is exactly what the renderer does. + replayed_sibling = library.apply( + pack, + edited_once.current.state, + _PRONOUN_OPERATOR_ID, + (BoundArgumentV1("value", refs_by_index[1]),), + _provenance(edited_once.current.state), + ) + assert replayed_sibling.succeeded and replayed_sibling.state is not None + assert pair.rejected == replayed_sibling.state.source + assert pair.rejected != pair.chosen + + +def test_merge_success_row_renders_via_merge_node_resolver() -> None: + fixture = _MergeFixture() + left = fixture.branch(name="left", target_name="title", replacement=":hero.heading") + right = fixture.branch(name="right", target_name="body", replacement=":hero.copy") + decision = merge_conversation_branches( + pack=fixture.pack, + base=fixture.base, + left=left, + right=right, + authority_resolver=fixture.resolve, + reference_table_builder=fixture.rebuild_merged_table, + ) + assert decision.succeeded + + row = extract_merge_preference_row( + left=left, + right=right, + decision=decision, + authority_resolver=fixture.resolve, + provenance_for=_merge_provenance, + ) + assert row is not None + assert row.semantic_relation is ReplayPreferenceRelation.MERGE_SUCCESS + + pack, library = fixture.resolve(left.input_node) + pair = render_replay_preference_pair( + row, + resolve_node=merge_node_resolver(left, right, decision), + pack=pack, + library=library, + provenance_for=_merge_provenance, + ) + + assert pair is not None + assert pair.prompt == left.output_node.state.source + assert pair.chosen == decision.continuation.merged_node.state.source + assert pair.rejected != pair.chosen + assert pair.meta["semantic_relation"] == "merge_success" + + +def test_rejected_merge_action_is_never_rendered() -> None: + """Defensive branch: a hypothetical rejected ``merge:`` renders ``None``. + + Not reachable through today's single-merge-candidate extraction (see + module docstring), but the renderer must never guess a merged state + instead of honestly declining. + """ + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + + contrived = OperatorReplayPreferenceRowV1( + input_state_id=edited.current_state_id, + chosen_action="undo", + rejected_action=f"merge:{edited.current_state_id},{root.root_state_id}", + chosen_output_state_id=root.root_state_id, + semantic_relation=ReplayPreferenceRelation.EDIT_THEN_UNDO, + correction_reason="user_undid_without_redo", + legal_set_fingerprint="0" * 64, + ) + + pair = render_replay_preference_pair( + contrived, + resolve_node=edited.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + assert pair is None + + +def test_render_replay_preference_pairs_batches_a_full_report() -> None: + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + + report = extract_replay_preference_rows( + undone, pack=pack, library=library, provenance_for=_provenance + ) + + pairs = render_replay_preference_pairs( + report.rows, + resolve_node=undone.node, + pack=pack, + library=library, + provenance_for=_provenance, + ) + + assert len(pairs) == len(report.rows) + assert all(isinstance(pair, PreferencePair) for pair in pairs) diff --git a/tests/test_scripts/test_build_replay_preference_pairs.py b/tests/test_scripts/test_build_replay_preference_pairs.py new file mode 100644 index 000000000..5557bc942 --- /dev/null +++ b/tests/test_scripts/test_build_replay_preference_pairs.py @@ -0,0 +1,109 @@ +"""SLM-418 (DSH5-10): the demo replay-preference-pairs corpus builder script.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from scripts.build_replay_preference_pairs import ( + _provenance, + build_demo_fork_choose_scenario, + build_demo_merge_scenario, + build_demo_partial_rollback_scenario, + build_demo_trace, + main, +) +from slm_training.dsl.operators import ReplayPreferenceRelation, extract_replay_preference_rows +from slm_training.harnesses.preference import load_pairs + + +def test_build_demo_trace_exercises_three_named_patterns() -> None: + pack, library, trace = build_demo_trace() + + report = extract_replay_preference_rows( + trace, pack=pack, library=library, provenance_for=_provenance + ) + assert set(report.counts_by_relation) == { + ReplayPreferenceRelation.EDIT_THEN_UNDO.value, + ReplayPreferenceRelation.UNDO_THEN_REDO.value, + ReplayPreferenceRelation.CHECKOUT_ANOTHER_STATE.value, + } + + +def test_build_demo_partial_rollback_scenario_yields_the_second_consecutive_undo() -> None: + pack, library, trace = build_demo_partial_rollback_scenario() + + report = extract_replay_preference_rows( + trace, pack=pack, library=library, provenance_for=_provenance + ) + # Both rows this trace produces are kept, honestly -- the first undo is + # its own edit_then_undo row, the second (no intervening edit) is the + # new partial_rollback relation this scenario targets. + assert report.counts_by_relation == { + ReplayPreferenceRelation.EDIT_THEN_UNDO.value: 1, + ReplayPreferenceRelation.PARTIAL_ROLLBACK.value: 1, + } + + +def test_build_demo_fork_choose_scenario_yields_one_fork_then_choose_row() -> None: + pack, library, trace = build_demo_fork_choose_scenario() + + report = extract_replay_preference_rows( + trace, pack=pack, library=library, provenance_for=_provenance + ) + assert report.counts_by_relation == { + ReplayPreferenceRelation.FORK_THEN_CHOOSE_ONE_BRANCH.value: 1, + } + + +def test_build_demo_merge_scenario_yields_one_real_merge_success_pair() -> None: + rows, pairs = build_demo_merge_scenario() + + assert len(rows) == 1 + assert rows[0].semantic_relation is ReplayPreferenceRelation.MERGE_SUCCESS + assert len(pairs) == 1 + pair = pairs[0] + assert pair.meta["pair_corpus"] == "replay_preference" + assert pair.meta["semantic_relation"] == "merge_success" + assert pair.chosen != pair.rejected + + +def test_main_writes_a_real_pairs_file_and_reports_honest_counts( + tmp_path: Path, capsys +) -> None: + out_path = tmp_path / "replay_demo_pairs.jsonl" + + exit_code = main(["--out", str(out_path)]) + + assert exit_code == 0 + report = json.loads(capsys.readouterr().out) + assert report["corpus_kind"] == "fixture_or_scratch" + assert report["rows"] == 7 + assert report["counts_by_relation"] == { + "edit_then_undo": 2, + "undo_then_redo": 1, + "checkout_another_state": 1, + "partial_rollback": 1, + "fork_then_choose_one_branch": 1, + "merge_success": 1, + } + # undo_then_redo never renders here: redo and "reapply the same + # deterministic zero-arg operator" are the identical text by + # construction, so the renderer's dedup guard correctly declines it. + assert report["pairs_rendered"] == 6 + assert report["pairs_dropped"] == 1 + + pairs = load_pairs(out_path) + assert len(pairs) == report["pairs_rendered"] + relations = {pair.meta["semantic_relation"] for pair in pairs} + assert relations == { + "edit_then_undo", + "checkout_another_state", + "partial_rollback", + "fork_then_choose_one_branch", + "merge_success", + } + for pair in pairs: + assert pair.meta["pair_corpus"] == "replay_preference" + assert pair.chosen != pair.rejected + assert pair.prompt