From ec7c63cfeb3c9426971606293ddc201d084e61f6 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 28 Jul 2026 15:08:57 +0000 Subject: [PATCH] docs+harness(preference): SLM-418 tenth slice -- same-domain-preferring rejection policy, honest lexicographic-artifact finding _pick_rejected (dsl/operators/replay_preference.py) now prefers a same-domain alternative (operator-vs-operator or control-vs-control) when the legal set offers one, falling back to the old any-domain lexicographic-first candidate only when it doesn't. This fixes the ninth slice's own finding that a serialized operator token always sorted first, so same_domain_pair_count was always 0. Real effect: same_domain_pair_count moves from 0 to 46 over the real 56-row synthetic corpus. A new pairwise scorer (train_history_control_pairwise_scorer, harness.preference. replay_preference_history_control_policy v2) trains on the new pairs and shows real positive held-out discrimination (accuracy=1.0, mean_margin>0) -- but rejected_kind_counts shows the rejected side is "checkout" for all 46 pairs, so this is honestly reported as a second lexicographic-tiebreak artifact (checkout: still sorts first among control prefixes whenever legal), not a hard preference signal. Verdict field says so directly rather than reporting a bare positive number. action_kind_of relocates from replay_preference_context_views.py to replay_preference.py (re-exported unchanged) so the new domain check reuses the same closed vocabulary instead of duplicating it. _pick_rejected has no external callers (grepped and confirmed); the four ninth-slice tests that encoded the old zero-pairing numbers are updated to the new real numbers, and new regression tests cover both the same-domain-preferred path and the any-domain fallback directly. Full relevant sweep: 143 passed. ruff/repo_policy/verify_version_stamps/ verify_decode_invariants all clean. dsl.operators.replay_preference bumped v8 -> v9; harness.preference. replay_preference_history_control_policy bumped v1 -> v2. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0196EiM7jnjjULXMDQpvQoDN --- docs/design/dsh5-10-replay-preference-rows.md | 206 +++++++++++- ...replay_preference_history_control_probe.py | 27 +- .../dsl/operators/replay_preference.py | 69 +++- .../replay_preference_context_views.py | 25 +- ...eplay_preference_history_control_policy.py | 301 ++++++++++++++++++ src/slm_training/resources/versions.json | 24 +- tests/test_dsl/test_replay_preference.py | 114 +++++++ ...eplay_preference_history_control_policy.py | 136 ++++++-- 8 files changed, 846 insertions(+), 56 deletions(-) diff --git a/docs/design/dsh5-10-replay-preference-rows.md b/docs/design/dsh5-10-replay-preference-rows.md index 2da4a117d..e4fbae610 100644 --- a/docs/design/dsh5-10-replay-preference-rows.md +++ b/docs/design/dsh5-10-replay-preference-rows.md @@ -1,6 +1,6 @@ # DSH5-10: replay-grounded preference rows from undo/redo history (SLM-418) -**Status:** partial slice, in progress (ninth increment). +**Status:** partial slice, in progress (tenth increment). **Claim class:** `wiring`. **Honest verdict:** not yet dispositioned -- this PR extends a scoped subset, not the full issue. @@ -1104,6 +1104,210 @@ real discrimination. This is a row-extraction-policy change (bumping `dsl.operators.replay_preference`), not a model-input-contract change, and is a real, separately-scoped tenth slice, not a partial version of this one. +## Tenth slice (2026-07-28) + +The ninth slice's own named next lever: change `_pick_rejected` +(`replay_preference.py`) to prefer a same-domain alternative when the legal +set actually offers one, falling back to the old any-domain choice only when +it doesn't, then re-run `evaluate_history_control_representability` to see +whether `same_domain_pair_count` moves off zero, and if so, train a pairwise +scorer over `HistoryControlPolicyInputV1.control_rows` and report honestly. + +### What was built + +* `src/slm_training/dsl/operators/replay_preference.py`: + * `_pick_rejected` now computes a coarse two-way domain + (`_rejection_domain_of`: "operator" or "control", the same split + `classify_replay_row_history_control_representability` already uses) for + `chosen` and prefers the lexicographically-first candidate whose domain + matches, falling back to the lexicographically-first candidate of *any* + domain -- this function's original v2-v9 behavior -- only when no + same-domain alternative exists. Every call site + (`EDIT_THEN_UNDO`/`PARTIAL_ROLLBACK`/`UNDO_THEN_REDO`/ + `CHECKOUT_ANOTHER_STATE`/`FORK_THEN_CHOOSE_ONE_BRANCH`/`MERGE_SUCCESS`) + is unchanged; only the candidate-selection policy inside `_pick_rejected` + itself changed. `PRONOUN_FOCUS_FOLLOWUP` never calls `_pick_rejected` (it + has its own sibling-candidate logic, already same-domain by + construction) and is untouched. + * `action_kind_of` moves here from `replay_preference_context_views.py` + (which now re-exports it unchanged, `# noqa: F401`, for every existing + caller) so `_pick_rejected`'s new domain check reuses the exact same + closed vocabulary every other domain-classification caller in this + corpus already depends on, instead of duplicating the prefix logic a + third time. This was necessary because `replay_preference_context_views.py` + already imports from `replay_preference.py` -- importing the other way + would be circular. + * `dsl.operators.replay_preference` bumped v8 -> v9; + `dsl.operators.replay_preference_context_views` gets a `no-bump:` note + (re-export relocation only, byte-for-byte identical behavior/API). +* `src/slm_training/harnesses/preference/replay_preference_history_control_policy.py` + (`harness.preference.replay_preference_history_control_policy` v1 -> v2): + new `train_history_control_pairwise_scorer` -- a small pairwise linear + scorer over `HistoryControlPolicyInputV1.control_rows`, same discipline as + the sixth slice's two-feature scorer and the eighth slice's argument-margin + probe: full-batch gradient descent on pairwise logistic loss, one-hot + `control_kind` features (the *only* structural field + `HistoryControlActionViewV1` carries), no bias, no external ML dependency, + never a DSH3 head or checkpoint. Trains on the same `train`/`held_out` + session split `synthesize_history_control_sessions` already assigns (via + `split_for_group`, unchanged). Fails closed (`ValueError`) if either split + has zero same-domain pairs -- the same NaN-avoidance discipline the sixth + slice's review fixes established for the sibling context-view harness. + `HistoryControlScorerReportV1` also carries `rejected_kind_counts` -- a + real, measured diagnostic over every pair (train and held-out), computed + specifically so a positive accuracy can never be read, unqualified, as + evidence of a hard preference signal (see "What was measured" below). +* `scripts/run_replay_preference_history_control_probe.py` -- now also runs + `train_history_control_pairwise_scorer` and prints its report whenever the + probe finds `same_domain_pair_count > 0`, alongside the existing + representability probe; prints `"scorer": null` when it doesn't. +* Regression tests: `tests/test_dsl/test_replay_preference.py` gains four + new tests -- a real-fixture test proving the concrete + edit-then-undo behavior flip (`rejected_action` is now + `checkout:`, not the operator token), a direct unit test of + `_pick_rejected` preferring a same-domain control alternative over a + lexicographically-earlier operator one, a direct unit test of the + any-domain fallback when no same-domain alternative exists (built via + `enumerate_operator_legal_set` with a single-member + `ordinary_nonoperator_actions`), and a test confirming `action_kind_of`'s + relocation preserves both import paths as the identical object. + `tests/test_harnesses/preference/test_replay_preference_history_control_policy.py` + updates its four ninth-slice tests that encoded the old + `same_domain_pair_count == 0` finding to the new, real, flipped numbers + (with docstrings explaining why), and adds two new tests directly + exercising `train_history_control_pairwise_scorer`'s real output, + including the lexicographic-artifact verdict and the negative `CHECKOUT` + weight that causes it. + +### What was measured (real run, `python -m scripts.run_replay_preference_history_control_probe`) + +* **`same_domain_pair_count` moved off zero, fully.** Over the real 46-row + trace-grounded, non-pronoun, non-merge subset of the seventh slice's own + 56-row synthetic corpus, every row now classifies `BOTH_CONTROL` + (`same_domain_pair_count=46`, `cross_domain_pair_count=0` -- a complete + flip from the ninth slice's `0`/`46`). `verdict` is now + `same_domain_pairs_found_fixture_scale`. +* **A pairwise scorer trains and shows real, positive held-out + discrimination: `pairwise_margin_accuracy=1.0`, `mean_margin≈4.36`** on 44 + train / 2 held-out pairs (the same `rollback_chain_1` held-out group the + ninth/context-view slices' own splits already assigned). +* **Honest interpretation -- this positive result is a second + lexicographic-tiebreak artifact, not a hard preference signal, and this + slice says so explicitly rather than reporting the bare number.** + `rejected_kind_counts` over every one of the 46 pairs (train and held-out + alike) is `{"checkout": 46}` -- the rejected side is **always** `checkout`, + with zero exceptions. Why: `_pick_rejected`'s same-domain preference still + breaks ties *within* a domain by plain lexicographic sort, and + `"checkout:"` sorts before every other control prefix + (`"merge:"`/`"redo:"`/`"undo"`) whenever a checkout candidate is legal -- + which `_available_history_actions` makes true at nearly every decision + state in this corpus (a `checkout:` candidate exists for every + *other* materialized state in the whole trace, and every trace here has + several). The trained scorer's weights confirm this directly: `[undo: + 2.21, redo: 0.70, checkout: -2.91, merge: 0.0]` -- a strongly negative + `CHECKOUT` weight and non-negative everything else is exactly what a + linear scorer needs to perfectly fit "reject checkout, always," nothing + more. This is structurally the same class of finding as the ninth slice's + own -- an ASCII-sort accident, not evidence a model had to work to + discriminate a genuinely hard preference -- just relocated one level down + (from cross-domain to within-domain) instead of eliminated. The report's + own `verdict` field says so directly: + `discrimination_is_lexicographic_tiebreak_artifact_fixture_scale`, not a + bare `some_discrimination_fixture_scale`. +* **Downstream-consumer impact, verified, not assumed.** `_pick_rejected` is + called only from within `replay_preference.py` itself (five call sites, + all listed above) -- grepped directly, confirmed no external caller exists + anywhere in `src/` or `tests/`. The sixth/seventh-slice + `preference_pairs_from_trace`/`preference_pair_from_replay_row` + `PreferencePair` conversion functions are downstream of extraction and + therefore *do* see the new `rejected_action` values on re-run, but no + test asserts an exact `rejected_action` string anywhere in this repo (only + `!=` / `.startswith("checkout:")`-style structural checks), and no + harness or script has ever written their output to a real corpus file or + fed it to `slm preference build-pairs`/`train` (confirmed by a repo-wide + grep: the only callers of those two functions are this module's own + tests). PR #1149's real preference-training corpus is independently + sourced and does not call `extract_replay_preference_rows`, + `_pick_rejected`, or either `PreferencePair` conversion function at all -- + it is structurally unaffected by this slice. The full relevant pytest + sweep (all five files that import or exercise `_pick_rejected`'s output, + directly or transitively) was re-run against the changed code and is + `143 passed`, with only the four ninth-slice tests that literally encoded + the old zero-pairing numbers updated to the new real numbers -- no other + test anywhere needed a change. +* **Is preferring same-domain rejection "more correct" as a default?** Yes, + in principle -- mixing an operator token against a control token is a + structurally easier discrimination (the `is_history_control` feature alone + already separated them, per the sixth/seventh slices' own ceiling-effect + finding) than a genuine same-domain comparison. But this slice's own + measurement shows the *specific* same-domain pairs this fixture now + produces are not, in fact, hard either -- they inherit a new, different + easy tell from the same remaining-lexicographic-tiebreak root cause. The + "more correct default" claim is about the *policy*, not a claim that this + particular fixture-scale corpus now yields informative pairs. + +### Explicitly out of scope (unchanged from, or newly confirmed by, this slice) + +* A genuinely non-lexicographic tie-break inside `_pick_rejected` (e.g. + rotating/hashing among same-domain candidates, or preferring diversity of + `control_kind` across a session) that would let the scorer's positive + result mean something -- not attempted here; see "Named next lever" below. +* `OperatorPolicyInputV1`, `OperatorActionViewV1`, `ReferenceModelViewV1`, + and `OperatorFeatureEncoder` -- all byte-for-byte unchanged, as in every + prior slice. +* Real SFT/preference training, a powered/real corpus, real + action/operator/argument accuracy against a certified checkpoint, and + CAP0/CAP1/CAP2 retention remain unattempted. +* A joint operator-and-control policy (design "A" from the adapter-gap doc) + is still not attempted; two independent, unmerged heads remain. + +No causal, calibration, or promotion claim is made. No checkpoint or model +card update applies -- this slice creates no checkpoint. Honesty tier: +`wiring` / `no_benefit_fixture_scale` for the row-extraction and +representability changes; the scorer result is real and positive but its own +`verdict` field names it a `fixture_scale` artifact, never a benefit claim. + +### Named next lever + +The real, concrete fix this slice's own finding points at: change +`_pick_rejected`'s within-domain tie-break to be non-lexicographic (e.g. +deterministic-hash-based rotation among same-domain candidates, so +`checkout`/`undo`/`redo` are not systematically favored or disfavored by +ASCII sort order alone), then re-run `train_history_control_pairwise_scorer` +to see whether `rejected_kind_counts` diversifies and, if it does, whether +the trained scorer's held-out discrimination survives losing its current +single-feature shortcut. This is a further row-extraction-policy refinement +(bumping `dsl.operators.replay_preference` again), not a model-input-contract +change, and is a real, separately-scoped eleventh slice, not a partial +version of this one. + +## Reproducibility (tenth slice) + +```bash +NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_harnesses/preference/test_operator_history_pairs.py tests/test_harnesses/preference/test_replay_preference_typed_policy_adapter.py tests/test_harnesses/preference/test_replay_preference_history_control_policy.py tests/test_evals/test_ambiguous_operator_followups.py tests/test_dsl/test_operator_conversation.py tests/test_models/test_operator_policy_view.py +NODE_OPTIONS= python -m scripts.run_replay_preference_history_control_probe +python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-3mu94n +python -m scripts.repo_policy +python -m scripts.verify_decode_invariants +ruff check src/slm_training/dsl/operators/replay_preference.py src/slm_training/dsl/operators/replay_preference_context_views.py src/slm_training/harnesses/preference/replay_preference_history_control_policy.py scripts/run_replay_preference_history_control_probe.py tests/test_dsl/test_replay_preference.py tests/test_harnesses/preference/test_replay_preference_history_control_policy.py +``` + +Result (this PR, tenth slice, `dsl.operators.replay_preference` v9, +`harness.preference.replay_preference_history_control_policy` v2): same +freshly built `.venv` (Python 3.12), `NODE_OPTIONS=` cleared for the OpenUI +bridge's Node 22 `--import tsx` incompatibility (same reason as every prior +slice): `143 passed`. `ruff check`: clean on every touched/created file. +`python -m scripts.verify_version_stamps --check --base +origin/claude/great-dirac-3mu94n`: `ok (7 changed file(s), 3 component(s) +touched)`. `python -m scripts.repo_policy`: `ok (tracked + untracked)`. +`python -m scripts.verify_decode_invariants`: clean (exit 0). The probe +script's real output: `same_domain_pair_count=46`, `cross_domain_pair_count=0`, +`verdict="same_domain_pairs_found_fixture_scale"`; scorer: +`train_pair_count=44`, `held_out_pair_count=2`, +`pairwise_margin_accuracy=1.0`, `mean_margin=4.358607031092988`, +`rejected_kind_counts={"checkout": 46}`, +`verdict="discrimination_is_lexicographic_tiebreak_artifact_fixture_scale"`. + ## Reproducibility (ninth slice) ```bash diff --git a/scripts/run_replay_preference_history_control_probe.py b/scripts/run_replay_preference_history_control_probe.py index c34b89433..0e1e10986 100644 --- a/scripts/run_replay_preference_history_control_probe.py +++ b/scripts/run_replay_preference_history_control_probe.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Run the SLM-418 (DSH5-10) ninth-slice history-control representability probe. +"""Run the SLM-418 (DSH5-10) history-control representability probe (+ scorer). Builds the representability report over the real 47-row (of 56) non-``PRONOUN_FOCUS_FOLLOWUP`` synthetic-corpus subset this slice's sibling @@ -7,11 +7,18 @@ same-domain (control-vs-control) chosen/rejected pair exists to train a pairwise scorer over. +Tenth slice: when ``same_domain_pair_count > 0`` (the tenth slice's +``_pick_rejected`` same-domain-preferring change, ``dsl/operators/ +replay_preference.py``), this script also trains and honestly reports the +pairwise linear scorer over ``HistoryControlPolicyInputV1.control_rows`` +(``train_history_control_pairwise_scorer``) -- never forced when no +same-domain pairs exist. + python -m scripts.run_replay_preference_history_control_probe Fixture-scale wiring evidence only -- never a certified decision-path or ship-readiness claim. See ``docs/design/dsh5-10-replay-preference-rows.md``'s -"Ninth slice" and +"Ninth slice"/"Tenth slice" and ``slm_training.harnesses.preference.replay_preference_history_control_policy``'s module docstring for exactly what this does and does not measure. Completes in a small fraction of a second, far inside @@ -25,6 +32,7 @@ from slm_training.harnesses.preference.replay_preference_history_control_policy import ( evaluate_history_control_representability, + train_history_control_pairwise_scorer, ) @@ -38,12 +46,19 @@ def main(argv: list[str] | None = None) -> int: ) args = parser.parse_args(argv) - report = evaluate_history_control_representability() - payload = json.dumps(report.to_dict(), indent=2, sort_keys=True) - print(payload) + probe = evaluate_history_control_representability() + payload: dict = {"probe": probe.to_dict()} + if probe.same_domain_pair_count > 0: + scorer = train_history_control_pairwise_scorer() + payload["scorer"] = scorer.to_dict() + else: + payload["scorer"] = None + + text = json.dumps(payload, indent=2, sort_keys=True) + print(text) if args.out: with open(args.out, "w", encoding="utf-8") as handle: - handle.write(payload + "\n") + handle.write(text + "\n") return 0 diff --git a/src/slm_training/dsl/operators/replay_preference.py b/src/slm_training/dsl/operators/replay_preference.py index b0880cb48..ca0875232 100644 --- a/src/slm_training/dsl/operators/replay_preference.py +++ b/src/slm_training/dsl/operators/replay_preference.py @@ -222,17 +222,80 @@ def legal_set_at( ) +def action_kind_of(serialized_action: str) -> str: + """Classify an opaque serialized action string into a small closed vocabulary. + + A structural prefix check only -- never an interpretation of the + action's arguments or any transcript content. ``"undo"`` matches + exactly; ``"redo:"``, ``"checkout:"``, and ``"merge:"`` match the fixed + canonical prefixes this module and ``merge.py`` already emit; anything + else is an operator (``AST_EDIT``) action. + + Tenth slice: moved here from ``replay_preference_context_views.py`` + (which now re-exports this exact function unchanged) so ``_pick_rejected`` + below can reuse the same closed vocabulary every other domain- + classification caller in this corpus already depends on + (``replay_preference_history_control_policy.py``'s + ``classify_replay_row_history_control_representability``, + ``replay_preference_typed_policy_adapter.py``'s representability check) + instead of duplicating the prefix logic a third time. + """ + if serialized_action == "undo": + return "undo" + for prefix, kind in (("redo:", "redo"), ("checkout:", "checkout"), ("merge:", "merge")): + if serialized_action.startswith(prefix): + return kind + return "operator" + + +def _rejection_domain_of(serialized_action: str) -> str: + """The coarse two-way domain ``_pick_rejected`` prefers to match: "operator" or "control". + + Deliberately coarser than :func:`action_kind_of`'s four-way control + vocabulary (``undo``/``redo``/``checkout``/``merge``) -- "domain" here + means the same operator-vs-control split + ``classify_replay_row_history_control_representability`` + (``replay_preference_history_control_policy.py``) already uses to decide + ``BOTH_CONTROL`` vs ``CHOSEN_CONTROL_REJECTED_OPERATOR``, not a specific + control kind. A rejected ``checkout:`` is exactly as same-domain + to a chosen ``undo`` as a rejected ``redo:`` would be. + """ + return "operator" if action_kind_of(serialized_action) == "operator" else "control" + + def _pick_rejected(legal_set: OperatorLegalSetV1, chosen: str) -> str | None: """A deterministic legal alternative to ``chosen``, or ``None`` if none exists. Drawn from the full legal set -- operator actions and available history - controls (``undo``/``redo:``) alike -- so a row is never dropped - just because the only unchosen alternative happens to be a control. + controls (``undo``/``redo:``/``checkout:``/``merge:``) + alike -- so a row is never dropped just because the only unchosen + alternative happens to be a control action. + + Tenth slice: prefers a same-domain alternative (operator-vs-operator, or + control-vs-control across any of undo/redo/checkout/merge) when the + legal set actually offers one, falling back to the lexicographically- + first any-domain candidate -- this function's original v2-v9 behavior -- + only when no same-domain alternative exists. Ties within a domain are + still broken lexicographically, so this stays fully deterministic. + + This is a deliberate strengthening, not merely a different tiebreak: the + ninth slice found ``same_domain_pair_count == 0`` over a real 46-row + corpus because a serialized operator token always sorts before every + control-token prefix, so the old lexicographic-first policy silently + always picked the "easy," cross-domain alternative even when a genuinely + harder same-domain one was legally available -- see "Tenth slice" in + ``docs/design/dsh5-10-replay-preference-rows.md``. """ candidates = sorted( action for action in legal_set.all_serialized_actions if action != chosen ) - return candidates[0] if candidates else None + if not candidates: + return None + chosen_domain = _rejection_domain_of(chosen) + same_domain = [ + action for action in candidates if _rejection_domain_of(action) == chosen_domain + ] + return same_domain[0] if same_domain else candidates[0] def _touched_refs(application: OperatorApplicationV1) -> frozenset: diff --git a/src/slm_training/dsl/operators/replay_preference_context_views.py b/src/slm_training/dsl/operators/replay_preference_context_views.py index 0778f9216..6668aed97 100644 --- a/src/slm_training/dsl/operators/replay_preference_context_views.py +++ b/src/slm_training/dsl/operators/replay_preference_context_views.py @@ -40,7 +40,10 @@ ConversationStateNodeV1, ConversationTraceV1, ) -from slm_training.dsl.operators.replay_preference import OperatorEventMemoryReportV1 +from slm_training.dsl.operators.replay_preference import ( + OperatorEventMemoryReportV1, + action_kind_of, # noqa: F401 -- re-exported unchanged for existing callers +) from slm_training.harness_core.versioning import build_version_stamp #: The issue's own matrix: "1/2/4/8/16 turn depth." @@ -89,21 +92,11 @@ class ContextView(str, Enum): } -def action_kind_of(serialized_action: str) -> str: - """Classify an opaque serialized action string into a small closed vocabulary. - - A structural prefix check only -- never an interpretation of the - action's arguments or any transcript content. ``"undo"`` matches - exactly; ``"redo:"``, ``"checkout:"``, and ``"merge:"`` match the fixed - canonical prefixes ``replay_preference.py``/``merge.py`` already emit; - anything else is an operator (``AST_EDIT``) action. - """ - if serialized_action == "undo": - return "undo" - for prefix, kind in (("redo:", "redo"), ("checkout:", "checkout"), ("merge:", "merge")): - if serialized_action.startswith(prefix): - return kind - return "operator" +# ``action_kind_of`` moved to ``replay_preference.py`` (tenth slice), which +# now also uses it internally for ``_pick_rejected``'s same-domain +# preference; re-exported here unchanged so every existing import of +# ``slm_training.dsl.operators.replay_preference_context_views.action_kind_of`` +# keeps working byte-for-byte. @dataclass(frozen=True) diff --git a/src/slm_training/harnesses/preference/replay_preference_history_control_policy.py b/src/slm_training/harnesses/preference/replay_preference_history_control_policy.py index 61fce1b28..1e8647258 100644 --- a/src/slm_training/harnesses/preference/replay_preference_history_control_policy.py +++ b/src/slm_training/harnesses/preference/replay_preference_history_control_policy.py @@ -98,7 +98,9 @@ from __future__ import annotations +import math import time +from collections.abc import Sequence from dataclasses import dataclass from enum import Enum @@ -134,6 +136,7 @@ "HistoryControlProbeReportV1", "HistoryControlRepresentabilityReportV1", "HistoryControlRowRepresentability", + "HistoryControlScorerReportV1", "HistoryControlSessionV1", "adapt_replay_row_to_history_control_policy_input", "build_history_control_policy_input", @@ -141,6 +144,7 @@ "classify_replay_row_history_control_representability", "evaluate_history_control_representability", "synthesize_history_control_sessions", + "train_history_control_pairwise_scorer", ] @@ -570,3 +574,300 @@ def evaluate_history_control_representability( "harness.preference.replay_preference_history_control_policy" ), ) + + +# --------------------------------------------------------------------------- # +# Tenth slice: a pairwise scorer over HistoryControlPolicyInputV1.control_rows. +# +# Only reachable when evaluate_history_control_representability() found +# same_domain_pair_count > 0 -- the tenth slice's own _pick_rejected change +# (dsl/operators/replay_preference.py) is what makes that true for this +# corpus. This is deliberately a tiny linear probe, same discipline as the +# sixth slice's two-feature scorer and the eighth slice's argument-margin +# probe: never a DSH3 head or checkpoint, no external ML dependency. +# --------------------------------------------------------------------------- # +_CONTROL_KIND_ORDER: tuple[ControlActionKind, ...] = ( + ControlActionKind.UNDO, + ControlActionKind.REDO, + ControlActionKind.CHECKOUT, + ControlActionKind.MERGE, +) + + +def _control_kind_features(kind: ControlActionKind) -> tuple[float, ...]: + """One-hot encoding over the closed four-member ``ControlActionKind`` vocabulary. + + ``control_kind`` is the *only* structural field ``HistoryControlActionViewV1`` + carries at all (its own docstring: no opaque state id is ever encoded) -- + so this is, by construction, the entire input space available to any + scorer trained over ``HistoryControlPolicyInputV1.control_rows``. + """ + return tuple(1.0 if kind is member else 0.0 for member in _CONTROL_KIND_ORDER) + + +def _sigmoid(x: float) -> float: + if x >= 0: + return 1.0 / (1.0 + math.exp(-x)) + ex = math.exp(x) + return ex / (1.0 + ex) + + +def _train_pairwise_control_kind_scorer( + pairs: Sequence[tuple[ControlActionKind, ControlActionKind]], + *, + steps: int, + lr: float, +) -> tuple[float, ...]: + """Full-batch gradient descent on pairwise logistic loss over one-hot control-kind features. + + Same convention as the sixth slice's ``_train_pairwise_linear_scorer`` + (``replay_preference_context_view_variants.py``): no bias term (it + cancels in a chosen-minus-rejected margin), plain full-batch GD, no + external ML dependency. Generalized to the four-dimensional one-hot + feature space instead of that function's fixed two floats, since + ``control_kind`` is categorical, not a pair of independent scalars. + """ + n_features = len(_CONTROL_KIND_ORDER) + weights = [0.0] * n_features + n = len(pairs) + if n == 0: + return tuple(weights) + for _ in range(steps): + grads = [0.0] * n_features + for chosen_kind, rejected_kind in pairs: + deltas = [ + c - r + for c, r in zip( + _control_kind_features(chosen_kind), + _control_kind_features(rejected_kind), + strict=True, + ) + ] + margin = sum(w * d for w, d in zip(weights, deltas, strict=True)) + grad_coeff = _sigmoid(margin) - 1.0 # d(-log sigmoid(margin))/d(margin) + for index, delta in enumerate(deltas): + grads[index] += grad_coeff * delta + weights = [w - lr * g / n for w, g in zip(weights, grads, strict=True)] + return tuple(weights) + + +def _control_kind_margin( + weights: Sequence[float], + chosen_kind: ControlActionKind, + rejected_kind: ControlActionKind, +) -> float: + deltas = [ + c - r + for c, r in zip( + _control_kind_features(chosen_kind), + _control_kind_features(rejected_kind), + strict=True, + ) + ] + return sum(w * d for w, d in zip(weights, deltas, strict=True)) + + +def _control_kind_pairs_for_split( + sessions: Sequence[HistoryControlSessionV1], *, split: str +) -> list[tuple[ControlActionKind, ControlActionKind]]: + """Real ``(chosen_kind, rejected_kind)`` pairs for one split's ``BOTH_CONTROL`` rows. + + Reuses :func:`adapt_replay_row_to_history_control_policy_input` (not a + shadow re-derivation) so the same fail-closed fingerprint check every + other consumer of that function gets applies here too. + """ + pairs: list[tuple[ControlActionKind, ControlActionKind]] = [] + for session in sessions: + if session.split != split: + continue + for row in session.rows: + result = adapt_replay_row_to_history_control_policy_input( + row, + session.trace, + library=session.library, + pack=session.pack, + provenance_for=_provenance, + ) + if result.representability is not HistoryControlRowRepresentability.BOTH_CONTROL: + continue + assert result.policy_input is not None + assert result.accepted_control_row is not None + assert result.rejected_control_row is not None + chosen_kind = result.policy_input.control_rows[ + result.accepted_control_row + ].control_kind + rejected_kind = result.policy_input.control_rows[ + result.rejected_control_row + ].control_kind + pairs.append((chosen_kind, rejected_kind)) + return pairs + + +@dataclass(frozen=True) +class HistoryControlScorerReportV1: + """A trained pairwise linear scorer over ``HistoryControlPolicyInputV1.control_rows``. + + Same honesty discipline as every prior slice's scorer/probe: measured + directly, never forced positive. ``pairwise_margin_accuracy``/ + ``mean_margin`` use the same held-out-pair convention as the eighth + slice's ``TypedPolicyArgumentProbeReportV1`` -- a tie (``margin == 0``) + earns 0.5 credit, never counted as a win. + + ``rejected_kind_counts`` is a real, measured diagnostic (over every + train + held-out pair, both splits), not a training input: it exists so + a positive ``pairwise_margin_accuracy`` can never be read, unqualified, + as evidence of a genuinely hard preference signal. See + ``_LEXICOGRAPHIC_ARTIFACT_NOTE`` -- when it shows only one distinct + rejected ``control_kind`` across the whole corpus, that is direct proof + the "same-domain" pairs this slice's ``_pick_rejected`` change produced + are still a plain-lexicographic-tiebreak artifact (just relocated + within the control domain, from the ninth slice's cross-domain one to + "``checkout`` always sorts first among control prefixes when legal"), + not a real hard-negative signal a scorer had to work to discriminate. + """ + + train_pair_count: int + held_out_pair_count: int + pairwise_margin_accuracy: float + mean_margin: float + weights: tuple[float, ...] + rejected_kind_counts: dict[str, int] + verdict: str + note: str + version_stamp: dict + schema: str = "history_control_scorer_report/v1" + + def to_dict(self) -> dict: + return { + "schema": self.schema, + "train_pair_count": self.train_pair_count, + "held_out_pair_count": self.held_out_pair_count, + "pairwise_margin_accuracy": self.pairwise_margin_accuracy, + "mean_margin": self.mean_margin, + "weights": list(self.weights), + "rejected_kind_counts": dict(self.rejected_kind_counts), + "verdict": self.verdict, + "note": self.note, + "version_stamp": self.version_stamp, + } + + +_NO_DISCRIMINATION_NOTE = ( + "held-out pairwise margin is at or below zero on average: this " + "four-dimensional one-hot control_kind scorer does not discriminate " + "chosen from rejected on this corpus. Not evidence of a training bug -- " + "control_kind is the only feature available, and the corpus's own " + "chosen/rejected control_kind pairing may simply not correlate with " + "kind identity at this scale. Never a causal, calibration, or " + "promotion claim." +) + +_LEXICOGRAPHIC_ARTIFACT_NOTE = ( + "held-out pairwise margin is positive, but this is NOT evidence of a " + "real hard preference signal: rejected_kind_counts shows every single " + "same-domain pair in this corpus (train and held-out alike) has the " + "identical rejected control_kind. _pick_rejected's same-domain " + "preference (this slice) still breaks ties within a domain by plain " + "lexicographic sort, and 'checkout:' sorts before every other control " + "prefix ('merge:'/'redo:'/'undo') whenever a checkout candidate is " + "legal, which this corpus's own trace shape makes true for nearly " + "every decision state. The scorer's positive margin is therefore " + "explained entirely by 'checkout is always the rejected side', a " + "second lexicographic-tiebreak artifact, not a signal the model had " + "to work to discriminate -- the exact same class of caution the ninth " + "slice raised for the cross-domain 'operator always sorts first' " + "artifact this slice fixed, now recurring one level down. Never a " + "causal, calibration, or promotion claim." +) + +_SOME_DISCRIMINATION_NOTE = ( + "held-out pairwise margin is positive on average, and rejected_kind_counts " + "shows more than one distinct rejected control_kind in this corpus, so " + "this is not explained by a single-kind lexicographic-tiebreak artifact " + "alone. Still a tiny linear probe over one categorical feature, on a " + "fixture-scale synthetic corpus -- never a causal, calibration, or " + "promotion claim, and not a certified decision-path result." +) + + +def train_history_control_pairwise_scorer( + *, + steps: int = 200, + lr: float = 0.5, + max_wall_seconds: float = MAX_HARNESS_WALL_SECONDS, +) -> HistoryControlScorerReportV1: + """Train and honestly evaluate a pairwise scorer over same-domain control pairs. + + Fails closed (``ValueError``) if either split has zero same-domain + pairs -- an empty split producing a NaN accuracy that could silently + read as "no benefit" was exactly the bug the sixth slice's review fixes + closed for the sibling context-view harness; this function does not + repeat it. Enforces the repo's hard run cap even though a real run over + this corpus completes in a small fraction of a second. + """ + start = time.monotonic() + sessions = synthesize_history_control_sessions() + train_pairs = _control_kind_pairs_for_split(sessions, split="train") + held_out_pairs = _control_kind_pairs_for_split(sessions, split="held_out") + if time.monotonic() - start > max_wall_seconds: + raise TimeoutError( + "history-control scorer training exceeded MAX_HARNESS_WALL_SECONDS " + f"({max_wall_seconds}s); a timed-out run is never evidence " + "(AGENTS.md hard run cap)" + ) + if not train_pairs: + raise ValueError( + "no same-domain train pairs to fit a history-control scorer on" + ) + if not held_out_pairs: + raise ValueError( + "no same-domain held-out pairs to evaluate a history-control scorer on" + ) + + weights = _train_pairwise_control_kind_scorer(train_pairs, steps=steps, lr=lr) + if time.monotonic() - start > max_wall_seconds: + raise TimeoutError( + "history-control scorer training exceeded MAX_HARNESS_WALL_SECONDS " + f"({max_wall_seconds}s); a timed-out run is never evidence " + "(AGENTS.md hard run cap)" + ) + + correct = 0.0 + margin_sum = 0.0 + for chosen_kind, rejected_kind in held_out_pairs: + margin = _control_kind_margin(weights, chosen_kind, rejected_kind) + margin_sum += margin + correct += 1.0 if margin > 0 else (0.5 if margin == 0 else 0.0) + accuracy = correct / len(held_out_pairs) + mean_margin = margin_sum / len(held_out_pairs) + + rejected_kind_counts: dict[str, int] = {} + for _chosen_kind, rejected_kind in (*train_pairs, *held_out_pairs): + rejected_kind_counts[rejected_kind.value] = ( + rejected_kind_counts.get(rejected_kind.value, 0) + 1 + ) + is_lexicographic_artifact = mean_margin > 0 and len(rejected_kind_counts) == 1 + + if mean_margin <= 0: + verdict = "no_discrimination_fixture_scale" + note = _NO_DISCRIMINATION_NOTE + elif is_lexicographic_artifact: + verdict = "discrimination_is_lexicographic_tiebreak_artifact_fixture_scale" + note = _LEXICOGRAPHIC_ARTIFACT_NOTE + else: + verdict = "some_discrimination_fixture_scale" + note = _SOME_DISCRIMINATION_NOTE + + return HistoryControlScorerReportV1( + train_pair_count=len(train_pairs), + held_out_pair_count=len(held_out_pairs), + pairwise_margin_accuracy=accuracy, + mean_margin=mean_margin, + weights=weights, + rejected_kind_counts=rejected_kind_counts, + verdict=verdict, + note=note, + version_stamp=build_version_stamp( + "harness.preference.replay_preference_history_control_policy" + ), + ) diff --git a/src/slm_training/resources/versions.json b/src/slm_training/resources/versions.json index 27e3f1103..c91c4a7fe 100644 --- a/src/slm_training/resources/versions.json +++ b/src/slm_training/resources/versions.json @@ -1112,13 +1112,18 @@ ] }, "dsl.operators.replay_preference": { - "version": "v8", + "version": "v9", "kind": "harness", "paths": [ "src/slm_training/dsl/operators/replay_preference.py", "tests/test_dsl/test_replay_preference.py" ], "history": [ + { + "version": "v9", + "date": "2026-07-28", + "note": "SLM-418 (DSH5-10) tenth slice: _pick_rejected now prefers a same-domain alternative (operator-vs-operator, or control-vs-control across any of undo/redo/checkout/merge) when the legal set actually offers one, falling back to the old any-domain lexicographic-first candidate only when it doesn't -- fixing the ninth slice's own documented finding that a serialized operator token always sorted first, so same_domain_pair_count was always 0. Also relocates action_kind_of here from dsl.operators.replay_preference_context_views (which now re-exports it unchanged) so this new domain check can reuse the same closed vocabulary every other domain-classification caller in this corpus already depends on, instead of duplicating the prefix logic a third time. Real effect on the real 56-row synthetic corpus: all 46 trace-grounded non-pronoun rows now classify BOTH_CONTROL (was CHOSEN_CONTROL_REJECTED_OPERATOR), same_domain_pair_count moves from 0 to 46 -- but every single rejected side is 'checkout' (verified: rejected_kind_counts == {checkout: 46}), because 'checkout:' still sorts first among control prefixes whenever legal in this corpus, which is nearly always. A trained pairwise scorer over the new pairs (harness.preference.replay_preference_history_control_policy v2) shows real positive held-out margin/accuracy, but this is honestly reported as a second lexicographic-tiebreak artifact, not a hard preference signal -- see docs/design/dsh5-10-replay-preference-rows.md's Tenth slice. No downstream corpus (sixth/seventh-slice PreferencePair conversion, PR #1149's real preference-training corpus) calls _pick_rejected outside this module; regression tests cover both the same-domain-preferred path and the any-domain fallback directly" + }, { "version": "v8", "date": "2026-07-28", @@ -1168,6 +1173,11 @@ "src/slm_training/dsl/operators/replay_preference_context_views.py" ], "history": [ + { + "version": "v1", + "date": "2026-07-28", + "note": "no-bump: SLM-418 (DSH5-10) tenth slice moves action_kind_of's definition to dsl.operators.replay_preference.py (which now also uses it internally for _pick_rejected's same-domain preference) and re-exports it here unchanged -- byte-for-byte identical behavior, same public symbol, same import path for every existing caller. No API or behavior change to this module" + }, { "version": "v1", "date": "2026-07-27", @@ -8060,6 +8070,11 @@ "docs/design/dsh5-10-replay-preference-rows.md" ], "history": [ + { + "version": "v3", + "date": "2026-07-28", + "note": "no-bump: SLM-418 (DSH5-10) tenth slice appends its own 'Tenth slice' and 'Reproducibility (tenth slice)' sections to docs/design/dsh5-10-replay-preference-rows.md (a shared doc this component's paths claim), and updates the top 'Status' line to 'tenth increment'; no change to replay_preference_context_view_variants.py itself -- the tenth slice's own dsl.operators.replay_preference v9 bump and harness.preference.replay_preference_history_control_policy v2 bump cover the actual new code" + }, { "version": "v3", "date": "2026-07-28", @@ -8109,7 +8124,7 @@ ] }, "harness.preference.replay_preference_history_control_policy": { - "version": "v1", + "version": "v2", "kind": "harness", "paths": [ "src/slm_training/harnesses/preference/replay_preference_history_control_policy.py", @@ -8117,6 +8132,11 @@ "tests/test_harnesses/preference/test_replay_preference_history_control_policy.py" ], "history": [ + { + "version": "v2", + "date": "2026-07-28", + "note": "SLM-418 (DSH5-10) tenth slice: re-run against the tenth slice's own dsl.operators.replay_preference v9 (_pick_rejected now prefers a same-domain alternative) -- same_domain_pair_count moves from 0 to 46 (evaluate_history_control_representability's verdict flips to same_domain_pairs_found_fixture_scale). Adds train_history_control_pairwise_scorer, a small pairwise linear scorer over the new HistoryControlPolicyInputV1.control_rows same-domain pairs (one-hot control_kind features, full-batch logistic-loss GD, same discipline as the sixth slice's two-feature scorer). Real result: held-out pairwise_margin_accuracy=1.0, mean_margin>0 (44 train / 2 held-out pairs) -- but rejected_kind_counts shows the rejected side is 'checkout' for all 46 pairs, so this is honestly reported as a second lexicographic-tiebreak artifact (checkout: still sorts first among control prefixes whenever legal in this corpus), not a hard preference signal; verdict is discrimination_is_lexicographic_tiebreak_artifact_fixture_scale, not a bare positive claim. Fails closed (ValueError) if either split has zero same-domain pairs. See docs/design/dsh5-10-replay-preference-rows.md's Tenth slice for the full disposition" + }, { "version": "v1", "date": "2026-07-28", diff --git a/tests/test_dsl/test_replay_preference.py b/tests/test_dsl/test_replay_preference.py index 555ec7b65..96795e413 100644 --- a/tests/test_dsl/test_replay_preference.py +++ b/tests/test_dsl/test_replay_preference.py @@ -894,3 +894,117 @@ def test_preference_pair_from_replay_row_refuses_to_fabricate_a_blank_prompt() - with pytest.raises(ValueError, match="non-empty"): preference_pair_from_replay_row(row, input_state_source=" ") + +# --------------------------------------------------------------------------- # +# Tenth slice: _pick_rejected's same-domain-preferring policy. +# +# The ninth slice found that, over a real 46-row corpus, _pick_rejected +# always picked a cross-domain operator action as `rejected` (a serialized +# operator token starts with "OPERATOR ", which sorts before every control +# prefix), leaving zero same-domain (control-vs-control) pairs to train a +# pairwise scorer over. This slice changes the policy to prefer a +# same-domain alternative when the legal set actually offers one, falling +# back to the old any-domain lexicographic-first behavior only when it +# doesn't -- these tests cover both branches directly, against real fixtures +# and against the private function itself. +# --------------------------------------------------------------------------- # +def test_edit_then_undo_now_rejects_a_same_domain_checkout_not_the_operator() -> None: + """The concrete real-fixture behavior flip this slice introduces. + + At the edit-then-undo decision state, the legal set has exactly one + same-domain (control) alternative to "undo" -- checkout: -- + plus at least one cross-domain operator action. Before this slice, + _pick_rejected's plain lexicographic sort over the *full* legal set + always picked the operator action here (see the ninth slice's own + documented "OPERATOR sorts before checkout:" finding). This slice's + same-domain preference now picks the control alternative instead. + """ + pack, library, root = _fixture() + edited, _application = _append(pack, library, root) + undone = undo_conversation(edited, provenance=_provenance(edited.current.state)) + + report = extract_replay_preference_rows( + undone, pack=pack, library=library, provenance_for=_provenance + ) + + row = report.rows[0] + assert row.chosen_action == "undo" + assert row.rejected_action == f"checkout:{root.root_state_id}" + assert not row.rejected_action.startswith("OPERATOR ") + + +def test_pick_rejected_prefers_same_domain_for_a_control_chosen_action() -> None: + """Direct unit coverage: a control ``chosen`` prefers a control ``rejected``. + + A hand-built legal set carries one operator action (which would have + sorted first under the old any-domain policy: "OPERATOR " < "undo") and + two control actions, "checkout:"/"undo". Choosing "undo" must reject + "checkout:" (same domain), never the operator action, even though the + operator action is lexicographically first overall. + """ + from slm_training.dsl.operators.legal_set import enumerate_operator_legal_set + from slm_training.dsl.operators.replay_preference import _pick_rejected + + pack, library, root = _fixture() + state = root.current.state + legal_set = enumerate_operator_legal_set( + pack=pack, + library=library, + state=state, + reference_table=root.current.reference_table, + provenance=_provenance(state), + ordinary_nonoperator_actions=("checkout:some-other-state", "undo"), + ) + assert legal_set.operator_actions # a real cross-domain candidate exists too + + rejected = _pick_rejected(legal_set, "undo") + + assert rejected == "checkout:some-other-state" + + +def test_pick_rejected_falls_back_to_any_domain_with_no_same_domain_alternative() -> None: + """Direct unit coverage: no same-domain alternative falls back to the old policy. + + A hand-built legal set carries "undo" as the *only* control action (so + no same-domain alternative exists once it is chosen) plus real operator + actions. ``_pick_rejected`` must still return a legal alternative -- + the lexicographic-first cross-domain candidate, this function's + original v2-v9 behavior -- rather than returning ``None`` and silently + dropping the row. + """ + from slm_training.dsl.operators.legal_set import enumerate_operator_legal_set + from slm_training.dsl.operators.replay_preference import _pick_rejected + + pack, library, root = _fixture() + state = root.current.state + legal_set = enumerate_operator_legal_set( + pack=pack, + library=library, + state=state, + reference_table=root.current.reference_table, + provenance=_provenance(state), + ordinary_nonoperator_actions=("undo",), + ) + assert legal_set.operator_actions + + rejected = _pick_rejected(legal_set, "undo") + + assert rejected is not None + assert rejected != "undo" + assert rejected.startswith("OPERATOR ") + + +def test_action_kind_of_is_re_exported_unchanged_from_context_views() -> None: + """``action_kind_of`` moved to this module this slice; the old import path stays live.""" + from slm_training.dsl.operators.replay_preference import action_kind_of as moved + from slm_training.dsl.operators.replay_preference_context_views import ( + action_kind_of as re_exported, + ) + + assert moved is re_exported + assert moved("undo") == "undo" + assert moved("redo:x") == "redo" + assert moved("checkout:x") == "checkout" + assert moved("merge:a:b") == "merge" + assert moved("OPERATOR fixture.op x=value:request-1:opaque") == "operator" + diff --git a/tests/test_harnesses/preference/test_replay_preference_history_control_policy.py b/tests/test_harnesses/preference/test_replay_preference_history_control_policy.py index a440fb9a3..552a6dc3d 100644 --- a/tests/test_harnesses/preference/test_replay_preference_history_control_policy.py +++ b/tests/test_harnesses/preference/test_replay_preference_history_control_policy.py @@ -1,8 +1,13 @@ -"""Tests for SLM-418 (DSH5-10) ninth slice: sibling history-control policy view. +"""Tests for SLM-418 (DSH5-10) ninth/tenth slices: sibling history-control policy view. -See ``docs/design/dsh5-10-replay-preference-rows.md``'s "Ninth slice" and +See ``docs/design/dsh5-10-replay-preference-rows.md``'s "Ninth slice"/"Tenth +slice" and ``slm_training.harnesses.preference.replay_preference_history_control_policy``'s -module docstring for the full disposition. +module docstring for the full disposition. The tenth slice's own +``_pick_rejected`` same-domain-preferring change flips several of this +module's real numbers (documented inline on each affected test), and adds +``train_history_control_pairwise_scorer`` coverage at the bottom of this +file. """ from __future__ import annotations @@ -25,6 +30,7 @@ classify_replay_row_history_control_representability, evaluate_history_control_representability, synthesize_history_control_sessions, + train_history_control_pairwise_scorer, ) from slm_training.models.operator_policy_view import OperatorPolicyViewError @@ -78,14 +84,23 @@ def test_the_other_six_relations_chosen_side_is_a_control_token(relation) -> Non assert classification is not HistoryControlRowRepresentability.NOT_REPRESENTABLE_OPERATOR_CHOSEN -def test_real_corpus_has_zero_same_domain_pairs_every_rejected_side_is_an_operator() -> None: - """The real, empirically verified finding this slice's docstring claims. - - Every one of the 47 non-``PRONOUN_FOCUS_FOLLOWUP`` rows in the real - synthetic corpus classifies ``CHOSEN_CONTROL_REJECTED_OPERATOR``, never - ``BOTH_CONTROL`` -- ``_pick_rejected`` always picks the operator token - because it sorts first lexicographically whenever one is legal, which it - always is in this fixture. +def test_real_corpus_now_has_same_domain_pairs_since_the_tenth_slice() -> None: + """The tenth slice's own re-run of this exact finding, now flipped. + + The ninth slice's ``_pick_rejected`` always picked a cross-domain + operator token as ``rejected`` (a serialized operator token sorts + before every control-token prefix). The tenth slice changed + ``_pick_rejected`` to prefer a same-domain alternative when the legal + set actually offers one -- and every decision state in this real + synthetic corpus does offer one (at least a ``checkout:`` + candidate, per ``_available_history_actions``). Every one of the 47 + non-``PRONOUN_FOCUS_FOLLOWUP`` rows now classifies ``BOTH_CONTROL``, + never ``CHOSEN_CONTROL_REJECTED_OPERATOR``. See + ``test_replay_preference.py``'s own direct ``_pick_rejected`` unit + tests for the underlying mechanism, and "Tenth slice" in + ``docs/design/dsh5-10-replay-preference-rows.md`` for why this + doesn't, by itself, mean the resulting pairs are a *hard* preference + signal -- see this file's own scorer tests below. """ by_relation = _rows_by_relation() for relation, rows in by_relation.items(): @@ -94,7 +109,7 @@ def test_real_corpus_has_zero_same_domain_pairs_every_rejected_side_is_an_operat for row in rows: assert ( classify_replay_row_history_control_representability(row) - is HistoryControlRowRepresentability.CHOSEN_CONTROL_REJECTED_OPERATOR + is HistoryControlRowRepresentability.BOTH_CONTROL ) @@ -106,14 +121,14 @@ def test_build_history_control_representability_report_matches_the_real_full_cor assert report.total_rows == 56 assert report.counts_by_representability == { "not_representable_operator_chosen": 9, - "chosen_control_rejected_cross_domain_operator": 47, + "representable_both_control": 47, } assert set(report.counts_by_relation["pronoun_focus_followup"]) == { "not_representable_operator_chosen" } for relation, counts in report.counts_by_relation.items(): if relation != "pronoun_focus_followup": - assert set(counts) == {"chosen_control_rejected_cross_domain_operator"} + assert set(counts) == {"representable_both_control"} # --------------------------------------------------------------------------- # @@ -134,17 +149,21 @@ def test_adapt_replay_row_builds_a_real_control_row_for_the_chosen_action() -> N provenance_for=_provenance, ) - assert ( - result.representability - is HistoryControlRowRepresentability.CHOSEN_CONTROL_REJECTED_OPERATOR - ) + # Tenth slice: _pick_rejected now prefers a same-domain alternative, and + # this fixture always has one (at least a checkout candidate), so this + # row -- like every non-pronoun row in the real corpus -- is now + # BOTH_CONTROL, not CHOSEN_CONTROL_REJECTED_OPERATOR. + assert result.representability is HistoryControlRowRepresentability.BOTH_CONTROL assert result.policy_input is not None assert result.accepted_control_row is not None accepted_view = result.policy_input.control_rows[result.accepted_control_row] assert accepted_view.control_kind is ControlActionKind.UNDO - # The rejected action is a cross-domain operator token, so no control row - # represents it -- this is the real, honestly reported limit. - assert result.rejected_control_row is None + # The rejected action is now a same-domain control token too, so a + # control row represents it as well. + assert result.rejected_control_row is not None + rejected_view = result.policy_input.control_rows[result.rejected_control_row] + assert row.rejected_action.startswith("checkout:") + assert rejected_view.control_kind is ControlActionKind.CHECKOUT def test_build_history_control_policy_input_never_carries_the_opaque_state_id() -> None: @@ -227,17 +246,78 @@ def test_adapt_replay_row_fails_closed_on_a_stale_legal_set_fingerprint() -> Non # --------------------------------------------------------------------------- # # evaluate_history_control_representability: the real end-to-end report. # --------------------------------------------------------------------------- # -def test_evaluate_history_control_representability_reports_the_real_zero_pairing_result() -> None: +def test_evaluate_history_control_representability_reports_the_real_same_domain_result() -> None: + """Tenth slice: this exact probe, re-run, now reports the flipped finding. + + ``_pick_rejected``'s same-domain preference (this slice) means every + one of the 46 trace-grounded, non-pronoun, non-merge rows in the real + corpus now pairs a control chosen action against a control rejected + action -- see ``docs/design/dsh5-10-replay-preference-rows.md``'s + "Tenth slice" for why this genuinely moves ``same_domain_pair_count`` + off zero without, by itself, proving a hard preference signal exists + (see the scorer tests below for that separate question). + """ report = evaluate_history_control_representability() - assert report.same_domain_pair_count == 0 - assert report.cross_domain_pair_count == 46 - assert report.verdict == "no_same_domain_pairs_fixture_scale" - assert "always the operator action" in report.note + assert report.same_domain_pair_count == 46 + assert report.cross_domain_pair_count == 0 + assert report.verdict == "same_domain_pairs_found_fixture_scale" + assert "same-domain" in report.note assert report.representability.total_rows == 46 assert report.representability.counts_by_representability == { - "chosen_control_rejected_cross_domain_operator": 46 + "representable_both_control": 46 } assert report.version_stamp["components"] == { - "harness.preference.replay_preference_history_control_policy": "v1" + "harness.preference.replay_preference_history_control_policy": "v2" } + + +# --------------------------------------------------------------------------- # +# Tenth slice: train_history_control_pairwise_scorer -- the stretch goal the +# ninth slice's own "Named next lever" pointed at, now reachable. +# --------------------------------------------------------------------------- # +def test_train_history_control_pairwise_scorer_reports_the_real_lexicographic_artifact() -> None: + """The real, measured, honest result -- not forced positive. + + ``same_domain_pair_count`` moved off zero, so a pairwise scorer over + ``HistoryControlPolicyInputV1.control_rows`` can now be trained. Its + held-out accuracy is real and positive (1.0 on n=2 held-out pairs), but + ``rejected_kind_counts`` shows the rejected side is *always* "checkout" + across the whole corpus -- ``_pick_rejected``'s same-domain preference + still breaks ties within a domain by plain lexicographic sort, and + "checkout:" sorts first among control prefixes whenever legal, which + this corpus's trace shape makes true almost everywhere. So this is + correctly reported as a lexicographic-tiebreak artifact, not a hard + preference signal -- same rigor as the eighth slice's zero-margin + finding, just for a positive number this time. + """ + report = train_history_control_pairwise_scorer() + + assert report.train_pair_count == 44 + assert report.held_out_pair_count == 2 + assert report.mean_margin > 0 + assert report.pairwise_margin_accuracy == 1.0 + assert report.rejected_kind_counts == {"checkout": 46} + assert report.verdict == "discrimination_is_lexicographic_tiebreak_artifact_fixture_scale" + assert "NOT evidence of a real hard preference signal" in report.note + assert report.version_stamp["components"] == { + "harness.preference.replay_preference_history_control_policy": "v2" + } + + +def test_train_history_control_pairwise_scorer_weights_penalize_checkout() -> None: + """Direct evidence for the artifact: the trained weight on CHECKOUT is negative. + + ``ControlActionKind`` order is ``(UNDO, REDO, CHECKOUT, MERGE)`` -- + since ``checkout`` is the rejected side of every single training pair, + a linear pairwise scorer only needs a negative CHECKOUT weight (and + non-negative others) to fit the corpus, exactly what a real trained run + produces. + """ + report = train_history_control_pairwise_scorer() + + undo_weight, redo_weight, checkout_weight, merge_weight = report.weights + assert checkout_weight < 0 + assert undo_weight > 0 + assert redo_weight > 0 + assert merge_weight == 0.0 # MERGE never appears as chosen or rejected in this corpus