Skip to content

docs(autotrain): joint seed x steps smoke sweep, real new evidence #1132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Tyler-R-Kendrick merged 2 commits into from
Jul 27, 2026
Merged

docs(autotrain): joint seed x steps smoke sweep, real new evidence #1132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f3adde1
SLM-418: convert replay preference rows to PreferencePair (DSH5-10)
claude Jul 27, 2026
811d62a
docs(autotrain): joint seed x steps smoke sweep, real new evidence
claude Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/design/autotrain-loop-ledger-20260725.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,3 +308,37 @@ fixture entirely and pick up one of the repo's actually-open threads (the
DSH5-10 SFT/preference-training and four-baseline comparison scope, or the
next queued `AP-007+` campaign arm) — the smoke-loop's role as a harness
liveness check is now well covered by the batches in this file.

## Joint seed x steps sweep (2026-07-27, scheduled autotrain-loop session)

Acting on the note above: 6 more rows, independently run for real against
`main` HEAD `f3adde1b` (DSH5-10 sixth slice, PR #1131, already merged),
varying `--seed` (1, 2, 3) and `--steps` (4, 16) jointly rather than one
knob at a time. Same fixture/model/recipe as every prior batch. Full
per-run numbers, the combined seed x steps grid, and the honest
result/scope notes: see
[joint seed x steps sweep](autotrain-wf-smoke-20260727-joint-seedvar-measured-results.md).

| run_id | steps | seed | stopped_on | last_loss | wall_s |
| --- | --- | --- | --- | --- | --- |
| `autotrain_wf_smoke_20260727_joint_seed1_steps4` | 4 | 1 | steps | 52.60234069824219 | 3.63 |
| `autotrain_wf_smoke_20260727_joint_seed2_steps4` | 4 | 2 | steps | 49.65409851074219 | 3.94 |
| `autotrain_wf_smoke_20260727_joint_seed3_steps4` | 4 | 3 | steps | 64.18412017822266 | 3.57 |
| `autotrain_wf_smoke_20260727_joint_seed1_steps16` | 16 | 1 | steps | 19.269824981689453 | 4.76 |
| `autotrain_wf_smoke_20260727_joint_seed2_steps16` | 16 | 2 | steps | 16.698604583740234 | 5.33 |
| `autotrain_wf_smoke_20260727_joint_seed3_steps16` | 16 | 3 | steps | 22.079015731811523 | 5.21 |

Total independently verified rows across this file: **25** (all prior
batches plus this joint sweep).

**Next steps note:** single-variable and now joint seed x steps variance are
both covered; the smoke-loop's role as a harness liveness check is
thoroughly exhausted at this fixture size. The next scheduled iteration
should move off this fixed fixture and pick up one of the repo's actually-
open threads — for DSH5-10 specifically, the real next step is locating and
wiring the `TypedOperatorPolicyScorer`
(`src/slm_training/harnesses/experiments/typed_operator_policy.py:316`) the
issue's own text names as the intended consumer, since PR #1131 found the
existing `PreferencePair` composite-reward shape is a structural mismatch
for exact-state action-token rows — or the next queued `AP-007+` campaign
arm.
67 changes: 67 additions & 0 deletions docs/design/autotrain-wf-smoke-20260727-joint-seedvar-measured-results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Joint seed x steps sweep (2026-07-27, scheduled autotrain-loop session)

**Honesty:** `fixture_or_scratch`. Not a ship claim.

Follow-up to the ledger's own "Next steps note" after batch #4: instead of
another single-variable check, this sweep varies `--seed` (1, 2, 3) *and*
`--steps` (4, 16) jointly against `main` HEAD `f3adde1b` (PR #1131, already
merged), same fixture/model/recipe as every prior batch:

```bash
python -m scripts.train_model \
--train-dir src/slm_training/resources/data/train/wf_smoke_v2 \
--model twotower --context-backend scratch --steps <4|16> \
--run-id <run_id> --no-sync-checkpoints --device cpu --seed <1|2|3>
```

Environment: fresh `.venv` (`python3.12 -m venv`, `pip install -e ".[dev,grammar]"`),
created in this scheduled session, not committed. Checked (not committed —
`outputs/` is gitignored) at `outputs/runs/<run_id>/train_summary.json`.

| run_id | steps | seed | stopped_on | last_loss | wall_s |
| --- | --- | --- | --- | --- | --- |
| `autotrain_wf_smoke_20260727_joint_seed1_steps4` | 4 | 1 | steps | 52.60234069824219 | 3.63 |
| `autotrain_wf_smoke_20260727_joint_seed2_steps4` | 4 | 2 | steps | 49.65409851074219 | 3.94 |
| `autotrain_wf_smoke_20260727_joint_seed3_steps4` | 4 | 3 | steps | 64.18412017822266 | 3.57 |
| `autotrain_wf_smoke_20260727_joint_seed1_steps16` | 16 | 1 | steps | 19.269824981689453 | 4.76 |
| `autotrain_wf_smoke_20260727_joint_seed2_steps16` | 16 | 2 | steps | 16.698604583740234 | 5.33 |
| `autotrain_wf_smoke_20260727_joint_seed3_steps16` | 16 | 3 | steps | 22.079015731811523 | 5.21 |

Combined with the already-verified single-variable rows (`seed=0` across
`steps` in {4, 8, 16, 32}; `steps=8` across `seed` in {0, 1, 2, 3}), the full
joint grid now covered is:

| steps \ seed | 0 | 1 | 2 | 3 |
| --- | --- | --- | --- | --- |
| 4 | 56.89 | 52.60 | 49.65 | 64.18 |
| 8 | 32.61 | 38.95 | 38.32 | 36.62 |
| 16 | 15.10 | 19.27 | 16.70 | 22.08 |
| 32 | 8.39 | - | - | - |

**Result:** at every step count, seed-to-seed spread is real but bounded
(steps=4: 49.65-64.18, spread 14.53; steps=16: 15.10-22.08, spread 6.98) and
shrinks in absolute terms as steps increase, while the seed-0 loss is not
systematically the lowest or highest at any step count (steps=4: seed 0 is
second-highest; steps=16: seed 0 is lowest) -- i.e. seed does not
introduce a directional bias, only variance, consistent with it only
touching model initialization on this from-scratch `twotower` config. This
is genuinely new joint evidence (not a repeat of either single-variable
batch): it's the first check that seed-variance and step-count both act on
the same fixture without cancelling or compounding unexpectedly.

Still `fixture_or_scratch`: n=1 per (seed, steps) cell, a 101-record
fixture, `context-backend scratch`. No convergence, generalization, ranking,
or ship claim is made -- this only characterizes how much this fixed
recipe's loss moves under its own declared knobs.

**Next steps note (carried forward):** the smoke-loop's role as a harness
liveness + single/joint-variable check is now thoroughly covered. Per the
ledger's own prior note, the next scheduled iteration should move off this
fixed fixture and pick up one of the repo's actually-open threads (the
DSH5-10 SFT/preference-training scope -- noting the sixth slice, PR #1131,
found the existing `PreferencePair` composite-reward shape is a structural
mismatch for exact-state action-token rows, so the real next step there is
locating and wiring the `TypedOperatorPolicyScorer`
(`src/slm_training/harnesses/experiments/typed_operator_policy.py:316`) the
issue's own text names as the intended consumer -- or the next queued
`AP-007+` campaign arm).
108 changes: 99 additions & 9 deletions docs/design/dsh5-10-replay-preference-rows.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DSH5-10: replay-grounded preference rows from undo/redo history (SLM-418)

**Status:** partial slice, in progress (fifth increment).
**Status:** partial slice, in progress (sixth increment).
**Claim class:** `wiring`.
**Honest verdict:** not yet dispositioned -- this PR extends a scoped
subset, not the full issue.
Expand All @@ -13,13 +13,18 @@ SFT/preference training against four context-view baselines, and (4)
held-out measurement of action/operator/argument/reference/branch accuracy,
calibration, and CAP0/CAP1/CAP2 retention.

This slice adds the seventh and final named pattern, **pronoun-focus-
followup** (see "Fifth slice (v6)" below), bringing extraction coverage to
7 of 7. All seven named patterns from the issue's own list now extract and
replay-verify. What remains is the issue's separate, still fully unattempted
training/measurement scope: SFT/preference training against the
DSH3-selected policy/control heads, the four-baseline comparison, held-out
benefit measurement, and turn-depth/context-view ablations.
The fifth slice added the seventh and final named pattern, **pronoun-focus-
followup**, bringing extraction coverage to 7 of 7 -- all seven named
patterns from the issue's own list now extract and replay-verify. This
slice (see "Sixth slice (v7)" below) takes the smallest next step of the
issue's still-unattempted scope: it converts an already-extracted row into
this repo's *existing* preference-training example format
(`PreferencePair`), so a future slice can feed rows into the existing
`slm preference` harness without inventing a second corpus shape. It does
**not** attempt SFT/preference training against the DSH3-selected
policy/control heads, the four-baseline comparison, held-out benefit
measurement, or turn-depth/context-view ablations -- all of that remains
exactly as unattempted as before this slice.

## What this PR delivers

Expand Down Expand Up @@ -288,10 +293,95 @@ training/evaluation work enumerated above.
snapshot is untouched, staying immutable point-in-time evidence from
before this slice landed.

## Sixth slice (v7)

* Added `preference_pair_from_replay_row` and `preference_pairs_from_trace`
to `src/slm_training/dsl/operators/replay_preference.py` -- pure
data-conversion functions, not another extraction pattern. They take an
already-extracted `OperatorReplayPreferenceRowV1` and materialize it into
`PreferencePair` (`slm_training.harnesses.preference`), the exact
`prompt`/`chosen`/`rejected`/`design_md`/`chosen_score`/`rejected_score`/
`meta` schema the existing `slm preference build-pairs`/`train` harness
(`.agents/skills/autotrain/references/preference.md`,
`src/slm_training/harnesses/preference/train.py`) already reads and
writes -- no second corpus shape is invented.
* **`prompt`** is `trace.node(row.input_state_id).state.source`: the exact,
already pack-authorized OpenUI source of the state the choice was made
from. This is the one piece of real context a row carries; a
`ConversationTraceV1`'s turns are AST operations, never user utterances,
so there is no natural-language instruction anywhere upstream to recover.
Using the state source as `prompt` is an honest, if unusual, reuse of the
field -- `pair.meta["schema"] == "operator_replay_preference_pair/v1"`
lets a caller tell these pairs apart from `build_pairs_from_candidates`
output, whose `prompt` is a design-task instruction. No instruction text
is fabricated.
* **`chosen`/`rejected`** are `row.chosen_action`/`row.rejected_action`
verbatim -- the row's own legal-set action tokens (e.g. `"undo"`,
`"checkout:<state>"`, `"merge:<pair>"`, or a serialized `OPERATOR <id>
..."` action). This slice deliberately does **not** attempt to replay the
rejected side to an alternate full OpenUI program: for an unchosen
operator action, no such program was ever materialized in the trace (only
the chosen action was actually applied), and reconstructing one would
require independently re-running `OperatorLibraryV1.apply` against a
legal-set candidate outside what extraction already computed -- real
additional work, out of scope for this slice's stated size. `chosen`/
`rejected` here are therefore two *action tokens* being preferred, not two
full-program renderings of one prompt the way `build_pairs_from_candidates`
pairs are; the doc says so explicitly rather than letting the shared field
names imply otherwise.
* `composite_reward` (and its `grammar_score`/`placeholder_score`/
`layout_metrics` components) is deliberately **never called** on these
tokens. Scoring `"undo"` or a serialized `OPERATOR ...` action as though it
were OpenUI source would silently manufacture a meaningless number (e.g.
`grammar_score` would legitimately return `0.0` for non-program text, but
that `0.0` would look like a real quality signal to a downstream reader).
`chosen_score`/`rejected_score` are therefore left at `PreferencePair`'s
own `0.0` default, and the honest distinction lives in `meta` instead.
* **Honest skip, not fabrication.** `preference_pairs_from_trace` looks up
each row's `input_state_id` via `trace.node(...)`; when it is not a member
of that trace's own `state_nodes` (the exact case for a `MERGE_SUCCESS`
row, which is grounded on a `BranchEditV1` tip rather than any single
trace -- see "Fourth slice (v5)" above), the row is skipped and recorded
in a separate `skipped` list (row + reason) rather than the function
fabricating a prompt or raising. `preference_pair_from_replay_row` also
refuses (`ValueError`) a blank/empty `input_state_source` rather than
emit a pair with a fabricated prompt.
`test_preference_pairs_from_trace_skips_a_row_whose_input_state_is_foreign_to_this_trace`
and
`test_preference_pair_from_replay_row_refuses_to_fabricate_a_blank_prompt`
(`tests/test_dsl/test_replay_preference.py`) prove both paths directly. A
`MERGE_SUCCESS` row itself still converts correctly --
`test_preference_pair_from_replay_row_converts_merge_success_row` calls
`preference_pair_from_replay_row` directly with
`left.output_node.state.source` supplied explicitly, exactly as the
function's docstring describes for that case.
* Six more regression tests cover one row of each of the other six
extracted relation types converting correctly end to end through
`preference_pairs_from_trace`: `EDIT_THEN_UNDO`/`PARTIAL_ROLLBACK` (one
trace produces both), `UNDO_THEN_REDO`, `CHECKOUT_ANOTHER_STATE`,
`FORK_THEN_CHOOSE_ONE_BRANCH`, and `PRONOUN_FOCUS_FOLLOWUP` -- 7 of 7
named patterns are exercised by this slice's tests (six via the trace
path, one, `MERGE_SUCCESS`, via the direct path).
* **Explicitly still not attempted:** no `PreferencePair` produced by this
slice has been written to a corpus file, fed to `slm preference
build-pairs`/`train`, or trained against; no composite-reward score is
computed for any row; no replay of a rejected operator action to an
alternate rendering is attempted; the DSH3-selected policy/control heads
(`TypedOperatorPolicyScorer`) and `ObjectiveView`/`DecisionStateV2`
materializers in `decision_events_v2.py` are untouched. This is
data-conversion wiring only, same honesty tier as every prior slice in
this doc.
* `dsl.operators.replay_preference` bumped v6 -> v7 in
`src/slm_training/resources/versions.json`; `dsl.operators.contracts`
gets a `no-bump:` history entry for the two new re-exports from
`operators/__init__.py`.

## Reproducibility

```bash
NODE_OPTIONS= pytest -q tests/test_dsl/test_replay_preference.py tests/test_dsl/test_operator_merge.py tests/test_dsl/test_operator_conversation.py tests/test_evals/test_advanced_operator_disposition.py tests/test_scripts/test_validate_advanced_operator_disposition.py
```

Result (this PR, real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean.
Result (fifth slice, v6): real run in a fresh `.venv` -- Python 3.12, `pip install -e ".[dev,grammar]"`, plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` for the G2/G8 schema-oracle gates the pack authority requires; the ambient `--import tsx` `NODE_OPTIONS` is rejected by this Node 22 build both for `npm ci` and for `pytest`, unrelated to this change): `61 passed`. Also verified: `ruff check` clean on every changed file; `python -m scripts.verify_version_stamps --check --base origin/claude/great-dirac-v82ph9` -- `ok (2 component(s) touched)`; `python -m scripts.repo_policy` -- `ok`; `python -m scripts.verify_decode_invariants` -- clean.

Result (this PR, sixth slice, v7): same command, same freshly built `.venv` (Python 3.12) plus `NODE_OPTIONS= npm ci` in `src/apps/openui_bridge` -- test file grew by 8 tests (7 conversion-coverage tests + 1 refusal test) in the same `tests/test_dsl/test_replay_preference.py`, no new test module needed: `69 passed`. Also verified: `ruff check` on every changed file (`src/slm_training/dsl/operators/replay_preference.py`, `src/slm_training/dsl/operators/__init__.py`, `tests/test_dsl/test_replay_preference.py`) -- `All checks passed!`; `python -m scripts.verify_version_stamps --check --base origin/main` -- `ok (vs 5f94b925a121; 4 changed file(s), 2 component(s) touched)`; `python -m scripts.repo_policy` -- `repo-policy: ok (tracked + untracked)`; `python -m scripts.verify_decode_invariants` -- exits clean (`0`).
4 changes: 4 additions & 0 deletions src/slm_training/dsl/operators/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,8 @@
ReplayPreferenceRelation,
extract_merge_preference_row,
extract_replay_preference_rows,
preference_pair_from_replay_row,
preference_pairs_from_trace,
)
from slm_training.dsl.operators.topology import (
CONTRACT_SUBTREE,
Expand Down Expand Up @@ -457,6 +459,8 @@
"prepare_operator_action",
"prepare_operator_transaction",
"preference_cost",
"preference_pair_from_replay_row",
"preference_pairs_from_trace",
"redo_conversation",
"replay_collapsed_instruction",
"replay_conversation_trace",
Expand Down
Loading
Loading