Skip to content

Ship track_locality.py: reproducible blacklist + whitelist track-locality measurement (#11) - #33

Merged
sepehrsadri merged 1 commit into
mainfrom
experiments/track-locality-issue11
Aug 8, 2026
Merged

Ship track_locality.py: reproducible blacklist + whitelist track-locality measurement (#11)#33
sepehrsadri merged 1 commit into
mainfrom
experiments/track-locality-issue11

Conversation

@sepehrsadri

Copy link
Copy Markdown
Contributor

Progress on #11.

What this does

  1. Adds experiments/track_locality.py, implementing both approaches the issue asks
    for, reported side by side:
    • Blacklist — hash each track's whole XML subtree, dropping a frozen, named,
      commented exclusion set (TRACK_BLACKLIST_ELEMENTS / TRACK_BLACKLIST_ATTRS). Every
      entry has a one-line reason and, where relevant, a pointer to which observed diff
      motivated it.
    • Whitelist (the "bonus, and better" ask, and AGENTS.md's own stated preference) —
      extract only the fields that define "the track" (name, volume, pan, on/off, output
      routing, device chain, clip sample/position/mute) and hash that instead.
    • --chain 'Backup/*.als' support, reusing als_semantic_diff.py's exact
      autosave-lineage-grouping convention (the AUTOSAVE_NAME regex and split-by-stem
      logic) rather than inventing a second one.
  2. Runs it against a real local Ableton chain and reports the actual output — see
    "Measurement" below.
  3. Updates docs/EXPERIMENTS.md §3 with that output, removes the "provisional" warning,
    and is explicit that this is a new measurement on smaller, different material than
    whatever produced the retracted "median 3" figure.
  4. Adds tests/test_track_locality.py, matching this repo's convention of a synthetic-
    fixture test file per experiments/ script. It pins: identity (no false positives),
    real content changes caught by both methods, view-state/warp-marker noise caught by
    neither, and — the one I actually expected might fail before writing it — that editing
    one track does not falsely flag an untouched track just because the edit shifted a
    positional id counter (AutomationTarget/ModulationTarget) elsewhere in the file.
    All 7 passed on first run, which is the exclusion set doing its job on synthetic
    material as well as real.

Measurement

Sample: 10 consecutive autosaves (9 pairs) of a 28-track Ableton Live set, one local
chain, this environment.

Method Median Range Per-pair counts (of 28)
Blacklist (frozen exclusion set) 9 0–15 5, 9, 4, 0, 15, 1, 11, 11, 12
Whitelist (named fields) 2 0–4 4, 3, 4, 0, 4, 0, 2, 0, 0

The two disagree by >4x on this material too, which is the whole point of shipping both:
on two of the nine pairs, als_semantic_diff.py independently reports "no musical change
detected", the whitelist agrees (0 of 28 changed on both), and the blacklist reports 11
and 12 changed tracks on those same two pairs — pure residual churn (chiefly warp-marker
re-analysis and FileRef/AuPreset/AutomationTarget id renumbering the exclusion set
doesn't fully catch). That is the exact "blacklists leak" warning in AGENTS.md,
reproduced on new material rather than just asserted.

The qualitative claim survives either way — even the less flattering, blacklist-inflated
median (9 of 28) still means two-thirds of a typical save is untouched.

Reproduce:

python3 experiments/track_locality.py --chain 'path/to/YourProject/Backup/*.als'

What this PR deliberately does NOT do — and why I'm not asking to close #11

The issue's number (median 3 of 28, provisional) and the "independent re-implementation"
table in the issue body both trace back to the original 26 GB / 30-project corpus,
which lived on a different machine and is not available in this environment. I have a
different, smaller, local chain (28 tracks, 9 pairs, one lineage) — genuinely useful for
validating the method, but not the same material, and not large enough to stand in as
"the" corrected headline number for a document that explicitly wants figures backed by
real, full-scale material (see the Material table at the top of EXPERIMENTS.md).

So this PR:

  • retracts the old "median 3" figure as unreproducible (not silently — the doc says so
    and says why),
  • publishes an honestly-scoped new number instead of a bigger, unverifiable claim,
  • ships the script + frozen exclusion set the issue actually asked for, so whoever does
    have access to the original corpus (or a comparably-sized one) can now get a real
    answer with one command instead of reverse-engineering an exclusion set from prose.

Recommendation: keep #11 open, re-scoped (or split into a follow-up) to "run
track_locality.py against the original/a comparably-sized corpus and publish that
number." I don't have that material here, and inventing a number to close the issue
would repeat exactly the mistake #11 was filed over.

Verification

  • ~/.cargo/bin/uvx ruff check experiments/ tests/ — clean
  • python3 .github/workflows/scripts/check_personal_paths.py — clean
  • python3 -m pytest tests/ -q — 293 passed, 13 loudly-skipped (opt-in real-fixture
    tests, expected in this environment)

🤖 Generated with Claude Code

…ty measurement

Progress on #11.

docs/EXPERIMENTS.md Sec 3 published "median 3 of 28 tracks change per save"
with no script implementing it, and the number was acknowledged provisional
because it is highly sensitive to an unpublished exclusion set.

- experiments/track_locality.py hashes each track's XML subtree two ways per
  save pair and reports both, side by side:
  - blacklist: hash everything, drop a frozen, named, commented exclusion
    set (TRACK_BLACKLIST_ELEMENTS / TRACK_BLACKLIST_ATTRS)
  - whitelist: extract only the fields that define "the track" (name, mix
    state, device chain, clip positions/samples) and hash that, per the
    whitelist-over-blacklist convention in AGENTS.md
  - --chain 'Backup/*.als' support, reusing als_semantic_diff.py's exact
    autosave-lineage-grouping convention (AUTOSAVE_NAME / split-by-stem)
    rather than reinventing it

- Run against a local chain of 10 consecutive autosaves (9 pairs) of a
  28-track Ableton Live set: blacklist median 9 (range 0-15), whitelist
  median 2 (range 0-4). The two disagree by >4x on this material too -- on
  two pairs als_semantic_diff.py independently reports "no musical change",
  the whitelist agrees (0 changed), and the blacklist reports 11 and 12
  changed tracks purely from residual churn (warp-marker re-analysis,
  FileRef/AuPreset/AutomationTarget id renumbering). That is the exact leak
  AGENTS.md warns about, reproduced on new material.

- docs/EXPERIMENTS.md Sec 3 updated with both numbers, the provisional
  warning removed, and an explicit statement that this is a new,
  independent measurement on different (much smaller) material than
  whatever produced the original figure -- the original corpus is not
  available in this environment, so no reproduction claim is made either
  way. The old "median 3" figure is retracted as unreproducible rather than
  quietly replaced.

- tests/test_track_locality.py: identity, mixer-field and clip-move changes
  caught by both methods, view-state and warp-marker-only changes caught by
  neither, and the specific bug TRACK_BLACKLIST_ATTRS exists to avoid --
  editing one track must not falsely flag an untouched track just because
  editing upstream shifted a positional id counter elsewhere in the file.

- experiments/README.md and tests/README.md updated with the new script/test.

Verified: ruff check clean, check_personal_paths.py clean, 293 passed / 13
loudly-skipped (opt-in real-fixture tests) via `python3 -m pytest tests/ -q`.

Not done here: the original 26 GB / 30-project corpus that produced the
retracted "median 3" figure is not available in this environment, so this
PR cannot correct that number at its source -- only retract it and replace
it with a new, honestly-scoped measurement. See PR description.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sepehrsadri
sepehrsadri merged commit a5b8559 into main Aug 8, 2026
14 checks passed
@sepehrsadri
sepehrsadri deleted the experiments/track-locality-issue11 branch August 8, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the track-locality number reproducible (or correct it)

1 participant