Ship track_locality.py: reproducible blacklist + whitelist track-locality measurement (#11) - #33
Merged
Merged
Conversation
…ty measurement Progress on #11. docs/EXPERIMENTS.md Sec 3 published "median 3 of 28 tracks change per save" with no script implementing it, and the number was acknowledged provisional because it is highly sensitive to an unpublished exclusion set. - experiments/track_locality.py hashes each track's XML subtree two ways per save pair and reports both, side by side: - blacklist: hash everything, drop a frozen, named, commented exclusion set (TRACK_BLACKLIST_ELEMENTS / TRACK_BLACKLIST_ATTRS) - whitelist: extract only the fields that define "the track" (name, mix state, device chain, clip positions/samples) and hash that, per the whitelist-over-blacklist convention in AGENTS.md - --chain 'Backup/*.als' support, reusing als_semantic_diff.py's exact autosave-lineage-grouping convention (AUTOSAVE_NAME / split-by-stem) rather than reinventing it - Run against a local chain of 10 consecutive autosaves (9 pairs) of a 28-track Ableton Live set: blacklist median 9 (range 0-15), whitelist median 2 (range 0-4). The two disagree by >4x on this material too -- on two pairs als_semantic_diff.py independently reports "no musical change", the whitelist agrees (0 changed), and the blacklist reports 11 and 12 changed tracks purely from residual churn (warp-marker re-analysis, FileRef/AuPreset/AutomationTarget id renumbering). That is the exact leak AGENTS.md warns about, reproduced on new material. - docs/EXPERIMENTS.md Sec 3 updated with both numbers, the provisional warning removed, and an explicit statement that this is a new, independent measurement on different (much smaller) material than whatever produced the original figure -- the original corpus is not available in this environment, so no reproduction claim is made either way. The old "median 3" figure is retracted as unreproducible rather than quietly replaced. - tests/test_track_locality.py: identity, mixer-field and clip-move changes caught by both methods, view-state and warp-marker-only changes caught by neither, and the specific bug TRACK_BLACKLIST_ATTRS exists to avoid -- editing one track must not falsely flag an untouched track just because editing upstream shifted a positional id counter elsewhere in the file. - experiments/README.md and tests/README.md updated with the new script/test. Verified: ruff check clean, check_personal_paths.py clean, 293 passed / 13 loudly-skipped (opt-in real-fixture tests) via `python3 -m pytest tests/ -q`. Not done here: the original 26 GB / 30-project corpus that produced the retracted "median 3" figure is not available in this environment, so this PR cannot correct that number at its source -- only retract it and replace it with a new, honestly-scoped measurement. See PR description. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Progress on #11.
What this does
experiments/track_locality.py, implementing both approaches the issue asksfor, reported side by side:
commented exclusion set (
TRACK_BLACKLIST_ELEMENTS/TRACK_BLACKLIST_ATTRS). Everyentry has a one-line reason and, where relevant, a pointer to which observed diff
motivated it.
extract only the fields that define "the track" (name, volume, pan, on/off, output
routing, device chain, clip sample/position/mute) and hash that instead.
--chain 'Backup/*.als'support, reusingals_semantic_diff.py's exactautosave-lineage-grouping convention (the
AUTOSAVE_NAMEregex and split-by-stemlogic) rather than inventing a second one.
"Measurement" below.
docs/EXPERIMENTS.md§3 with that output, removes the "provisional" warning,and is explicit that this is a new measurement on smaller, different material than
whatever produced the retracted "median 3" figure.
tests/test_track_locality.py, matching this repo's convention of a synthetic-fixture test file per
experiments/script. It pins: identity (no false positives),real content changes caught by both methods, view-state/warp-marker noise caught by
neither, and — the one I actually expected might fail before writing it — that editing
one track does not falsely flag an untouched track just because the edit shifted a
positional id counter (
AutomationTarget/ModulationTarget) elsewhere in the file.All 7 passed on first run, which is the exclusion set doing its job on synthetic
material as well as real.
Measurement
Sample: 10 consecutive autosaves (9 pairs) of a 28-track Ableton Live set, one local
chain, this environment.
The two disagree by >4x on this material too, which is the whole point of shipping both:
on two of the nine pairs,
als_semantic_diff.pyindependently reports "no musical changedetected", the whitelist agrees (0 of 28 changed on both), and the blacklist reports 11
and 12 changed tracks on those same two pairs — pure residual churn (chiefly warp-marker
re-analysis and
FileRef/AuPreset/AutomationTargetid renumbering the exclusion setdoesn't fully catch). That is the exact "blacklists leak" warning in AGENTS.md,
reproduced on new material rather than just asserted.
The qualitative claim survives either way — even the less flattering, blacklist-inflated
median (9 of 28) still means two-thirds of a typical save is untouched.
Reproduce:
python3 experiments/track_locality.py --chain 'path/to/YourProject/Backup/*.als'What this PR deliberately does NOT do — and why I'm not asking to close #11
The issue's number (median 3 of 28, provisional) and the "independent re-implementation"
table in the issue body both trace back to the original 26 GB / 30-project corpus,
which lived on a different machine and is not available in this environment. I have a
different, smaller, local chain (28 tracks, 9 pairs, one lineage) — genuinely useful for
validating the method, but not the same material, and not large enough to stand in as
"the" corrected headline number for a document that explicitly wants figures backed by
real, full-scale material (see the Material table at the top of EXPERIMENTS.md).
So this PR:
and says why),
have access to the original corpus (or a comparably-sized one) can now get a real
answer with one command instead of reverse-engineering an exclusion set from prose.
Recommendation: keep #11 open, re-scoped (or split into a follow-up) to "run
track_locality.pyagainst the original/a comparably-sized corpus and publish thatnumber." I don't have that material here, and inventing a number to close the issue
would repeat exactly the mistake #11 was filed over.
Verification
~/.cargo/bin/uvx ruff check experiments/ tests/— cleanpython3 .github/workflows/scripts/check_personal_paths.py— cleanpython3 -m pytest tests/ -q— 293 passed, 13 loudly-skipped (opt-in real-fixturetests, expected in this environment)
🤖 Generated with Claude Code