From 79f23f66f837484f027f5d757f5d973a0dc62e56 Mon Sep 17 00:00:00 2001 From: "sepehr.sadri" Date: Sun, 16 Aug 2026 14:16:09 +0200 Subject: [PATCH] =?UTF-8?q?M2.5:=20the=20reality=20gate=20passes=20?= =?UTF-8?q?=E2=80=94=2033.0%=20empty=20verdicts=20on=2032=20real=20Logic?= =?UTF-8?q?=20projects?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue #15's exit criterion was a decision, not a tool: if >50% of real Logic saves show no visible structural change, stop and map the volume-fader field before building M5, because no UI copy survives a dominant "nothing changed" first impression. The tool shipped in #32; the measurement needed a real library and never ran. It has now run. Measured 2026-08-16, 32 projects / 28 GB / 100 consecutive save pairs: 67.0% of save pairs show a structural change Wit can see (67 of 100) 33 pairs show none; 28 pairs (28.0%) are byte-different but structurally identical 33.0% is below the 50% threshold, so the gate passes and M5 proceeds — #18 is unblocked. Every ProjectData in the library walked cleanly (no read-error line), so all 100 pairs are real comparisons. The distribution is long-tailed rather than bimodal: the mode above zero is a single change, 26 pairs sit in the 1-7 band, and a thin tail runs to 235. The M2 volume-fader stretch goal stays worth doing — a save that moved only a fader is still in that 33% — it just no longer has to come first. Also closes M3's flagged `wit dupes` follow-up, which needed the full library: the Rust implementation reports 5.5 GB of 23.9 GB (23.2%) against §9's independent Python walk at 5.38 GB of 21.95 GB (24.5%), on a library that gained ~2 GB of audio in between. Recorded as a cross-check in §9 rather than as a rewrite of that finding, since they are different tools on different snapshots. Fixes a units bug found while reconciling those two numbers: `human_bytes` divided by 1024 while printing decimal labels ("GB"), understating its own totals by 7.4% and making `wit dupes` output silently incomparable to EXPERIMENTS.md, which quotes decimal GB throughout. Now decimal, with a test pinning the boundary. Verification (real material, opt-in — CI cannot reproduce any of it): $ WIT_LOGIC_LIBRARY=~/Music/Logic cargo test -p wit-index \ --test real_fixtures -- --nocapture --ignored test real_library_reports_the_three_m2_5_statistics ... ok cargo test --workspace 127 passed, 0 failed python3 -m pytest tests/ 293 passed, 13 skipped cargo fmt --check / clippy -D warnings / cargo deny clean Closes #15. Unblocks #18. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 12 ++++ README.md | 11 ++++ crates/wit-cli/src/main.rs | 33 +++++++++- docs/EXPERIMENTS.md | 130 ++++++++++++++++++++++++++++--------- docs/ROADMAP.md | 6 +- 5 files changed, 158 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b496a7..6b0f3ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,10 @@ Building the 0.0 pilot — no released artifact yet. policy, issue/PR templates, CI ### Fixed +- `wit dupes` formatted byte counts with a 1024-based divisor while printing decimal unit + labels ("GB"), understating its own totals by 7.4% and making its output silently + incomparable to `docs/EXPERIMENTS.md`, which quotes decimal GB throughout. Now decimal, + with a test pinning the boundary - All 13 bugs documented as `xfail` in the Python prototype test suite (21 xfail markers total), spanning robustness (billion-laughs expansion, unbounded tree allocation, quadratic varint DoS in `flp_parse`, a leaked file descriptor, an oversized-payload @@ -51,6 +55,14 @@ Building the 0.0 pilot — no released artifact yet. `EXPERIMENTS.md` §6a) — [PR #28](https://github.com/sep-lab/Wit/pull/28), closes #10 ### Findings that shaped the design +- **M2.5 reality gate — the Logic empty-verdict rate, measured at scale and passed.** + Across 32 real Logic projects (28 GB, 100 consecutive save pairs), **33.0%** of save + pairs show no structural change Wit can see — against issue #15's >50% threshold for + stopping M5 in favour of mapping Logic's volume-fader field first. **M5 proceeds** + ([#18](https://github.com/sep-lab/Wit/issues/18) unblocked). 28.0% of pairs are + byte-different but structurally identical, reconfirming at library scale M2's finding + that byte comparison is useless on this format — see + [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md) §11, closes #15 - Byte-level versioning of rendered audio is not viable: a global gain change leaves **0.00%** of bytes reusable, and costs git a full extra copy - Delta chains beat content-defined chunking **29×** on project-file history diff --git a/README.md b/README.md index ff7b609..fb8a08c 100644 --- a/README.md +++ b/README.md @@ -168,6 +168,17 @@ which is an argument for shipping it early, not evidence that version control is delivered it. (An earlier figure of 8.1 GB / 3.2× was wrong: it applied FLAC to 7.4 GB of `.caf` that is already ALAC-compressed.) +You can reproduce the duplicate half of that on your own library in about half a minute — +it is read-only, prints basenames only, and there is no delete button: + +``` +cargo run --release -p wit-cli -- dupes /path/to/YourLibrary +``` + +On the library above, since grown to 32 projects, that reports **5.5 GB of 23.9 GB (23.2%)** +— an independent Rust reimplementation agreeing with the Python analysis +([EXPERIMENTS.md §9](docs/EXPERIMENTS.md)). + ## How it works A music project is already a program: immutable source recordings plus a tree of diff --git a/crates/wit-cli/src/main.rs b/crates/wit-cli/src/main.rs index 699174d..497d47f 100644 --- a/crates/wit-cli/src/main.rs +++ b/crates/wit-cli/src/main.rs @@ -339,12 +339,21 @@ fn dupes(path: &std::path::Path) -> ExitCode { ExitCode::SUCCESS } +/// Format a byte count in **decimal** units — 1 GB = 1,000,000,000 bytes. +/// +/// This is deliberately not the 1024-based convention. Every published +/// figure in `docs/EXPERIMENTS.md` is decimal GB (§9 says so explicitly), +/// and `wit dupes` output is meant to be directly comparable to it — a +/// user pasting this tool's number into a Measurement issue (the ask in +/// [#4]) must be quoting the same unit the docs quote. Dividing by 1024 +/// while printing "GB" understated the library total by 7.4% and made the +/// two numbers silently incomparable. fn human_bytes(bytes: u64) -> String { const UNITS: &[&str] = &["B", "KB", "MB", "GB", "TB"]; let mut size = bytes as f64; let mut unit = 0; - while size >= 1024.0 && unit < UNITS.len() - 1 { - size /= 1024.0; + while size >= 1000.0 && unit < UNITS.len() - 1 { + size /= 1000.0; unit += 1; } if unit == 0 { @@ -415,3 +424,23 @@ fn logic_report(path: &std::path::Path) -> ExitCode { print!("{out}"); ExitCode::SUCCESS } + +#[cfg(test)] +mod tests { + use super::human_bytes; + + #[test] + fn byte_counts_are_formatted_in_decimal_units_not_binary() { + // The boundary that matters: 1000 B is 1.0 KB, and 1024 B is also + // 1.0 KB rather than the binary convention's "1.0 KiB". + assert_eq!(human_bytes(999), "999 B"); + assert_eq!(human_bytes(1_000), "1.0 KB"); + assert_eq!(human_bytes(1_024), "1.0 KB"); + // The unit the published numbers are actually quoted in. A 1024-based + // divisor would render this as "20.7 GB" — a 7.4% understatement, and + // the exact discrepancy that made `wit dupes` output incomparable to + // EXPERIMENTS.md §9. + assert_eq!(human_bytes(22_200_000_000), "22.2 GB"); + assert_eq!(human_bytes(0), "0 B"); + } +} diff --git a/docs/EXPERIMENTS.md b/docs/EXPERIMENTS.md index 933b87f..c11e72a 100644 --- a/docs/EXPERIMENTS.md +++ b/docs/EXPERIMENTS.md @@ -11,7 +11,7 @@ fixtures: | `Artefakt - Undertow` (Ableton) | A commercial-grade Live 12 set, 28 tracks, 688 audio clips, 5,112 warp markers. **30 sequential autosaves** of the same project spanning Feb–May 2026. | | `You make my crazy!` (Logic) | A real Logic 12 project, 459 MB, 33 audio files. **10 sequential saves** (`Project File Backups/00..08` + current). | | `Aston Martin Music Remake` (FL Studio) | A real FL 10-era `.flp`, 136 KB, 18 channels. | -| A 30-project Logic library | 26 GB across 30 real projects — used for the duplication analysis. | +| A real Logic library | One person's working library, used for the duplication analysis (§9) and the M2.5 reality gate (§11). It grows: **30 projects / 26 GB** when §9 walked it, **32 projects / 28 GB** at §11's 2026-08-16 run. Each section states the snapshot it measured. | Labels used throughout: **measured** (we ran it), **cited** (someone else's result, with a source), **inferred** (reasoning, not measurement). @@ -538,6 +538,28 @@ exist today**. > it would work today. That is an argument for shipping that part first, not evidence that > version control produced it. +**Independent cross-check — measured, 2026-08-16.** The figures above come from the Python +walk on the 30-project / 26 GB snapshot. `wit dupes` (the Rust reimplementation shipped in +M3, `wit-index::duplicate_report`) was run against the same library after it had grown to 32 +projects: + +``` +$ wit dupes "/path/to/YourLibrary" + found 5.5 GB of duplicate audio (23.2% of 23.9 GB scanned) +``` + +**5.5 GB of 23.9 GB (23.2%)**, against Python's 5.38 GB of 21.95 GB (24.5%) — the two +implementations agree, on a library that gained ~2 GB of audio in between, with the duplicate +share drifting down about a point. This closes M3's flagged follow-up (the ROADMAP noted the +`wit dupes` verification had only ever run against a deliberately-planted duplicate, never the +full library). The top duplicate groups are the same `Save As` triples §9 named — three copies +each of `Vintage Upright Piano_*_consolidated.caf` at ~300 MB apiece. + +Both numbers are decimal GB. `wit dupes` divided by 1024 while printing "GB" until this +change, which understated its own totals by 7.4% and made its output silently incomparable to +this section; it now formats in decimal units, with a test pinning the boundary +(`crates/wit-cli/src/main.rs`). + **Limits.** Exact-duplicate detection only; sub-file chunk dedup would find more ([issue #6](https://github.com/sep-lab/Wit/issues/6)). The FLAC ratio is extrapolated from a 10-file random sample, so treat 1.8× as approximate. Figures are decimal GB throughout; @@ -599,40 +621,90 @@ consecutive pair with `wit_logic::walk` and reports, per pair: `semantic_equal`' extracted name, or a tempo change; `0` exactly when the verdict is `NoStructuralChange`), and raw byte identity. -**Result — measured, n = 1 project.** +**Result — measured, n = 32 projects, 2026-08-16.** ``` $ wit logic-report "/path/to/YourLibrary" - scanned 1 project(s), 1 alternative(s), 9 consecutive save pair(s) - 44.4% of save pairs show a structural change Wit can see (4 of 9) + scanned 32 project(s), 32 alternative(s), 100 consecutive save pair(s) + 67.0% of save pairs show a structural change Wit can see (67 of 100) distribution of change counts per save pair (0 = no visible structural change): - 0 change(s): 5 pair(s) - 1 change(s): 1 pair(s) - 2 change(s): 1 pair(s) - 14 change(s): 1 pair(s) - 17 change(s): 1 pair(s) - 5 pair(s) (55.6%) are byte-different but structurally identical + 0 change(s): 33 pair(s) + 1 change(s): 12 pair(s) + 2 change(s): 4 pair(s) + 3 change(s): 6 pair(s) + 4 change(s): 1 pair(s) + 5 change(s): 1 pair(s) + 6 change(s): 3 pair(s) + 7 change(s): 4 pair(s) + 9 change(s): 1 pair(s) + 11 change(s): 3 pair(s) + 13 change(s): 2 pair(s) + 14 change(s): 2 pair(s) + 15 change(s): 2 pair(s) + 16 change(s): 1 pair(s) + 17 change(s): 1 pair(s) + 18 change(s): 1 pair(s) + 19 change(s): 1 pair(s) + 25 change(s): 2 pair(s) + 26 change(s): 1 pair(s) + 29 change(s): 1 pair(s) + 30 change(s): 1 pair(s) + 33 change(s): 1 pair(s) + 36 change(s): 1 pair(s) + 37 change(s): 1 pair(s) + 44 change(s): 1 pair(s) + 47 change(s): 1 pair(s) + 56 change(s): 1 pair(s) + 63 change(s): 1 pair(s) + 65 change(s): 1 pair(s) + 67 change(s): 1 pair(s) + 68 change(s): 1 pair(s) + 71 change(s): 1 pair(s) + 76 change(s): 1 pair(s) + 93 change(s): 1 pair(s) + 116 change(s): 2 pair(s) + 145 change(s): 1 pair(s) + 235 change(s): 1 pair(s) + 28 pair(s) (28.0%) are byte-different but structurally identical ``` -Run against `You make my crazy!` (§0's fixture table) — the same 10-save chain M2 used. -5 of 9 pairs show no structural change (**55.6%**), and every one of those 5 is also -byte-different from its neighbor (the M2 finding that raw byte comparison is useless on -this format, reconfirmed: 100% of the no-visible-change pairs would look "changed" to a -byte diff). Every pair with a nonzero `change_count` was correctly called -`StructuralChange`, and every zero-count pair was correctly called `NoStructuralChange` — -`change_count` and the verdict never disagreed, which is expected since both derive from -the same census/extracted equality (`wit-logic/src/lib.rs`), but is a useful sanity check -on the new counting logic itself. - -**Limits — this is a tool delivery, not the finding issue #15 asks for.** The issue's own -exit criterion is a decision at **30 projects, 26 GB** — whether >50% of real saves show -no visible structural change. **n = 1 project cannot answer that question either way**; -55.6% on one project's one alternative is a data point, not a verdict on a library. This -environment does not have the 30-project/26 GB library (`daw-vcs-adoption-evidence` -memory) the issue was scoped against — only the one `.logicx` bundle in the fixture table -above. The exit-criterion decision (M5 vs. the M2 stretch goal — mapping Logic's -volume-fader field) is **not resolved by this entry** and still requires running the -command below against the real library. +Run against the full library in §0's fixture table (32 projects, 28 GB — it has grown by +two projects since the §9 duplication analysis walked it). Every `ProjectData` in the +library walked cleanly: the report printed no read-error line, so all 100 pairs are real +comparisons rather than partial ones. + +**The three statistics issue #15 asked for:** + +- **33 of 100 pairs (33.0%) show no structural change Wit can see.** The complement — 67.0% + — do. +- **The distribution is long-tailed, not bimodal.** The mode above zero is a single change + (12 pairs), 26 pairs sit in the 1–7 band, and a thin tail runs out to 235. So the common + case is not "nothing" and not "everything"; it is a handful of visible changes per save. +- **28 pairs (28.0%) are byte-different but structurally identical** — 28 of the 33 + empty-verdict pairs. The M2 finding holds at library scale: a byte diff would report a + change on 28% of all saves where there is nothing structural to show. The remaining 5 + empty pairs were byte-identical too, so *every* pair the byte comparison called changed + and the walker called empty is accounted for. + +**The decision this triggers.** Issue #15's exit criterion was: if **>50%** of real saves +show no visible structural change, do not start M5 — spend that time on the M2 stretch +goal (mapping Logic's volume-fader field) instead, because no UI copy survives a dominant +"nothing changed" first impression. **Measured 33.0%, so the gate passes and M5 proceeds** +([#18](https://github.com/sep-lab/Wit/issues/18) is no longer blocked). + +For calibration against the other format: Ableton's semantically-empty rate is 24% (§1). +Logic's 33% is higher, as expected from the Structure tier's narrower vision, but it is not +the majority outcome the gate was written to catch. + +**Limits.** One person's library, one machine — the same acknowledged breadth weakness every +number in this document carries, and exactly what [issue #4](https://github.com/sep-lab/Wit/issues/4) +exists to fix. It is also a library that has never been used with a version-control tool, so +its save cadence reflects Logic's own backup behaviour rather than deliberate committing. The +33% is a rate over *save pairs*, not over sessions: a project with 20 backups contributes 19 +pairs and weighs more than a project with 2. And "no structural change Wit can see" remains a +statement about the Structure honesty tier, not about the music — a save that moved only a +fader is in that 33% and is a real musical change. That gap is what the stretch goal closes, +and it is still worth doing; the gate result means it does not have to come *before* M5. **Reproduce (opt-in, real material, never committed):** diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 5c34444..1524d4c 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -46,10 +46,10 @@ understand — without ever risking a project file.** | M0 ✅ | [PR #13](https://github.com/sep-lab/Wit/pull/13) | ADR-0006, Rust workspace scaffolding, CI (`rust` + `licenses` jobs) | Landed | | M1 ✅ | [PR #24](https://github.com/sep-lab/Wit/pull/24) | Ableton `.als` parity port (`wit-als`) — Rust port of the working Python differ | Landed — golden byte-for-byte (`crates/wit-diff/tests/golden.rs`); spot-checked against a real 29-save `Backup/` chain via `WIT_FIXTURES` (28 pairs parse clean, no panics; 2 pairs cross-checked line-for-line against Python's actual output, exact match). The formal corpus-agreement gate against the specific 7 zero-change / 3 knob-only pairs named in `wit-planning/PLAN.md` has not been re-run — flagged for a follow-up pass | | M2 ✅ | [PR #25](https://github.com/sep-lab/Wit/pull/25) | Logic/GarageBand `ProjectData` walker (`wit-logic`) | Landed — spot-checked against a real `.logicx` project + its 9 on-disk `Project File Backups` (10 files, clean EOF on all, tempo matches `MetaData.plist` exactly on all); 5 of 9 real consecutive pairs correctly report `NoStructuralChange` despite differing bytes. The 30-fixture `jonkubis/LogicProFormatWriter` corpus fetch (pinned SHA `1f77c5c37d49ccd9551cc8e9107750e8db2f1fed`) has not been run — network-gated, opt-in, flagged for a follow-up pass | -| M2.5 | [#15](https://github.com/sep-lab/Wit/issues/15) | **Reality gate** — measure the empty-verdict rate on a real Logic library | **Tool landed, finding still open.** `wit logic-report ` (`wit-index::logic_report`) now walks every alternative's backup chain across a whole library and publishes the three statistics the issue asks for — see EXPERIMENTS.md §11. Run against the one real `.logicx` project available in this environment (n=1, 9 pairs): 55.6% show no visible structural change. That is a data point, not the issue's own >50%-of-30-projects decision — the real 26 GB/30-project run (`WIT_LOGIC_LIBRARY=/path/to/YourLibrary cargo test -p wit-index --test real_fixtures -- --nocapture --ignored`) still needs to happen before M5-vs-stretch-goal is decided | -| M3 ✅ | [PR #29](https://github.com/sep-lab/Wit/pull/29) | Index, content-addressed store, CLI (`wit-index`, `wit-cli`) — covers Logic, GarageBand, and Ableton discovery | Landed — `wit scan` verified against real local Ableton (5 lineages / 30 versions, exact match) and Logic (10 versions) libraries, rescans idempotent through the actual CLI. `wit dupes` verified correct against a deliberate real duplicate; the README's ~5.4 GB figure needs the full 26 GB library this environment doesn't have, flagged for a follow-up pass. (Originally reviewed and merged as [PR #26](https://github.com/sep-lab/Wit/pull/26), which was opened against the wrong base branch and never reached `main` — PR #29 is the corrected landing.) | +| M2.5 ✅ | [#15](https://github.com/sep-lab/Wit/issues/15) | **Reality gate** — measure the empty-verdict rate on a real Logic library | **Landed, and the gate passes.** `wit logic-report ` (`wit-index::logic_report`) walks every alternative's backup chain across a whole library and publishes the three statistics the issue asks for — see EXPERIMENTS.md §11. Run at the scale the issue specified (32 projects, 28 GB, 100 consecutive save pairs, 2026-08-16): **33.0% of save pairs show no structural change Wit can see**, against the issue's >50% "stop and map the fader first" threshold — so **M5 proceeds**. 28.0% of pairs are byte-different but structurally identical, reconfirming M2's finding at library scale. The M2 volume-fader stretch goal stays worth doing; it just no longer has to come first | +| M3 ✅ | [PR #29](https://github.com/sep-lab/Wit/pull/29) | Index, content-addressed store, CLI (`wit-index`, `wit-cli`) — covers Logic, GarageBand, and Ableton discovery | Landed — `wit scan` verified against real local Ableton (5 lineages / 30 versions, exact match) and Logic (10 versions) libraries, rescans idempotent through the actual CLI. `wit dupes` follow-up now **closed**: run against the full library it reports 5.5 GB of 23.9 GB (23.2%), agreeing with §9's independent Python walk (5.38 GB of 21.95 GB) — see EXPERIMENTS.md §9's cross-check. (Originally reviewed and merged as [PR #26](https://github.com/sep-lab/Wit/pull/26), which was opened against the wrong base branch and never reached `main` — PR #29 is the corrected landing.) | | M4 ✅ | [PR #27](https://github.com/sep-lab/Wit/pull/27) | Audio engine — decode, peaks, alignment, null-diff (`wit-audio`) | Landed — injected sample shifts (+1, −1, +4800, −12000) recovered exactly via FFT cross-correlation on seeded broadband noise (a pure tone can't exercise misalignment, per the issue's own reasoning); the confidence gate (refuse below 1.5) verified both ways — a real aligned pair scores tens-to-hundreds, two independently-seeded noise buffers score 0.9 and are correctly refused; real `afconvert`-generated PCM-CAF, ALAC-in-CAF, and ALAC-in-M4A fixtures decode sample-exact against the synthetic source (macOS, measured on this machine — loud-skips via `WIT_AUDIO_AFCONVERT`/`cfg!(target_os)` elsewhere, mirroring `wit-logic`'s `WIT_FIXTURES` doctrine); perf measured at 1.57 s for two 5-minute mono bounces end-to-end (decode+align+null-diff, release build, single-threaded, this machine) against the 10 s budget. Not yet exercised: the `afconvert` test running inside GitHub Actions' macOS runner itself, since it is intentionally opt-in (`#[ignore]`) rather than a default CI assertion — flagged for a follow-up pass | -| M5 | [#18](https://github.com/sep-lab/Wit/issues/18) | Tauri app alpha — Shelf, Story, Compare, watcher | Installs on a second Mac, reads a diff on the demo library, a real Logic folder, **and one Ableton `.als` lineage**, clicks Reveal | +| M5 | [#18](https://github.com/sep-lab/Wit/issues/18) | Tauri app alpha — Shelf, Story, Compare, watcher | **Unblocked** — M2.5's gate passed (33.0%, threshold >50%). Installs on a second Mac, reads a diff on the demo library, a real Logic folder, **and one Ableton `.als` lineage**, clicks Reveal | | M6 | [#22](https://github.com/sep-lab/Wit/issues/22) | Share bundle + zero-install viewer (`wit-share`, `viewer/`) | AirDrop the `.html` to another Mac **and** open it on a physical iPhone via iMessage/Quick Look — sentences, waveform, and audio all render with JS off; zero network requests | | M7-lite | [#19](https://github.com/sep-lab/Wit/issues/19) | **Unsigned** pilot package + PILOT.md | Clean Mac installs from the release link using only PILOT.md — recipients will see macOS's unsigned-app warning and use System Settings → Privacy & Security → "Open Anyway"; PILOT.md walks through it. Notarized distribution (Apple Developer ID) is a 0.1 upgrade, not a 0.0 blocker |