From a58cb7c42c2ad83d053bdf1ceaa60b4e371042ae Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 17:44:36 -0400 Subject: [PATCH 01/38] Record what prep wrote so --clean can reach the orphans stems cannot (#84) --clean matches orphans by stem, which structurally cannot reach two cases the normal iterate-on-recordings loop produces. Drop an input between runs and its clips are orphaned under a stem no rerun ever writes. Re-run a basename collision with only one of the pair and the disambiguated `take-1_*.wav` clips are orphaned the same way. Both linger and warn; training and eval glob the whole directory and read them as data nobody asked for. Prep now records the clips it writes in .prep-manifest.json, and --clean uses that record to remove the orphans the stem match misses. The record is what separates a clip prep made from a clip a user curated, so the guarantee that --clean never deletes a file prep did not create still holds. Provenance is not permission, though. "Prep made this file" does not say the file belongs to the dataset being refreshed, and without that second fact a mistyped --output -- positives re-run into the background directory -- would delete everything there, since none of those stems are in the inputs. So the manifest rule only arms when the run looks like a refresh of the recorded set: same label, at least one stem in common. A stray --output shares neither and falls back to the stem rule, which cannot reach past what the run just wrote. The write path refuses to relabel a directory recorded under another label, because doing so would hand a second stray run the matching label it needs. The prior clips of a take this run attempted but wrote nothing for stay excluded, manifest or not: that is the silent re-recording, where they are the only copy. The lingering warning now says which case a file is in, since a take that came back silent wants "check the recording," not the "remove them by hand" the old copy gave every file. Refs #84, #8 wake-20260723T1705-0bf0ee --- CHANGELOG.md | 12 ++ docs/wake-threshold-tuning.md | 20 +++- prep_wake_samples.py | 199 ++++++++++++++++++++++++++----- test_prep_wake_samples.py | 212 ++++++++++++++++++++++++++++++++++ 4 files changed, 410 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6072ca8..8127a87 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,18 @@ All notable changes to computah are recorded here. The format follows leftover count. `test_prep_wake_samples.py` covers the re-run-with-fewer-clips case with and without `--clean`, and that `--clean` spares files this run did not record. +- `prep_wake_samples.py` output manifest (#84): prep records the clips it writes + in `.prep-manifest.json` in the output dir, and `--clean` uses it to remove the + orphans the stem match cannot reach -- clips from an input dropped between runs, + and clips under a disambiguated stem (`take-1_000.wav`) that a later run no + longer produces. The record is what separates those from audio a user curated, + so `--clean` still never deletes a file prep did not create. Provenance alone is + not enough to delete, though: the manifest rule only arms when the run refreshes + the recorded set (same `--label`, at least one stem in common), so a mistyped + `--output` that lands a positives run in a background directory falls back to the + stem match instead of wiping it. `--clean` also still spares the prior clips of a + take this run attempted but wrote nothing for. An absent or unreadable manifest + degrades `--clean` to the stem match rather than failing the run. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 5b624a8..1c17a2c 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -33,10 +33,22 @@ repo. The evaluation script below is committed; the audio it reads is not. Re-running prep into a populated `samples/` dir refreshes it in place. If a re-recording yields fewer clips than last time, the extra clips from the old run would linger and the training globs would still read them, so prep warns and -names the count. Pass `--clean` to remove those leftovers in the same run. It is -deliberately narrow: it only deletes leftover clips of a take this run -re-recorded (a `_NNN.wav` clip whose stem the run wrote), so a source -recording or a hand-curated clip for a different stem is named but left in place. +names the count. Pass `--clean` to remove those leftovers in the same run. + +`--clean` only ever deletes clips prep itself made. It removes the now-unused +higher-numbered clips of a take this run re-recorded, and the clips a previous +run recorded for a take that is no longer in the inputs -- the second kind is +matched against `.prep-manifest.json`, a record prep keeps in the output dir of +what it wrote there. A source recording, a hand-curated clip, and anything else +absent from that record is named but left in place. + +That record proves prep made a file, not that the file belongs to what you are +refreshing, so the manifest half only applies when the run matches the recorded +set: same `--label`, and at least one take in common. Point `--clean` at the +wrong directory and it falls back to the narrow stem match, which cannot reach +past the clips the run just wrote. Prep also spares the prior clips of a take it +tried and got nothing from (a silent or bad re-recording): those are the only +copy, and it will not trade them for a run that produced nothing. ```bash .venv/bin/python prep_wake_samples.py --input computah_normal.wav \ diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 9b0cf99..a1376e2 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -24,6 +24,7 @@ from __future__ import annotations import argparse +import json import re import shutil import subprocess @@ -323,6 +324,77 @@ def _summarize(paths: list[Path], limit: int = 3) -> str: return shown +MANIFEST_NAME = ".prep-manifest.json" + + +def _read_manifest(out_dir: Path) -> tuple[str | None, set[str]]: + """The label and clip names a previous prep run recorded here. + + Fails soft on every read problem -- absent, unreadable, not JSON, wrong + shape -- returning ``(None, set())``. An empty record means "no record of + what prep made," which narrows `--clean` back to the stem rule below. + Failing loud would be worse in both directions: a corrupt manifest would + abort a run that has real work to do, and a manifest trusted despite a + partial read could name files prep never wrote. + """ + try: + raw = json.loads((out_dir / MANIFEST_NAME).read_text()) + except (OSError, ValueError): + return None, set() + if not isinstance(raw, dict): + return None, set() + clips = raw.get("clips") + if not isinstance(clips, list): + return None, set() + label = raw.get("label") + return ( + label if isinstance(label, str) else None, + {c for c in clips if isinstance(c, str)}, + ) + + +def _write_manifest( + out_dir: Path, label: str, names: set[str], prior_label: str | None +) -> None: + """Record which files in `out_dir` prep created, for a later --clean run. + + Written after the clips are on disk, and a failure only warns: the clips are + the output that matters, and losing the manifest costs precision on a future + --clean, not correctness. A run that wrote nothing into a directory that does + not exist has nothing to record and does not create one. + + A directory already recorded under a different label is left alone. Rewriting + it would relabel the other dataset's files as this one's, and a second stray + run into the same directory would then find a matching label and an + overlapping stem and read them as its own orphans. The mismatch is worth + saying out loud either way: it usually means ``--output`` points somewhere + unintended. + """ + if not out_dir.is_dir(): + return + if prior_label is not None and prior_label != label: + print( + f"warning: {out_dir} was last written as {prior_label!r} clips, not " + f"{label!r}; leaving its record alone. If --output is right, clear the " + f"directory (or remove {MANIFEST_NAME}) to start its record over", + file=sys.stderr, + ) + return + try: + (out_dir / MANIFEST_NAME).write_text( + json.dumps( + {"version": 1, "label": label, "clips": sorted(names)}, indent=2 + ) + + "\n" + ) + except OSError as e: + print( + f"warning: could not write {out_dir / MANIFEST_NAME} ({e}); " + f"a later --clean will fall back to matching re-recorded stems only", + file=sys.stderr, + ) + + def _clip_stem(name: str) -> str | None: """The stem of a "_NNN.wav" clip name, or None if it isn't one.""" m = _CLIP_NAME.fullmatch(name) @@ -340,11 +412,15 @@ def process( ) -> int: """Process every input file; return the total clip count written. - With ``clean`` set, a re-recorded take's leftover clips (a prior run's - now-unused higher-numbered clips for a stem this run wrote) are removed after - this run writes, so its own count matches what is on disk. It stays narrow on - purpose: an unrelated take's clips, a source recording, or hand-added audio is - named but never deleted, so a stray ``--output`` never silently wipes files. + With ``clean`` set, prep's own leftover clips are removed after this run + writes, so its count matches what is on disk. Two kinds qualify: a + re-recorded take's now-unused higher-numbered clips (matched by stem), and + clips a previous run recorded for a take that is not in this run's inputs at + all (matched against ``MANIFEST_NAME``, and only when this run refreshes the + recorded set -- same label, overlapping stems). Everything else is named but + never deleted -- a source recording, hand-added audio, another dataset a + stray ``--output`` landed in, and the prior clips of a take this run + attempted but wrote nothing for. """ # Checked before any audio is decoded, so an occupied --output is named once # rather than surfacing as whichever exception the run happens to reach first: @@ -365,6 +441,9 @@ def process( return 0 stems = _unique_stems(files) + # Read before anything is written, so it describes the directory as this run + # found it. + prior_label, prior = _read_manifest(out_dir) written: set[Path] = set() seen_inodes: set[int] = set() all_durs: list[float] = [] @@ -409,32 +488,94 @@ def process( run_stems = { stem for stem in (_clip_stem(p.name) for p in written) if stem is not None } + # The manifest widens --clean to the orphans the stem rule cannot reach: a + # clip prep recorded writing on an earlier run, whose stem is not in this + # run's inputs at all. That covers a dropped input and a basename collision + # whose disambiguated stem changed -- both leave clips no rerun will ever + # overwrite, and the manifest is what proves prep made them rather than a + # user curating the directory. + # + # But "prep made this file" is provenance, not permission. It does not say + # the file belongs to the dataset this run is refreshing, and without that + # second fact a mistyped --output -- positives re-run into the background + # directory -- would delete every clip there, since none of those stems are + # in the inputs. So the manifest rule only arms when this run looks like a + # refresh of the recorded set: same label, and at least one stem in common. + # A stray --output shares neither, and falls back to the stem rule, which + # cannot reach outside what this run just wrote. + # + # A stem this run DID attempt but wrote nothing for is excluded on purpose, + # manifest or not: that is the silent-re-recording case, where the prior + # clips are the only copy. The manifest says prep created a file; it does + # not say the file is expendable. + attempted = set(stems) + prior_stems = { + stem for stem in (_clip_stem(n) for n in prior) if stem is not None + } + refreshes_prior = prior_label == label and bool(run_stems & prior_stems) stale = _stale_clips(out_dir, written) - if stale: - removable = ( - [p for p in stale if _clip_stem(p.name) in run_stems] if clean else [] - ) - for p in removable: - p.unlink() - if removable: - print( - f"removed {len(removable)} clip(s) left from an earlier run " - f"({_summarize(removable)})" + removable = ( + [ + p + for p in stale + if _clip_stem(p.name) in run_stems + or ( + refreshes_prior + and p.name in prior + and _clip_stem(p.name) not in attempted ) - lingering = [p for p in stale if p not in removable] - if lingering: + ] + if clean + else [] + ) + for p in removable: + p.unlink() + if removable: + print( + f"removed {len(removable)} clip(s) left from an earlier run " + f"({_summarize(removable)})" + ) + lingering = [p for p in stale if p not in removable] + # Carry forward the prior run's record for files still on disk, so a + # directory built up over several runs keeps one provenance list rather than + # only remembering the most recent run. + _write_manifest( + out_dir, + label, + {p.name for p in written} | {p.name for p in lingering if p.name in prior}, + prior_label, + ) + if lingering: + # Why a file lingered decides what the user should do about it, and the + # cases want opposite advice. A take this run re-recorded and got nothing + # from kept its previous clips because they are the only copy -- telling + # that user to delete them by hand would undo the guard and lose the + # recording. Split the message rather than send one that is right for the + # common case and harmful for the case prep went out of its way to + # protect. + empty_takes = [p for p in lingering if _clip_stem(p.name) in attempted] + if empty_takes and clean: fix = ( - "these are not clips this run re-recorded, so --clean leaves them; " - "remove them by hand" - if clean - else "rerun with --clean to remove them, or clear the directory by hand" + "this run re-recorded " + f"{_summarize(empty_takes)} and got no clips from it, so --clean " + "kept the earlier ones rather than leave you with none; check the " + "recording before removing anything" ) - print( - f"warning: {len(lingering)} file(s) in {out_dir} are left from an " - f"earlier run ({_summarize(lingering)}). Training and eval read " - f"every clip in this directory, so {fix}.", - file=sys.stderr, + elif clean: + fix = ( + "prep has no record of creating these, so --clean leaves them; " + "remove them by hand if they are not wanted" ) + else: + fix = "rerun with --clean to remove them, or clear the directory by hand" + print( + f"warning: {len(lingering)} file(s) in {out_dir} are left from an " + f"earlier run ({_summarize(lingering)}). Training and eval read every " + f"clip in this directory, so they count as data nobody asked for: " + f"{fix}.", + file=sys.stderr, + ) + if all_durs: a = np.array(all_durs) print( @@ -476,9 +617,9 @@ def main() -> int: p.add_argument( "--clean", action="store_true", - help="remove a re-recorded take's leftover clips in --output (a prior run's " - "now-unused higher-numbered clips for a take in this run); leaves unrelated " - "or hand-added audio in place", + help="remove prep's own leftover clips in --output: a re-recorded take's " + "now-unused higher-numbered clips, plus clips a previous run recorded for a " + "take no longer in the inputs; leaves hand-added audio in place", ) args = p.parse_args() diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index bb3ea33..78b5dfd 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -521,6 +521,211 @@ def test_stem_assignment_ignores_input_order(d: Path) -> None: ) +def _burst_take(path: Path, n: int) -> None: + """Write a take of `n` well-separated utterances at `path`.""" + path.parent.mkdir(parents=True, exist_ok=True) + sf.write( + path, make_bursts(n, 0.5, 0.8, prep.TARGET_SR), prep.TARGET_SR, subtype="PCM_16" + ) + + +def test_manifest_cleans_dropped_input_orphans(d: Path) -> None: + """--clean removes the orphans of an input dropped between runs (#84 case 1). + + The stem rule alone cannot reach these: `other` is not in the rerun at all, + so no clip this run writes ever makes its old ones stale. The manifest is + what distinguishes them from audio a user put in the directory. + """ + src = d / "dropped_src" + _burst_take(src / "a" / "take.wav", 3) + _burst_take(src / "b" / "other.wav", 2) + out = d / "dropped_out" + prep.process([src / "a" / "take.wav", src / "b" / "other.wav"], out, "positive", 0.3, 0.2, 3.0) + check( + (out / prep.MANIFEST_NAME).is_file(), + "a run records what it wrote in the manifest", + ) + + prep.process([src / "a" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + present == ["take_000.wav", "take_001.wav", "take_002.wav"], + f"--clean removes a dropped input's orphans ({present})", + ) + + +def test_manifest_cleans_changed_collider_stem_orphans(d: Path) -> None: + """--clean removes orphans whose disambiguated stem vanished (#84 case 2). + + Two `take.wav` inputs become stems `take` and `take-1`. Rerun with only one + and `take-1` is not a stem any more, so its clips are orphaned under a name + nothing will overwrite. + """ + src = d / "collider_src" + _burst_take(src / "a" / "take.wav", 3) + _burst_take(src / "b" / "take.wav", 2) + out = d / "collider_out" + prep.process([src / "a" / "take.wav", src / "b" / "take.wav"], out, "positive", 0.3, 0.2, 3.0) + first = sorted(p.name for p in out.glob("*.wav")) + check( + any(n.startswith("take-1_") for n in first), + f"the collider run writes a disambiguated stem ({first})", + ) + + prep.process([src / "b" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + not any(n.startswith("take-1_") for n in present), + f"--clean removes orphans of a stem the rerun no longer produces ({present})", + ) + + +def test_manifest_never_authorizes_deleting_curated_audio(d: Path) -> None: + """The manifest widens --clean only to files prep recorded writing. + + A hand-added clip whose stem no run ever wrote is exactly the shape the new + rule reaches for -- an orphan of an unattempted stem -- so this pins that it + is still spared. Being absent from the manifest is the whole guarantee. + """ + src = d / "curated_src" + _burst_take(src / "take.wav", 3) + out = d / "curated_out" + prep.process([src / "take.wav"], out, "positive", 0.3, 0.2, 3.0) + _burst_take(out / "custom_001.wav", 1) + _burst_take(out / "my_recording.wav", 1) + + prep.process([src / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + "custom_001.wav" in present and "my_recording.wav" in present, + f"--clean spares audio prep did not create ({present})", + ) + + +def test_manifest_spares_prior_clips_when_rerun_writes_nothing(d: Path) -> None: + """The manifest must not reopen the silent-re-recording data loss. + + Those clips ARE in the manifest, so a rule that removed every manifested + clip the run did not rewrite would delete the only copy of a take whose + re-recording came back silent. The stem has to be excluded because the run + attempted it, regardless of what the manifest says. + """ + src = d / "silent_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "silent_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + + _burst_take(take, 3) + second = prep.process([take], out, "positive", 0.3, 5.0, 10.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + second == 0 + and present == ["take_000.wav", "take_001.wav", "take_002.wav"], + f"a manifested clip is still spared when its take wrote nothing ({present})", + ) + + +def test_manifest_spares_a_silent_take_alongside_a_good_one(d: Path) -> None: + """The `attempted` guard, with the manifest rule actually armed. + + When the whole run comes back empty nothing is written, so the manifest rule + never arms and the guard is never reached. The case that reaches it is a run + where one take records fine and another comes back silent: the good take + supplies the stem overlap that arms the rule, and then only `attempted` + stands between the silent take's earlier clips -- its only copy -- and + unlink(). Delete that clause and this is the test that goes red. + """ + src = d / "mixed_src" + good = src / "good.wav" + flaky = src / "flaky.wav" + _burst_take(good, 2) + _burst_take(flaky, 3) + out = d / "mixed_out" + prep.process([good, flaky], out, "positive", 0.3, 0.2, 3.0) + before = sorted(p.name for p in out.glob("*.wav")) + check(len(before) == 5, f"both takes record on the first run ({before})") + + # Re-record both. `good` is fine; `flaky` comes back as one long unbroken + # tone -- a mic left running, say -- which the max-duration cap drops + # entirely, so it writes nothing while `good` writes and arms the rule. + _burst_take(good, 2) + sf.write( + flaky, + np.ones(int(prep.TARGET_SR * 4.0), dtype=np.float32) * 0.5, + prep.TARGET_SR, + subtype="PCM_16", + ) + prep.process([good, flaky], out, "positive", 0.3, 0.2, 1.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + sorted(n for n in present if n.startswith("flaky_")) + == ["flaky_000.wav", "flaky_001.wav", "flaky_002.wav"], + f"--clean spares a silent take's only copy while cleaning beside it ({present})", + ) + + +def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: + """A corrupt manifest degrades --clean, it does not fail the run. + + The clips are the output that matters. Reading garbage as "no record" costs + precision on the orphans only the manifest can reach; treating it as fatal + would block a run that has real work to do. + """ + src = d / "corrupt_src" + _burst_take(src / "a" / "take.wav", 3) + _burst_take(src / "b" / "other.wav", 2) + out = d / "corrupt_out" + prep.process([src / "a" / "take.wav", src / "b" / "other.wav"], out, "positive", 0.3, 0.2, 3.0) + (out / prep.MANIFEST_NAME).write_text("{not json") + + _burst_take(src / "a" / "take.wav", 2) + total = prep.process([src / "a" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + total == 2 and "take_002.wav" not in present and "other_000.wav" in present, + f"a corrupt manifest leaves the stem rule working ({present})", + ) + + +def test_manifest_spares_a_different_dataset_on_stray_output(d: Path) -> None: + """A mistyped --output must not let the manifest wipe another dataset. + + Provenance says prep created a file, not that the file belongs to the set + being refreshed, so the manifest rule only arms on a matching label with an + overlapping stem. Without that, every clip in a background dir is one prep + wrote and no positives rerun attempts, which would be enough to delete the + whole directory. + """ + src = d / "stray_src" + _burst_take(src / "noise.wav", 2) + bg = d / "stray_background" + prep.process([src / "noise.wav"], bg, "background", 0.3, 0.2, 3.0) + before = sorted(p.name for p in bg.glob("*.wav")) + check(before == ["noise_000.wav"], f"the background dir starts populated ({before})") + + # The typo: a positives run pointed at the background dir. + _burst_take(src / "computah.wav", 3) + prep.process([src / "computah.wav"], bg, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in bg.glob("*.wav")) + check( + "noise_000.wav" in present, + f"--clean spares a dataset this run is not refreshing ({present})", + ) + + # And the guard has to survive the typo being repeated. If the stray run had + # relabeled the directory's record as "positive", this second one would find + # a matching label plus an overlapping stem and read the background clip as + # its own orphan. + _burst_take(src / "computah.wav", 2) + prep.process([src / "computah.wav"], bg, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in bg.glob("*.wav")) + check( + "noise_000.wav" in present, + f"a repeated stray --output still spares the other dataset ({present})", + ) + + def main() -> int: test_segment_count() with tempfile.TemporaryDirectory(prefix="prep-wake-") as tmp: @@ -541,6 +746,13 @@ def main() -> int: test_in_run_collision_raises(d) test_refresh_after_adding_collider_leaves_no_orphans(d) test_stem_assignment_ignores_input_order(d) + test_manifest_cleans_dropped_input_orphans(d) + test_manifest_cleans_changed_collider_stem_orphans(d) + test_manifest_never_authorizes_deleting_curated_audio(d) + test_manifest_spares_prior_clips_when_rerun_writes_nothing(d) + test_manifest_spares_a_silent_take_alongside_a_good_one(d) + test_manifest_unreadable_falls_back_to_stem_rule(d) + test_manifest_spares_a_different_dataset_on_stray_output(d) n_pass = sum(1 for ok, _ in results if ok) print(f"=== {n_pass}/{len(results)} checks passed ===") return 0 if n_pass == len(results) else 1 From c8100668f7969033a5692a74ffe7bca061fb810d Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 17:46:57 -0400 Subject: [PATCH 02/38] Apply ruff format to the two touched files CI gates on `ruff format --check .`; both files were written by hand and missed it. --- prep_wake_samples.py | 8 ++------ test_prep_wake_samples.py | 33 ++++++++++++++++++++++++++------- 2 files changed, 28 insertions(+), 13 deletions(-) diff --git a/prep_wake_samples.py b/prep_wake_samples.py index a1376e2..4e3ed97 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -382,9 +382,7 @@ def _write_manifest( return try: (out_dir / MANIFEST_NAME).write_text( - json.dumps( - {"version": 1, "label": label, "clips": sorted(names)}, indent=2 - ) + json.dumps({"version": 1, "label": label, "clips": sorted(names)}, indent=2) + "\n" ) except OSError as e: @@ -509,9 +507,7 @@ def process( # clips are the only copy. The manifest says prep created a file; it does # not say the file is expendable. attempted = set(stems) - prior_stems = { - stem for stem in (_clip_stem(n) for n in prior) if stem is not None - } + prior_stems = {stem for stem in (_clip_stem(n) for n in prior) if stem is not None} refreshes_prior = prior_label == label and bool(run_stems & prior_stems) stale = _stale_clips(out_dir, written) removable = ( diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 78b5dfd..4cfbd10 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -540,7 +540,14 @@ def test_manifest_cleans_dropped_input_orphans(d: Path) -> None: _burst_take(src / "a" / "take.wav", 3) _burst_take(src / "b" / "other.wav", 2) out = d / "dropped_out" - prep.process([src / "a" / "take.wav", src / "b" / "other.wav"], out, "positive", 0.3, 0.2, 3.0) + prep.process( + [src / "a" / "take.wav", src / "b" / "other.wav"], + out, + "positive", + 0.3, + 0.2, + 3.0, + ) check( (out / prep.MANIFEST_NAME).is_file(), "a run records what it wrote in the manifest", @@ -565,7 +572,9 @@ def test_manifest_cleans_changed_collider_stem_orphans(d: Path) -> None: _burst_take(src / "a" / "take.wav", 3) _burst_take(src / "b" / "take.wav", 2) out = d / "collider_out" - prep.process([src / "a" / "take.wav", src / "b" / "take.wav"], out, "positive", 0.3, 0.2, 3.0) + prep.process( + [src / "a" / "take.wav", src / "b" / "take.wav"], out, "positive", 0.3, 0.2, 3.0 + ) first = sorted(p.name for p in out.glob("*.wav")) check( any(n.startswith("take-1_") for n in first), @@ -620,8 +629,7 @@ def test_manifest_spares_prior_clips_when_rerun_writes_nothing(d: Path) -> None: second = prep.process([take], out, "positive", 0.3, 5.0, 10.0, clean=True) present = sorted(p.name for p in out.glob("*.wav")) check( - second == 0 - and present == ["take_000.wav", "take_001.wav", "take_002.wav"], + second == 0 and present == ["take_000.wav", "take_001.wav", "take_002.wav"], f"a manifested clip is still spared when its take wrote nothing ({present})", ) @@ -676,11 +684,20 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: _burst_take(src / "a" / "take.wav", 3) _burst_take(src / "b" / "other.wav", 2) out = d / "corrupt_out" - prep.process([src / "a" / "take.wav", src / "b" / "other.wav"], out, "positive", 0.3, 0.2, 3.0) + prep.process( + [src / "a" / "take.wav", src / "b" / "other.wav"], + out, + "positive", + 0.3, + 0.2, + 3.0, + ) (out / prep.MANIFEST_NAME).write_text("{not json") _burst_take(src / "a" / "take.wav", 2) - total = prep.process([src / "a" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + total = prep.process( + [src / "a" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True + ) present = sorted(p.name for p in out.glob("*.wav")) check( total == 2 and "take_002.wav" not in present and "other_000.wav" in present, @@ -702,7 +719,9 @@ def test_manifest_spares_a_different_dataset_on_stray_output(d: Path) -> None: bg = d / "stray_background" prep.process([src / "noise.wav"], bg, "background", 0.3, 0.2, 3.0) before = sorted(p.name for p in bg.glob("*.wav")) - check(before == ["noise_000.wav"], f"the background dir starts populated ({before})") + check( + before == ["noise_000.wav"], f"the background dir starts populated ({before})" + ) # The typo: a positives run pointed at the background dir. _burst_take(src / "computah.wav", 3) From d46da58debcee1847eab141298153a9740c85bd4 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 19:10:39 -0400 Subject: [PATCH 03/38] Gate every --clean removal and the record write on one question (#84) The cloud review found the dataset guard was applied to the manifest branch alone, leaving three holes in the same class. A label mismatch disarmed the manifest rule but not the stem match, so a positives run into a negatives directory still deleted that dataset's higher-numbered take_* clips. Worse, deciding this at --clean time is too late: _write_unique has already overwritten take_000.wav with the wrong dataset's audio, while the warning claimed the files were left alone. A mismatched label is now refused before any audio is decoded, beside the sibling precondition that already rejects a non-directory --output. There is no reading of it where writing positives into a directory of negatives is what the user meant. The stem match ignored the record even when one existed, so a clip a user dropped in beside a manifested dataset was deleted by filename shape alone -- the one case the "only removes what prep made" guarantee most needed to cover. It consults the record now, falling back to shape only for a directory that predates manifests. The write path seeded its own stems into a record it had just refused to delete from, so repeating the same wrong --output would find the overlap it needed and start deleting. Removals and the write are now gated on the same question. Three tests, one per hole: a mismatched label refused before writing, a hand-added clip spared under a recorded stem, and a same-label stray that never gains permission by being repeated. 49/49. --- CHANGELOG.md | 19 +++++--- docs/wake-threshold-tuning.md | 11 +++-- prep_wake_samples.py | 83 +++++++++++++++++++++-------------- test_prep_wake_samples.py | 75 +++++++++++++++++++++++++++++++ 4 files changed, 146 insertions(+), 42 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8127a87..b23c618 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,12 +23,19 @@ All notable changes to computah are recorded here. The format follows and clips under a disambiguated stem (`take-1_000.wav`) that a later run no longer produces. The record is what separates those from audio a user curated, so `--clean` still never deletes a file prep did not create. Provenance alone is - not enough to delete, though: the manifest rule only arms when the run refreshes - the recorded set (same `--label`, at least one stem in common), so a mistyped - `--output` that lands a positives run in a background directory falls back to the - stem match instead of wiping it. `--clean` also still spares the prior clips of a - take this run attempted but wrote nothing for. An absent or unreadable manifest - degrades `--clean` to the stem match rather than failing the run. + not enough to delete, though, and the record is also what lets prep refuse a + mistyped `--output` outright: a run whose `--label` disagrees with the record + already in the directory now errors before a single clip is decoded, since by + `--clean` time it would already have overwritten that dataset's `take_000.wav`. + Past that, a run still has to be refreshing the record it found (at least one + take in common) before it may delete from it or write to it — folding its stems + into someone else's record would hand a repeat of the same mistake the overlap it + needs to start deleting. Once a record exists the stem match + consults it too, so a clip a user dropped in beside a manifested dataset is + spared even when it shares a stem prep writes. `--clean` also still spares the + prior clips of a take this run attempted but wrote nothing for. An absent or + unreadable manifest degrades `--clean` to the stem match rather than failing the + run. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 1c17a2c..48d4bdc 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -43,10 +43,13 @@ what it wrote there. A source recording, a hand-curated clip, and anything else absent from that record is named but left in place. That record proves prep made a file, not that the file belongs to what you are -refreshing, so the manifest half only applies when the run matches the recorded -set: same `--label`, and at least one take in common. Point `--clean` at the -wrong directory and it falls back to the narrow stem match, which cannot reach -past the clips the run just wrote. Prep also spares the prior clips of a take it +refreshing. A `--label` that disagrees with the record already in the directory +is refused outright, before anything is decoded: that is a mistyped `--output`, +and waiting until `--clean` would be too late, since the run would already have +overwritten that dataset's clips. Past that, a run has to be refreshing the +record it found (at least one take in common) before it may delete from it or +write to it. A directory with no record yet counts as yours. +Prep also spares the prior clips of a take it tried and got nothing from (a silent or bad re-recording): those are the only copy, and it will not trade them for a run that produced nothing. diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 4e3ed97..8823ba6 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -353,9 +353,7 @@ def _read_manifest(out_dir: Path) -> tuple[str | None, set[str]]: ) -def _write_manifest( - out_dir: Path, label: str, names: set[str], prior_label: str | None -) -> None: +def _write_manifest(out_dir: Path, label: str, names: set[str]) -> None: """Record which files in `out_dir` prep created, for a later --clean run. Written after the clips are on disk, and a failure only warns: the clips are @@ -363,23 +361,11 @@ def _write_manifest( --clean, not correctness. A run that wrote nothing into a directory that does not exist has nothing to record and does not create one. - A directory already recorded under a different label is left alone. Rewriting - it would relabel the other dataset's files as this one's, and a second stray - run into the same directory would then find a matching label and an - overlapping stem and read them as its own orphans. The mismatch is worth - saying out loud either way: it usually means ``--output`` points somewhere - unintended. + The caller decides whether this run may claim the directory (see + ``refreshes_prior``); this only writes. """ if not out_dir.is_dir(): return - if prior_label is not None and prior_label != label: - print( - f"warning: {out_dir} was last written as {prior_label!r} clips, not " - f"{label!r}; leaving its record alone. If --output is right, clear the " - f"directory (or remove {MANIFEST_NAME}) to start its record over", - file=sys.stderr, - ) - return try: (out_dir / MANIFEST_NAME).write_text( json.dumps({"version": 1, "label": label, "clips": sorted(names)}, indent=2) @@ -432,6 +418,21 @@ def process( ) return 0 + # Refused before any audio is decoded, because by the time --clean is + # evaluated the damage is already done: _write_unique would have overwritten + # this directory's take_000.wav with the new dataset's. A label mismatch is + # the signature of a mistyped --output, and there is no reading of it where + # writing positives into a directory of negatives is what the user meant. + prior_label, prior = _read_manifest(out_dir) + if prior_label is not None and prior_label != label: + print( + f"error: --output {out_dir} holds {prior_label!r} clips, not {label!r}; " + f"nothing was written. Point --output at this run's directory, or clear " + f"that one (or remove its {MANIFEST_NAME}) to reuse it", + file=sys.stderr, + ) + return 0 + files = _inputs(inputs) if not files: joined = ", ".join(str(p) for p in inputs) @@ -439,9 +440,6 @@ def process( return 0 stems = _unique_stems(files) - # Read before anything is written, so it describes the directory as this run - # found it. - prior_label, prior = _read_manifest(out_dir) written: set[Path] = set() seen_inodes: set[int] = set() all_durs: list[float] = [] @@ -508,17 +506,31 @@ def process( # not say the file is expendable. attempted = set(stems) prior_stems = {stem for stem in (_clip_stem(n) for n in prior) if stem is not None} - refreshes_prior = prior_label == label and bool(run_stems & prior_stems) + # A mismatched label was already refused above, so the question left is + # whether this run is refreshing the record it found or landed beside it: a + # directory with no record is unclaimed and counts as ours, and one with a + # record needs a take in common. Without the overlap, prep neither deletes nor + # writes -- folding this run's stems into that record would hand a repeat of + # the same mistake the overlap it needs to start deleting. + refreshes_prior = not prior or bool(run_stems & prior_stems) stale = _stale_clips(out_dir, written) removable = ( [ p for p in stale - if _clip_stem(p.name) in run_stems - or ( - refreshes_prior - and p.name in prior - and _clip_stem(p.name) not in attempted + if refreshes_prior + and ( + # A re-recorded take's now-unused clips. Once there is a record, + # this consults it too: without that, a hand-added take_003.wav + # beside a manifested `take` dataset would be deleted by shape + # alone, and the promise that --clean only removes what prep made + # would hold everywhere except the case a user is most likely to + # hit. With no record (a directory from before manifests), shape + # is all there is. + (_clip_stem(p.name) in run_stems and (not prior or p.name in prior)) + # An orphan of a take this run does not have: a dropped input, or + # a collider whose disambiguated stem changed. + or (p.name in prior and _clip_stem(p.name) not in attempted) ) ] if clean @@ -535,12 +547,19 @@ def process( # Carry forward the prior run's record for files still on disk, so a # directory built up over several runs keeps one provenance list rather than # only remembering the most recent run. - _write_manifest( - out_dir, - label, - {p.name for p in written} | {p.name for p in lingering if p.name in prior}, - prior_label, - ) + if refreshes_prior: + _write_manifest( + out_dir, + label, + {p.name for p in written} | {p.name for p in lingering if p.name in prior}, + ) + elif out_dir.is_dir(): + print( + f"warning: {out_dir} holds a {label!r} dataset with no take in common " + f"with this run, so its record was left as it was. Clips this run wrote " + f"are on disk; a later --clean will not treat them as part of it", + file=sys.stderr, + ) if lingering: # Why a file lingered decides what the user should do about it, and the # cases want opposite advice. A take this run re-recorded and got nothing diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 4cfbd10..406fe3e 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -745,6 +745,78 @@ def test_manifest_spares_a_different_dataset_on_stray_output(d: Path) -> None: ) +def test_clean_disarmed_entirely_on_a_label_mismatch(d: Path) -> None: + """A label mismatch has to disarm the stem match too, not just the manifest. + + The stems collide in practice: `take.wav` is as plausible a negatives + filename as a positives one. Deciding this at --clean time is too late -- + by then the run has already overwritten take_000.wav with the wrong + dataset's audio -- so the check runs before any decode. + """ + src = d / "mismatch_src" + _burst_take(src / "take.wav", 3) + neg = d / "mismatch_neg" + prep.process([src / "take.wav"], neg, "negative", 0.3, 0.2, 3.0) + before = sorted(p.name for p in neg.glob("*.wav")) + check(len(before) == 3, f"the negatives dir starts with three clips ({before})") + + _burst_take(src / "take.wav", 1) + total = prep.process([src / "take.wav"], neg, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in neg.glob("*.wav")) + check( + total == 0 and present == before, + f"a mismatched --output is refused before anything is written ({present})", + ) + + +def test_clean_spares_a_hand_added_clip_under_a_recorded_stem(d: Path) -> None: + """The stem match consults the record once there is one. + + A clip a user dropped in beside a manifested dataset shares the stem prep + writes, so shape alone cannot tell it from an orphan. Deleting it would break + the guarantee this whole change rests on: --clean removes only what prep + made. + """ + src = d / "handadd_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "handadd_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + _burst_take(out / "take_007.wav", 1) # kept by hand, never prep's + + _burst_take(take, 2) + prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + "take_007.wav" in present and "take_002.wav" not in present, + f"--clean takes its own orphan and leaves the hand-added clip ({present})", + ) + + +def test_a_same_label_stray_does_not_seed_the_manifest(d: Path) -> None: + """A stray run must not fold its stems into another dataset's record. + + Same label, no take in common: the run is disarmed, but if it still wrote + itself into the record then repeating the typo would find the overlap it + needs and start deleting the original dataset as dropped inputs. The write + is gated on the same question as the removals. + """ + src = d / "seed_src" + _burst_take(src / "noise.wav", 2) + _burst_take(src / "computah.wav", 3) + out = d / "seed_out" + prep.process([src / "noise.wav"], out, "positive", 0.3, 0.2, 3.0) + + # The typo, twice. + for _ in range(2): + prep.process([src / "computah.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + "noise_000.wav" in present and "noise_001.wav" in present, + f"a repeated same-label stray never gains permission to delete ({present})", + ) + + def main() -> int: test_segment_count() with tempfile.TemporaryDirectory(prefix="prep-wake-") as tmp: @@ -772,6 +844,9 @@ def main() -> int: test_manifest_spares_a_silent_take_alongside_a_good_one(d) test_manifest_unreadable_falls_back_to_stem_rule(d) test_manifest_spares_a_different_dataset_on_stray_output(d) + test_clean_disarmed_entirely_on_a_label_mismatch(d) + test_clean_spares_a_hand_added_clip_under_a_recorded_stem(d) + test_a_same_label_stray_does_not_seed_the_manifest(d) n_pass = sum(1 for ok, _ in results if ok) print(f"=== {n_pass}/{len(results)} checks passed ===") return 0 if n_pass == len(results) else 1 From 60ca24d708b0de77bb10cf097ee2f5f4f1fc91a4 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:03:06 -0400 Subject: [PATCH 04/38] test(prep): reproduce manifest ownership gaps --- test_prep_wake_samples.py | 87 ++++++++++++++++++++++++++++++++++----- 1 file changed, 76 insertions(+), 11 deletions(-) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 406fe3e..30aa472 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -705,6 +705,32 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: ) +def test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d: Path) -> None: + """The first manifest must not strand clips from a pre-manifest run. + + A normal refresh without --clean warns that its higher-numbered leftovers + can be removed by rerunning with --clean. When that refresh is also the run + that introduces the manifest, the record has to retain provenance for those + leftovers; otherwise the promised follow-up clean can no longer distinguish + them from hand-added audio. + """ + src = d / "bootstrap_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "bootstrap_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + (out / prep.MANIFEST_NAME).unlink() # Simulate an output from before manifests. + + _burst_take(take, 2) + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + present == ["take_000.wav", "take_001.wav"], + f"the first manifest leaves its warned-about leftovers cleanable ({present})", + ) + + def test_manifest_spares_a_different_dataset_on_stray_output(d: Path) -> None: """A mistyped --output must not let the manifest wipe another dataset. @@ -745,6 +771,42 @@ def test_manifest_spares_a_different_dataset_on_stray_output(d: Path) -> None: ) +def test_same_label_shared_stem_stray_is_refused_before_writing(d: Path) -> None: + """One generic stem match is not proof that two datasets are the same. + + A prior dataset and a stray run can both contain ``take.wav`` while coming + from different source paths. Treating that basename collision as ownership + lets --clean delete every other manifested stem in the prior dataset. The + output must remain untouched unless the run can prove it is refreshing the + recorded source set. + """ + src = d / "shared_stem_src" + original = src / "original" + stray = src / "stray" + _burst_take(original / "take.wav", 3) + _burst_take(original / "other.wav", 2) + _burst_take(stray / "take.wav", 1) + out = d / "shared_stem_out" + prep.process( + [original / "take.wav", original / "other.wav"], + out, + "positive", + 0.3, + 0.2, + 3.0, + ) + before = sorted(p.name for p in out.glob("*.wav")) + + total = prep.process( + [stray / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True + ) + present = sorted(p.name for p in out.glob("*.wav")) + check( + total == 0 and present == before, + f"a same-label basename collision cannot claim another dataset ({present})", + ) + + def test_clean_disarmed_entirely_on_a_label_mismatch(d: Path) -> None: """A label mismatch has to disarm the stem match too, not just the manifest. @@ -793,13 +855,12 @@ def test_clean_spares_a_hand_added_clip_under_a_recorded_stem(d: Path) -> None: ) -def test_a_same_label_stray_does_not_seed_the_manifest(d: Path) -> None: - """A stray run must not fold its stems into another dataset's record. +def test_same_label_stray_is_refused_before_writing(d: Path) -> None: + """A no-overlap run must not pollute another same-label dataset. - Same label, no take in common: the run is disarmed, but if it still wrote - itself into the record then repeating the typo would find the overlap it - needs and start deleting the original dataset as dropped inputs. The write - is gated on the same question as the removals. + The manifest already proves the output belongs to a recorded source set. + With no source in common, the likely wrong --output is known before any + audio is decoded, so warning after clips have landed is too late. """ src = d / "seed_src" _burst_take(src / "noise.wav", 2) @@ -807,13 +868,15 @@ def test_a_same_label_stray_does_not_seed_the_manifest(d: Path) -> None: out = d / "seed_out" prep.process([src / "noise.wav"], out, "positive", 0.3, 0.2, 3.0) - # The typo, twice. - for _ in range(2): + before = sorted(p.name for p in out.glob("*.wav")) + totals = [ prep.process([src / "computah.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + for _ in range(2) + ] present = sorted(p.name for p in out.glob("*.wav")) check( - "noise_000.wav" in present and "noise_001.wav" in present, - f"a repeated same-label stray never gains permission to delete ({present})", + totals == [0, 0] and present == before, + f"a repeated same-label stray writes nothing ({present})", ) @@ -843,10 +906,12 @@ def main() -> int: test_manifest_spares_prior_clips_when_rerun_writes_nothing(d) test_manifest_spares_a_silent_take_alongside_a_good_one(d) test_manifest_unreadable_falls_back_to_stem_rule(d) + test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d) test_manifest_spares_a_different_dataset_on_stray_output(d) + test_same_label_shared_stem_stray_is_refused_before_writing(d) test_clean_disarmed_entirely_on_a_label_mismatch(d) test_clean_spares_a_hand_added_clip_under_a_recorded_stem(d) - test_a_same_label_stray_does_not_seed_the_manifest(d) + test_same_label_stray_is_refused_before_writing(d) n_pass = sum(1 for ok, _ in results if ok) print(f"=== {n_pass}/{len(results)} checks passed ===") return 0 if n_pass == len(results) else 1 From 9c7e75c5e0ca94bb996446f2778e00dcc2669e08 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:19:19 -0400 Subject: [PATCH 05/38] Prevent stray sample runs from claiming manifested output --- CHANGELOG.md | 36 ++-- docs/wake-threshold-tuning.md | 38 +++-- prep_wake_samples.py | 299 ++++++++++++++++++++-------------- test_prep_wake_samples.py | 88 +++++++++- 4 files changed, 311 insertions(+), 150 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b23c618..db7e4fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,25 +17,23 @@ All notable changes to computah are recorded here. The format follows leftover count. `test_prep_wake_samples.py` covers the re-run-with-fewer-clips case with and without `--clean`, and that `--clean` spares files this run did not record. -- `prep_wake_samples.py` output manifest (#84): prep records the clips it writes - in `.prep-manifest.json` in the output dir, and `--clean` uses it to remove the - orphans the stem match cannot reach -- clips from an input dropped between runs, - and clips under a disambiguated stem (`take-1_000.wav`) that a later run no - longer produces. The record is what separates those from audio a user curated, - so `--clean` still never deletes a file prep did not create. Provenance alone is - not enough to delete, though, and the record is also what lets prep refuse a - mistyped `--output` outright: a run whose `--label` disagrees with the record - already in the directory now errors before a single clip is decoded, since by - `--clean` time it would already have overwritten that dataset's `take_000.wav`. - Past that, a run still has to be refreshing the record it found (at least one - take in common) before it may delete from it or write to it — folding its stems - into someone else's record would hand a repeat of the same mistake the overlap it - needs to start deleting. Once a record exists the stem match - consults it too, so a clip a user dropped in beside a manifested dataset is - spared even when it shares a stem prep writes. `--clean` also still spares the - prior clips of a take this run attempted but wrote nothing for. An absent or - unreadable manifest degrades `--clean` to the stem match rather than failing the - run. +- `prep_wake_samples.py` output manifest (#84): prep records each resolved source + path and the clips it owns in `.prep-manifest.json` in the output dir. + `--clean` uses that provenance to remove orphans the stem match cannot reach -- + clips from an input dropped between runs, and clips under a disambiguated stem + (`take-1_000.wav`) that a later run no longer produces. A run must share an + exact recorded source path before it may write to that dataset, so a different + dataset with the same label and generic basename cannot overwrite or clean it. + A source this run attempted but got no clips from retains its prior clips as the + only good copy, including when same-basename inputs changed its output stem. + With a readable source map, hand-curated audio stays outside it and is never + deleted. The first successful manifest run adopts same-stem leftovers from a + pre-manifest run so a warned-about orphan remains cleanable; a no-output run + cannot claim an unrecorded directory. An absent or unreadable manifest degrades + `--clean` to the narrow stem match, which cannot distinguish a prep leftover + from a hand-added `_NNN.wav`. A readable older manifest without + source ownership is refused until the user removes it to bootstrap ownership + explicitly. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 48d4bdc..40f130d 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -35,23 +35,37 @@ re-recording yields fewer clips than last time, the extra clips from the old run would linger and the training globs would still read them, so prep warns and names the count. Pass `--clean` to remove those leftovers in the same run. -`--clean` only ever deletes clips prep itself made. It removes the now-unused -higher-numbered clips of a take this run re-recorded, and the clips a previous -run recorded for a take that is no longer in the inputs -- the second kind is -matched against `.prep-manifest.json`, a record prep keeps in the output dir of -what it wrote there. A source recording, a hand-curated clip, and anything else -absent from that record is named but left in place. +With a readable source-aware manifest, `--clean` only deletes clips prep recorded +making. It removes the now-unused higher-numbered clips of a take this run +re-recorded, and the clips a previous run recorded for a take that is no longer +in the inputs. The `.prep-manifest.json` in the output dir maps each resolved +source path to the clips prep wrote for it. A source recording, a hand-curated +clip, and anything else absent from that map is named but left in place. That record proves prep made a file, not that the file belongs to what you are refreshing. A `--label` that disagrees with the record already in the directory is refused outright, before anything is decoded: that is a mistyped `--output`, and waiting until `--clean` would be too late, since the run would already have -overwritten that dataset's clips. Past that, a run has to be refreshing the -record it found (at least one take in common) before it may delete from it or -write to it. A directory with no record yet counts as yours. -Prep also spares the prior clips of a take it -tried and got nothing from (a silent or bad re-recording): those are the only -copy, and it will not trade them for a run that produced nothing. +overwritten that dataset's clips. Past that, a run must contain at least one +exact source path from the record before it may delete from the dataset or write +to it. A shared basename is not ownership. A directory with no record yet counts +as yours; its first successful run also records same-stem legacy leftovers so a +follow-up `--clean` can remove them. A run that produces no clips does not claim +an unrecorded directory. Prep also spares the prior clips of a source it tried +and got nothing from (a silent or bad re-recording): those are the only copy, and +it will not trade them for a run that produced nothing. + +To add a new recording to an existing dataset, include at least one source +already recorded there in the same invocation. This gives prep explicit +ownership proof while it adds the new source to the manifest. + +An absent or unreadable manifest falls back to the narrow same-stem cleanup +rule. That fallback has no provenance: `--clean` can remove any +`_NNN.wav` file, including a hand-added file with that shape. Check the +warning list and move any such file before cleaning. A readable older manifest +without source ownership is refused; after confirming the output directory, +remove that manifest once to bootstrap the source map from the next successful +run. ```bash .venv/bin/python prep_wake_samples.py --input computah_normal.wav \ diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 8823ba6..2ca4892 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -25,6 +25,7 @@ import argparse import json +import os import re import shutil import subprocess @@ -327,48 +328,78 @@ def _summarize(paths: list[Path], limit: int = 3) -> str: MANIFEST_NAME = ".prep-manifest.json" -def _read_manifest(out_dir: Path) -> tuple[str | None, set[str]]: - """The label and clip names a previous prep run recorded here. +def _source_key(path: Path) -> str: + """A stable local identity for one source recording.""" + return os.path.normcase(str(path.resolve())) + + +def _read_manifest( + out_dir: Path, +) -> tuple[str | None, set[str], dict[str, set[str]] | None]: + """The label, clips, and source ownership a previous prep run recorded here. Fails soft on every read problem -- absent, unreadable, not JSON, wrong - shape -- returning ``(None, set())``. An empty record means "no record of - what prep made," which narrows `--clean` back to the stem rule below. - Failing loud would be worse in both directions: a corrupt manifest would - abort a run that has real work to do, and a manifest trusted despite a - partial read could name files prep never wrote. + shape -- returning ``(None, set(), None)``. An empty record means "no record + of what prep made," which narrows `--clean` back to the stem rule below. + A version-1 record still returns its label and clips but no source map; the + caller can then refuse to guess ownership and tell the user how to bootstrap + a new record explicitly. """ try: raw = json.loads((out_dir / MANIFEST_NAME).read_text()) except (OSError, ValueError): - return None, set() + return None, set(), None if not isinstance(raw, dict): - return None, set() + return None, set(), None clips = raw.get("clips") if not isinstance(clips, list): - return None, set() + return None, set(), None label = raw.get("label") - return ( - label if isinstance(label, str) else None, - {c for c in clips if isinstance(c, str)}, - ) - - -def _write_manifest(out_dir: Path, label: str, names: set[str]) -> None: - """Record which files in `out_dir` prep created, for a later --clean run. + clip_names = {c for c in clips if isinstance(c, str)} + sources_raw = raw.get("sources") + if not isinstance(sources_raw, dict): + return label if isinstance(label, str) else None, clip_names, None + + sources: dict[str, set[str]] = {} + for source, names in sources_raw.items(): + if not isinstance(source, str) or not isinstance(names, list): + return label if isinstance(label, str) else None, clip_names, None + sources[source] = {name for name in names if isinstance(name, str)} + owned_clips = set().union(*sources.values()) if sources else set() + if owned_clips != clip_names: + return label if isinstance(label, str) else None, clip_names, None + return label if isinstance(label, str) else None, clip_names, sources + + +def _write_manifest( + out_dir: Path, label: str, source_clips: dict[str, set[str]] +) -> None: + """Record each source and the clips it owns, for a later --clean run. Written after the clips are on disk, and a failure only warns: the clips are the output that matters, and losing the manifest costs precision on a future --clean, not correctness. A run that wrote nothing into a directory that does not exist has nothing to record and does not create one. - The caller decides whether this run may claim the directory (see - ``refreshes_prior``); this only writes. + The caller verifies ownership and assembles the source map; this only writes. """ if not out_dir.is_dir(): return try: + clips = set().union(*source_clips.values()) if source_clips else set() (out_dir / MANIFEST_NAME).write_text( - json.dumps({"version": 1, "label": label, "clips": sorted(names)}, indent=2) + json.dumps( + { + "version": 2, + "label": label, + "clips": sorted(clips), + "sources": { + source: sorted(names) + for source, names in sorted(source_clips.items()) + }, + }, + indent=2, + ) + "\n" ) except OSError as e: @@ -400,11 +431,11 @@ def process( writes, so its count matches what is on disk. Two kinds qualify: a re-recorded take's now-unused higher-numbered clips (matched by stem), and clips a previous run recorded for a take that is not in this run's inputs at - all (matched against ``MANIFEST_NAME``, and only when this run refreshes the - recorded set -- same label, overlapping stems). Everything else is named but - never deleted -- a source recording, hand-added audio, another dataset a - stray ``--output`` landed in, and the prior clips of a take this run - attempted but wrote nothing for. + all (matched against ``MANIFEST_NAME``, and only when this run contains a + source path recorded for that dataset). Everything else is named but never + deleted -- a source recording, hand-added audio, another dataset a stray + ``--output`` landed in, and the prior clips of a take this run attempted but + wrote nothing for. """ # Checked before any audio is decoded, so an occupied --output is named once # rather than surfacing as whichever exception the run happens to reach first: @@ -423,7 +454,7 @@ def process( # this directory's take_000.wav with the new dataset's. A label mismatch is # the signature of a mistyped --output, and there is no reading of it where # writing positives into a directory of negatives is what the user meant. - prior_label, prior = _read_manifest(out_dir) + prior_label, _prior_clips, prior_sources = _read_manifest(out_dir) if prior_label is not None and prior_label != label: print( f"error: --output {out_dir} holds {prior_label!r} clips, not {label!r}; " @@ -432,6 +463,15 @@ def process( file=sys.stderr, ) return 0 + if prior_label is not None and prior_sources is None: + print( + f"error: --output {out_dir} has a {MANIFEST_NAME} without source " + "ownership, so this run cannot prove it belongs there; nothing was " + f"written. Remove {out_dir / MANIFEST_NAME} to bootstrap ownership " + "from these inputs, or clear the directory first", + file=sys.stderr, + ) + return 0 files = _inputs(inputs) if not files: @@ -439,22 +479,42 @@ def process( print(f"no audio files found at {joined}", file=sys.stderr) return 0 + source_keys = [_source_key(f) for f in files] + # This guards writes as well as --clean: _write_unique intentionally refreshes + # prior filenames in place, so a different source with the same basename could + # overwrite take_000.wav before cleanup runs. Add a new source alongside one + # recorded source in the same invocation to prove which dataset owns the dir. + if prior_sources is not None and not (set(source_keys) & prior_sources.keys()): + print( + f"error: --output {out_dir} belongs to a {label!r} dataset with " + "different source recordings; nothing was written. Point --output at " + f"this run's directory, or clear that one (or remove its {MANIFEST_NAME}) " + "to reuse it", + file=sys.stderr, + ) + return 0 + stems = _unique_stems(files) written: set[Path] = set() + written_by_source: dict[str, set[Path]] = {source: set() for source in source_keys} seen_inodes: set[int] = set() all_durs: list[float] = [] - for f, stem in zip(files, stems): + for f, stem, source in zip(files, stems, source_keys): audio = load_mono_16k(f) dur = len(audio) / TARGET_SR if label == "background": # Continuous negative audio: keep it whole, just normalized. - written.add(_write_unique(out_dir, stem, 0, audio, seen_inodes)) + path = _write_unique(out_dir, stem, 0, audio, seen_inodes) + written.add(path) + written_by_source[source].add(path) print(f" {f.name}: {dur:.1f}s background -> 1 file") continue clips, stats = segment_on_silence(audio, min_gap_s, min_dur_s, max_dur_s) for i, clip in enumerate(clips): - written.add(_write_unique(out_dir, stem, i, clip, seen_inodes)) + path = _write_unique(out_dir, stem, i, clip, seen_inodes) + written.add(path) + written_by_source[source].add(path) all_durs.append(len(clip) / TARGET_SR) note = "" if stats["too_short"] or stats["too_long"]: @@ -466,100 +526,89 @@ def process( total = len(written) print(f"\nwrote {total} {label} clip(s) to {out_dir}") - # Files this run did not write are left from an earlier run over a different - # input set. Training and eval read every clip in the directory, so they - # would score as extra data nobody asked for. --clean removes the leftover - # clips of a take this run just re-recorded (same stem, now-unused index), so - # the reported count matches what is on disk. It is deliberately narrow: a - # clip whose stem this run did not write -- an unrelated take, a source - # recording, hand-added audio -- is named but never auto-deleted, because it - # is indistinguishable from data the user curated on purpose. + # Files this run did not write are left from an earlier input set. Once a + # manifest exists, cleanup is based on source ownership rather than inferred + # from output stems: prior clips belonging to a source this run successfully + # re-recorded, or to a source dropped from this run, are removable. Prior clips + # belonging to a source this run attempted but got no clips from are kept as + # the only good copy. Files absent from the source map are user-owned. # - # Scope by the stems this run actually WROTE a clip for, read back from - # `written`, not by the input list `stems`: an input that produced zero clips - # this run (silence, a bad re-recording, thresholds that dropped every - # segment) has no new clip to make its earlier good clips stale, so --clean - # must not delete them. Keying off written paths leaves them to linger and - # warn instead of wiping the only copy. + # Without a readable manifest, the older stem rule remains the narrow + # fallback. It can clean a re-recorded take's higher-numbered leftovers but + # cannot reach a dropped input safely. run_stems = { stem for stem in (_clip_stem(p.name) for p in written) if stem is not None } - # The manifest widens --clean to the orphans the stem rule cannot reach: a - # clip prep recorded writing on an earlier run, whose stem is not in this - # run's inputs at all. That covers a dropped input and a basename collision - # whose disambiguated stem changed -- both leave clips no rerun will ever - # overwrite, and the manifest is what proves prep made them rather than a - # user curating the directory. - # - # But "prep made this file" is provenance, not permission. It does not say - # the file belongs to the dataset this run is refreshing, and without that - # second fact a mistyped --output -- positives re-run into the background - # directory -- would delete every clip there, since none of those stems are - # in the inputs. So the manifest rule only arms when this run looks like a - # refresh of the recorded set: same label, and at least one stem in common. - # A stray --output shares neither, and falls back to the stem rule, which - # cannot reach outside what this run just wrote. - # - # A stem this run DID attempt but wrote nothing for is excluded on purpose, - # manifest or not: that is the silent-re-recording case, where the prior - # clips are the only copy. The manifest says prep created a file; it does - # not say the file is expendable. - attempted = set(stems) - prior_stems = {stem for stem in (_clip_stem(n) for n in prior) if stem is not None} - # A mismatched label was already refused above, so the question left is - # whether this run is refreshing the record it found or landed beside it: a - # directory with no record is unclaimed and counts as ours, and one with a - # record needs a take in common. Without the overlap, prep neither deletes nor - # writes -- folding this run's stems into that record would hand a repeat of - # the same mistake the overlap it needs to start deleting. - refreshes_prior = not prior or bool(run_stems & prior_stems) + current_sources = set(source_keys) + written_sources = {source for source, paths in written_by_source.items() if paths} stale = _stale_clips(out_dir, written) - removable = ( - [ - p - for p in stale - if refreshes_prior - and ( - # A re-recorded take's now-unused clips. Once there is a record, - # this consults it too: without that, a hand-added take_003.wav - # beside a manifested `take` dataset would be deleted by shape - # alone, and the promise that --clean only removes what prep made - # would hold everywhere except the case a user is most likely to - # hit. With no record (a directory from before manifests), shape - # is all there is. - (_clip_stem(p.name) in run_stems and (not prior or p.name in prior)) - # An orphan of a take this run does not have: a dropped input, or - # a collider whose disambiguated stem changed. - or (p.name in prior and _clip_stem(p.name) not in attempted) + removable: list[Path] = [] + if clean: + if prior_sources is None: + removable = [p for p in stale if _clip_stem(p.name) in run_stems] + else: + replaceable_sources = { + source + for source in prior_sources + if source not in current_sources or source in written_sources + } + removable_names = set().union( + *(prior_sources[source] for source in replaceable_sources) ) - ] - if clean - else [] - ) - for p in removable: - p.unlink() - if removable: - print( - f"removed {len(removable)} clip(s) left from an earlier run " - f"({_summarize(removable)})" - ) - lingering = [p for p in stale if p not in removable] - # Carry forward the prior run's record for files still on disk, so a - # directory built up over several runs keeps one provenance list rather than - # only remembering the most recent run. - if refreshes_prior: - _write_manifest( - out_dir, - label, - {p.name for p in written} | {p.name for p in lingering if p.name in prior}, - ) - elif out_dir.is_dir(): + removable = [p for p in stale if p.name in removable_names] + removed: list[Path] = [] + for path in removable: + try: + path.unlink() + except OSError as e: + print( + f"warning: could not remove {path} ({e}); leaving it recorded for " + "a later --clean", + file=sys.stderr, + ) + else: + removed.append(path) + if removed: print( - f"warning: {out_dir} holds a {label!r} dataset with no take in common " - f"with this run, so its record was left as it was. Clips this run wrote " - f"are on disk; a later --clean will not treat them as part of it", - file=sys.stderr, + f"removed {len(removed)} clip(s) left from an earlier run " + f"({_summarize(removed)})" ) + lingering = [p for p in stale if p not in removed] + lingering_names = {p.name for p in lingering} + + # Carry forward every still-present owned clip under its original source, + # then replace or add the clips written for current sources. On the first + # manifest run, claim legacy leftovers only when their stem maps to a source + # that successfully wrote this run; that preserves the old cleanup fallback + # and keeps a silent take from claiming unknown files. + next_sources: dict[str, set[str]] = {} + if prior_sources is not None: + for source, names in prior_sources.items(): + kept = names & lingering_names + if kept: + next_sources[source] = kept + else: + source_for_stem = { + stem: source + for stem, source in zip(stems, source_keys) + if source in written_sources + } + for path in lingering: + source = source_for_stem.get(_clip_stem(path.name)) + if source is not None: + next_sources.setdefault(source, set()).add(path.name) + # Keep current sources in the record even when they wrote nothing. If their + # old clips are already absent, the empty entry still proves the next rerun + # comes from the same source instead of deadlocking the directory. + for source, paths in written_by_source.items(): + next_sources.setdefault(source, set()).update(path.name for path in paths) + + # A no-output run cannot establish ownership in an unrecorded directory. + # Leaving it unclaimed lets a later successful rerun bootstrap the legacy + # clips instead of stranding them behind an empty manifest. + if prior_sources is not None or written: + _write_manifest(out_dir, label, next_sources) + if lingering: # Why a file lingered decides what the user should do about it, and the # cases want opposite advice. A take this run re-recorded and got nothing @@ -568,7 +617,20 @@ def process( # recording. Split the message rather than send one that is right for the # common case and harmful for the case prep went out of its way to # protect. - empty_takes = [p for p in lingering if _clip_stem(p.name) in attempted] + if prior_sources is None: + attempted = set(stems) + empty_takes = [ + p for p in lingering if _clip_stem(p.name) in attempted - run_stems + ] + else: + silent_names = set().union( + *( + prior_sources[source] + for source in current_sources - written_sources + if source in prior_sources + ) + ) + empty_takes = [p for p in lingering if p.name in silent_names] if empty_takes and clean: fix = ( "this run re-recorded " @@ -634,7 +696,8 @@ def main() -> int: action="store_true", help="remove prep's own leftover clips in --output: a re-recorded take's " "now-unused higher-numbered clips, plus clips a previous run recorded for a " - "take no longer in the inputs; leaves hand-added audio in place", + "take no longer in the inputs; without a readable manifest, falls back to " + "same-stem filenames", ) args = p.parse_args() diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 30aa472..5686f57 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -8,6 +8,7 @@ from __future__ import annotations +import json import sys import tempfile from pathlib import Path @@ -456,7 +457,13 @@ def test_refresh_after_adding_collider_leaves_no_orphans(d: Path) -> None: out = d / "refresh_out" prep.process([a / "computah.wav"], out, "positive", 0.3, 0.2, 3.0) prep.process( - [a / "computah.wav", b / "computah.wav"], out, "positive", 0.3, 0.2, 3.0 + [a / "computah.wav", b / "computah.wav"], + out, + "positive", + 0.3, + 0.2, + 3.0, + clean=True, ) files = sorted(p.name for p in out.glob("*.wav")) @@ -589,6 +596,39 @@ def test_manifest_cleans_changed_collider_stem_orphans(d: Path) -> None: ) +def test_manifest_tracks_a_silent_colliding_source_by_path(d: Path) -> None: + """A silent rerun keeps its own prior clips even when its stem changes. + + Two same-basename inputs need source-level ownership: when only the second + source returns and produces no clips, its old disambiguated clips are the + copy to preserve, while the dropped first source's clips can be cleaned. + Stem membership alone cannot tell those cases apart. + """ + src = d / "silent_collider_src" + first = src / "a" / "take.wav" + silent = src / "b" / "take.wav" + _burst_take(first, 3) + _burst_take(silent, 2) + out = d / "silent_collider_out" + prep.process([first, silent], out, "positive", 0.3, 0.2, 3.0) + + _label, _clips, sources = prep._read_manifest(out) + source_keys = {prep._source_key(first), prep._source_key(silent)} + check( + sources is not None and set(sources) == source_keys, + "the manifest records exact source ownership", + ) + silent_clips = sources[prep._source_key(silent)] if sources is not None else set() + + total = prep.process([silent], out, "positive", 0.3, 5.0, 10.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + total == 0 and present == sorted(silent_clips), + f"a silent colliding source keeps its clips while the dropped source cleans " + f"({present})", + ) + + def test_manifest_never_authorizes_deleting_curated_audio(d: Path) -> None: """The manifest widens --clean only to files prep recorded writing. @@ -634,6 +674,25 @@ def test_manifest_spares_prior_clips_when_rerun_writes_nothing(d: Path) -> None: ) +def test_manifest_empty_source_remains_refreshable(d: Path) -> None: + """An owned source with no clips must not deadlock its output directory.""" + src = d / "empty_source_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "empty_source_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + for clip in out.glob("*.wav"): + clip.unlink() + + empty = prep.process([take], out, "positive", 0.3, 5.0, 10.0, clean=True) + refreshed = prep.process([take], out, "positive", 0.3, 0.2, 3.0) + present = sorted(p.name for p in out.glob("*.wav")) + check( + empty == 0 and refreshed == 3 and len(present) == 3, + f"an empty owned source accepts its next successful refresh ({present})", + ) + + def test_manifest_spares_a_silent_take_alongside_a_good_one(d: Path) -> None: """The `attempted` guard, with the manifest rule actually armed. @@ -705,6 +764,30 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: ) +def test_manifest_without_source_ownership_is_refused(d: Path) -> None: + """A readable legacy record cannot prove which dataset owns the output.""" + src = d / "legacy_manifest_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "legacy_manifest_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + + manifest_path = out / prep.MANIFEST_NAME + manifest = json.loads(manifest_path.read_text()) + manifest["version"] = 1 + manifest.pop("sources") + manifest_path.write_text(json.dumps(manifest)) + before = sorted(p.name for p in out.glob("*.wav")) + + _burst_take(take, 1) + total = prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + total == 0 and present == before, + f"a manifest without source ownership cannot authorize a write ({present})", + ) + + def test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d: Path) -> None: """The first manifest must not strand clips from a pre-manifest run. @@ -902,10 +985,13 @@ def main() -> int: test_stem_assignment_ignores_input_order(d) test_manifest_cleans_dropped_input_orphans(d) test_manifest_cleans_changed_collider_stem_orphans(d) + test_manifest_tracks_a_silent_colliding_source_by_path(d) test_manifest_never_authorizes_deleting_curated_audio(d) test_manifest_spares_prior_clips_when_rerun_writes_nothing(d) + test_manifest_empty_source_remains_refreshable(d) test_manifest_spares_a_silent_take_alongside_a_good_one(d) test_manifest_unreadable_falls_back_to_stem_rule(d) + test_manifest_without_source_ownership_is_refused(d) test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d) test_manifest_spares_a_different_dataset_on_stray_output(d) test_same_label_shared_stem_stray_is_refused_before_writing(d) From b35b9636df5337a507f675adda19cbb6733b42e1 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:28:27 -0400 Subject: [PATCH 06/38] fix(prep): harden manifest ownership recovery --- CHANGELOG.md | 16 +++--- docs/wake-threshold-tuning.md | 18 ++++--- prep_wake_samples.py | 93 +++++++++++++++++++++++------------ test_prep_wake_samples.py | 43 ++++++++++++++-- 4 files changed, 121 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index db7e4fe..82202c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,13 +27,15 @@ All notable changes to computah are recorded here. The format follows A source this run attempted but got no clips from retains its prior clips as the only good copy, including when same-basename inputs changed its output stem. With a readable source map, hand-curated audio stays outside it and is never - deleted. The first successful manifest run adopts same-stem leftovers from a - pre-manifest run so a warned-about orphan remains cleanable; a no-output run - cannot claim an unrecorded directory. An absent or unreadable manifest degrades - `--clean` to the narrow stem match, which cannot distinguish a prep leftover - from a hand-added `_NNN.wav`. A readable older manifest without - source ownership is refused until the user removes it to bootstrap ownership - explicitly. + deleted. A non-clean legacy refresh that leaves ambiguous same-stem files does + not write a manifest; the next `--clean` remains on the documented filename + fallback instead of turning a guess into durable provenance. A no-output run + cannot claim an unrecorded directory. Manifest writes use atomic replacement, + and an absent or unreadable manifest degrades `--clean` to the narrow stem + match with an explicit warning. That fallback cannot distinguish a prep + leftover from a hand-added `_NNN.wav`. A readable older manifest + without source ownership is refused until the user removes it to bootstrap + ownership explicitly. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 40f130d..9e76e26 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -49,18 +49,22 @@ and waiting until `--clean` would be too late, since the run would already have overwritten that dataset's clips. Past that, a run must contain at least one exact source path from the record before it may delete from the dataset or write to it. A shared basename is not ownership. A directory with no record yet counts -as yours; its first successful run also records same-stem legacy leftovers so a -follow-up `--clean` can remove them. A run that produces no clips does not claim -an unrecorded directory. Prep also spares the prior clips of a source it tried -and got nothing from (a silent or bad re-recording): those are the only copy, and -it will not trade them for a run that produced nothing. +as yours. If a non-clean legacy refresh leaves same-stem files behind, prep does +not write a manifest because it cannot prove whether those files are old prep +output or audio curated by hand. The follow-up `--clean` stays on the legacy +filename rule rather than recording that guess as provenance. A run that +produces no clips does not claim an unrecorded directory. Prep also spares the +prior clips of a source it tried and got nothing from (a silent or bad +re-recording): those are the only copy, and it will not trade them for a run +that produced nothing. To add a new recording to an existing dataset, include at least one source already recorded there in the same invocation. This gives prep explicit ownership proof while it adds the new source to the manifest. -An absent or unreadable manifest falls back to the narrow same-stem cleanup -rule. That fallback has no provenance: `--clean` can remove any +Manifest updates use atomic replacement. An absent or unreadable manifest falls +back to the narrow same-stem cleanup rule with a warning that source ownership +protection is unavailable. That fallback has no provenance: `--clean` can remove any `_NNN.wav` file, including a hand-added file with that shape. Check the warning list and move any such file before cleaning. A readable older manifest without source ownership is refused; after confirming the output directory, diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 2ca4892..e31f867 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -345,9 +345,17 @@ def _read_manifest( caller can then refuse to guess ownership and tell the user how to bootstrap a new record explicitly. """ + manifest_path = out_dir / MANIFEST_NAME try: - raw = json.loads((out_dir / MANIFEST_NAME).read_text()) - except (OSError, ValueError): + raw = json.loads(manifest_path.read_text()) + except FileNotFoundError: + return None, set(), None + except (OSError, ValueError) as e: + print( + f"warning: could not read {manifest_path} ({e}); source ownership " + "protection is unavailable, so this run will use legacy filename cleanup", + file=sys.stderr, + ) return None, set(), None if not isinstance(raw, dict): return None, set(), None @@ -385,29 +393,49 @@ def _write_manifest( """ if not out_dir.is_dir(): return - try: - clips = set().union(*source_clips.values()) if source_clips else set() - (out_dir / MANIFEST_NAME).write_text( - json.dumps( - { - "version": 2, - "label": label, - "clips": sorted(clips), - "sources": { - source: sorted(names) - for source, names in sorted(source_clips.items()) - }, + clips = set().union(*source_clips.values()) if source_clips else set() + payload = ( + json.dumps( + { + "version": 2, + "label": label, + "clips": sorted(clips), + "sources": { + source: sorted(names) + for source, names in sorted(source_clips.items()) }, - indent=2, - ) - + "\n" + }, + indent=2, ) + + "\n" + ) + temporary: Path | None = None + try: + with tempfile.NamedTemporaryFile( + mode="w", + encoding="utf-8", + dir=out_dir, + prefix=f".{MANIFEST_NAME}.", + suffix=".tmp", + delete=False, + ) as tmp: + temporary = Path(tmp.name) + tmp.write(payload) + tmp.flush() + os.fsync(tmp.fileno()) + os.replace(temporary, out_dir / MANIFEST_NAME) except OSError as e: print( f"warning: could not write {out_dir / MANIFEST_NAME} ({e}); " f"a later --clean will fall back to matching re-recorded stems only", file=sys.stderr, ) + finally: + if temporary is not None: + try: + temporary.unlink(missing_ok=True) + except OSError: + pass def _clip_stem(name: str) -> str | None: @@ -575,28 +603,22 @@ def process( ) lingering = [p for p in stale if p not in removed] lingering_names = {p.name for p in lingering} + ambiguous_legacy = prior_sources is None and any( + _clip_stem(path.name) in run_stems for path in lingering + ) # Carry forward every still-present owned clip under its original source, - # then replace or add the clips written for current sources. On the first - # manifest run, claim legacy leftovers only when their stem maps to a source - # that successfully wrote this run; that preserves the old cleanup fallback - # and keeps a silent take from claiming unknown files. + # then replace or add the clips written for current sources. A no-manifest + # run cannot claim same-stem leftovers: they may be old prep output or audio + # curated by hand. Leave the record absent until legacy --clean resolves + # that ambiguity, so v2 never turns an inferred filename match into durable + # provenance. next_sources: dict[str, set[str]] = {} if prior_sources is not None: for source, names in prior_sources.items(): kept = names & lingering_names if kept: next_sources[source] = kept - else: - source_for_stem = { - stem: source - for stem, source in zip(stems, source_keys) - if source in written_sources - } - for path in lingering: - source = source_for_stem.get(_clip_stem(path.name)) - if source is not None: - next_sources.setdefault(source, set()).add(path.name) # Keep current sources in the record even when they wrote nothing. If their # old clips are already absent, the empty entry still proves the next rerun # comes from the same source instead of deadlocking the directory. @@ -606,8 +628,15 @@ def process( # A no-output run cannot establish ownership in an unrecorded directory. # Leaving it unclaimed lets a later successful rerun bootstrap the legacy # clips instead of stranding them behind an empty manifest. - if prior_sources is not None or written: + if (prior_sources is not None or written) and not ambiguous_legacy: _write_manifest(out_dir, label, next_sources) + elif ambiguous_legacy: + print( + f"warning: {MANIFEST_NAME} was not written because same-stem legacy " + "files remain and prep cannot prove whether it created them; review " + "the warning below before rerunning with --clean", + file=sys.stderr, + ) if lingering: # Why a file lingered decides what the user should do about it, and the diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 5686f57..44cefe0 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -8,9 +8,11 @@ from __future__ import annotations +import io import json import sys import tempfile +from contextlib import redirect_stderr from pathlib import Path import numpy as np @@ -754,14 +756,21 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: (out / prep.MANIFEST_NAME).write_text("{not json") _burst_take(src / "a" / "take.wav", 2) - total = prep.process( - [src / "a" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True - ) + errors = io.StringIO() + with redirect_stderr(errors): + total = prep.process( + [src / "a" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True + ) present = sorted(p.name for p in out.glob("*.wav")) check( total == 2 and "take_002.wav" not in present and "other_000.wav" in present, f"a corrupt manifest leaves the stem rule working ({present})", ) + check( + "manifest" in errors.getvalue() + and "legacy filename cleanup" in errors.getvalue(), + "a corrupt manifest warns that ownership protection is unavailable", + ) def test_manifest_without_source_ownership_is_refused(d: Path) -> None: @@ -814,6 +823,33 @@ def test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d: Path) -> None: ) +def test_manifest_bootstrap_does_not_adopt_ambiguous_same_stem_audio( + d: Path, +) -> None: + """A non-clean legacy refresh must not claim ambiguous clips as prep-owned. + + Before a manifest exists, a same-stem clip can be a prep leftover or audio + curated by hand. Leaving the directory on the documented legacy path keeps + that ambiguity visible; recording the clip in v2 would make a later + source-aware --clean delete it under a false provenance claim. + """ + src = d / "bootstrap_curated_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "bootstrap_curated_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + (out / prep.MANIFEST_NAME).unlink() + _burst_take(out / "take_007.wav", 1) + + _burst_take(take, 2) + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + present = sorted(p.name for p in out.glob("*.wav")) + check( + not (out / prep.MANIFEST_NAME).exists() and "take_007.wav" in present, + f"ambiguous same-stem audio stays outside a source-aware record ({present})", + ) + + def test_manifest_spares_a_different_dataset_on_stray_output(d: Path) -> None: """A mistyped --output must not let the manifest wipe another dataset. @@ -993,6 +1029,7 @@ def main() -> int: test_manifest_unreadable_falls_back_to_stem_rule(d) test_manifest_without_source_ownership_is_refused(d) test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d) + test_manifest_bootstrap_does_not_adopt_ambiguous_same_stem_audio(d) test_manifest_spares_a_different_dataset_on_stray_output(d) test_same_label_shared_stem_stray_is_refused_before_writing(d) test_clean_disarmed_entirely_on_a_label_mismatch(d) From 37a6ce1adf577422b431ae3289446cca64aad2b2 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:31:43 -0400 Subject: [PATCH 07/38] docs(prep): clarify safe clean source set --- docs/wake-threshold-tuning.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 9e76e26..fcad75d 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -58,9 +58,13 @@ prior clips of a source it tried and got nothing from (a silent or bad re-recording): those are the only copy, and it will not trade them for a run that produced nothing. -To add a new recording to an existing dataset, include at least one source -already recorded there in the same invocation. This gives prep explicit -ownership proof while it adds the new source to the manifest. +To add a new recording to an existing dataset without `--clean`, include at +least one source already recorded there in the same invocation. This gives prep +explicit ownership proof while it adds the new source to the manifest. If you +use `--clean`, include every source whose clips the dataset should keep: omitted +sources are treated as intentionally dropped and their prep-owned clips are +removed. A safe incremental workflow is to add recordings without `--clean`, +then run a full-input `--clean` pass after reviewing the complete source list. Manifest updates use atomic replacement. An absent or unreadable manifest falls back to the narrow same-stem cleanup rule with a warning that source ownership From c1db2debd9de428b6bec18a5e0faeeeb7cc71403 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:38:35 -0400 Subject: [PATCH 08/38] test(prep): reproduce remaining manifest collisions --- test_prep_wake_samples.py | 66 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 44cefe0..594a8b0 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -465,7 +465,6 @@ def test_refresh_after_adding_collider_leaves_no_orphans(d: Path) -> None: 0.3, 0.2, 3.0, - clean=True, ) files = sorted(p.name for p in out.glob("*.wav")) @@ -631,6 +630,49 @@ def test_manifest_tracks_a_silent_colliding_source_by_path(d: Path) -> None: ) +def test_new_collider_cannot_overwrite_a_silent_sources_prior_stem(d: Path) -> None: + """A new collider must not take a silent source's manifested filenames. + + Cleanup is too late to protect the only good clips if another input has + already refreshed their paths in place. Manifested stems therefore stay + reserved for their source while output names are assigned. + """ + src = d / "stem_theft_src" + first = src / "a" / "take.wav" + silent = src / "b" / "take.wav" + newcomer = src / "c" / "take.wav" + _burst_take(first, 3) + _burst_take(silent, 2) + out = d / "stem_theft_out" + prep.process([first, silent], out, "positive", 0.3, 0.2, 3.0) + + _label, _clips, sources = prep._read_manifest(out) + silent_key = prep._source_key(silent) + silent_names = sources[silent_key] if sources is not None else set() + silent_bytes = {name: (out / name).read_bytes() for name in silent_names} + + sf.write( + silent, + np.ones(int(prep.TARGET_SR * 4.0), dtype=np.float32) * 0.5, + prep.TARGET_SR, + subtype="PCM_16", + ) + _burst_take(newcomer, 3) + prep.process([silent, newcomer], out, "positive", 0.3, 0.2, 1.0, clean=True) + _label, _clips, refreshed_sources = prep._read_manifest(out) + present = sorted(p.name for p in out.glob("*.wav")) + preserved = all( + (out / name).exists() and (out / name).read_bytes() == payload + for name, payload in silent_bytes.items() + ) + check( + preserved + and refreshed_sources is not None + and refreshed_sources.get(silent_key) == silent_names, + f"a newcomer cannot overwrite a silent source's only good clips ({present})", + ) + + def test_manifest_never_authorizes_deleting_curated_audio(d: Path) -> None: """The manifest widens --clean only to files prep recorded writing. @@ -773,6 +815,26 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: ) +def test_manifest_wrong_shape_warns_before_legacy_fallback(d: Path) -> None: + """Readable JSON with an invalid manifest shape must not degrade silently.""" + src = d / "malformed_manifest_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "malformed_manifest_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + (out / prep.MANIFEST_NAME).write_text("[]\n") + + _burst_take(take, 2) + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) + check( + "manifest" in errors.getvalue() + and "legacy filename cleanup" in errors.getvalue(), + "a wrong-shape manifest warns before ownership protection degrades", + ) + + def test_manifest_without_source_ownership_is_refused(d: Path) -> None: """A readable legacy record cannot prove which dataset owns the output.""" src = d / "legacy_manifest_src" @@ -1022,11 +1084,13 @@ def main() -> int: test_manifest_cleans_dropped_input_orphans(d) test_manifest_cleans_changed_collider_stem_orphans(d) test_manifest_tracks_a_silent_colliding_source_by_path(d) + test_new_collider_cannot_overwrite_a_silent_sources_prior_stem(d) test_manifest_never_authorizes_deleting_curated_audio(d) test_manifest_spares_prior_clips_when_rerun_writes_nothing(d) test_manifest_empty_source_remains_refreshable(d) test_manifest_spares_a_silent_take_alongside_a_good_one(d) test_manifest_unreadable_falls_back_to_stem_rule(d) + test_manifest_wrong_shape_warns_before_legacy_fallback(d) test_manifest_without_source_ownership_is_refused(d) test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d) test_manifest_bootstrap_does_not_adopt_ambiguous_same_stem_audio(d) From fbe805c1d2fffb200e7480f7b5ef03225f95014a Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:41:52 -0400 Subject: [PATCH 09/38] fix(prep): reserve manifested stems before writes --- CHANGELOG.md | 25 +++++----- docs/wake-threshold-tuning.md | 4 ++ prep_wake_samples.py | 89 ++++++++++++++++++++++++++++++++--- test_prep_wake_samples.py | 13 ++--- 4 files changed, 107 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82202c0..7f426dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,18 +24,19 @@ All notable changes to computah are recorded here. The format follows (`take-1_000.wav`) that a later run no longer produces. A run must share an exact recorded source path before it may write to that dataset, so a different dataset with the same label and generic basename cannot overwrite or clean it. - A source this run attempted but got no clips from retains its prior clips as the - only good copy, including when same-basename inputs changed its output stem. - With a readable source map, hand-curated audio stays outside it and is never - deleted. A non-clean legacy refresh that leaves ambiguous same-stem files does - not write a manifest; the next `--clean` remains on the documented filename - fallback instead of turning a guess into durable provenance. A no-output run - cannot claim an unrecorded directory. Manifest writes use atomic replacement, - and an absent or unreadable manifest degrades `--clean` to the narrow stem - match with an explicit warning. That fallback cannot distinguish a prep - leftover from a hand-added `_NNN.wav`. A readable older manifest - without source ownership is refused until the user removes it to bootstrap - ownership explicitly. + Manifested stems remain reserved for their source while new output names are + assigned, so a new same-basename recording cannot overwrite the only good + clips from a source whose rerun is silent. A source this run attempted but got + no clips from retains those prior clips. With a readable source map, + hand-curated audio stays outside it and is never deleted. A non-clean legacy + refresh that leaves ambiguous same-stem files does not write a manifest; the + next `--clean` remains on the documented filename fallback instead of turning + a guess into durable provenance. A no-output run cannot claim an unrecorded + directory. Manifest writes use atomic replacement, and an absent or unreadable + manifest degrades `--clean` to the narrow stem match with an explicit warning. + That fallback cannot distinguish a prep leftover from a hand-added + `_NNN.wav`. A readable older manifest without source ownership is + refused until the user removes it to bootstrap ownership explicitly. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index fcad75d..8d2452e 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -41,6 +41,10 @@ re-recorded, and the clips a previous run recorded for a take that is no longer in the inputs. The `.prep-manifest.json` in the output dir maps each resolved source path to the clips prep wrote for it. A source recording, a hand-curated clip, and anything else absent from that map is named but left in place. +Recorded output stems also remain reserved for their source while the next run +assigns names. A new same-basename recording therefore cannot overwrite another +source's manifested clips before `--clean` has a chance to apply its ownership +rules. That record proves prep made a file, not that the file belongs to what you are refreshing. A `--label` that disagrees with the record already in the directory diff --git a/prep_wake_samples.py b/prep_wake_samples.py index e31f867..a793771 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -346,21 +346,27 @@ def _read_manifest( a new record explicitly. """ manifest_path = out_dir / MANIFEST_NAME + + def warn_legacy_fallback(reason: object) -> None: + print( + f"warning: could not use {manifest_path} ({reason}); source ownership " + "protection is unavailable, so this run will use legacy filename cleanup", + file=sys.stderr, + ) + try: raw = json.loads(manifest_path.read_text()) except FileNotFoundError: return None, set(), None except (OSError, ValueError) as e: - print( - f"warning: could not read {manifest_path} ({e}); source ownership " - "protection is unavailable, so this run will use legacy filename cleanup", - file=sys.stderr, - ) + warn_legacy_fallback(e) return None, set(), None if not isinstance(raw, dict): + warn_legacy_fallback("expected a JSON object") return None, set(), None clips = raw.get("clips") if not isinstance(clips, list): + warn_legacy_fallback("expected a clips list") return None, set(), None label = raw.get("label") clip_names = {c for c in clips if isinstance(c, str)} @@ -444,6 +450,77 @@ def _clip_stem(name: str) -> str | None: return m.group(1) if m else None +def _manifest_aware_stems( + files: list[Path], + source_keys: list[str], + prior_sources: dict[str, set[str]] | None, +) -> list[str]: + """Assign output stems without taking names owned by another source. + + The first run uses the path-stable basename rule. Once a source manifest + exists, a returning source keeps one of its recorded stems when possible, + and every prior stem stays reserved from other sources. That reservation is + needed before any clip is written: cleanup cannot save a silent source's + only good clips after a newcomer has already overwritten their filenames. + """ + generated = _unique_stems(files) + if prior_sources is None: + return generated + + source_stems: dict[str, set[str]] = {} + owners_by_stem: dict[str, set[str]] = {} + for source, names in prior_sources.items(): + stems = {stem for name in names if (stem := _clip_stem(name)) is not None} + source_stems[source] = stems + for stem in stems: + owners_by_stem.setdefault(stem, set()).add(source) + + assigned: dict[int, str] = {} + used: set[str] = set() + + # Anchor returning sources to a stem they already own. Prefer the stem the + # current basename rule would choose, then a sole prior stem. A malformed + # record in which two sources share a stem is never used as an anchor. + for i in sorted(range(len(files)), key=lambda j: str(files[j].resolve())): + source = source_keys[i] + owned = source_stems.get(source, set()) + preferred: str | None = None + if generated[i] in owned and owners_by_stem.get(generated[i]) == {source}: + preferred = generated[i] + elif len(owned) == 1: + candidate = next(iter(owned)) + if owners_by_stem.get(candidate) == {source}: + preferred = candidate + if preferred is not None and preferred not in used: + assigned[i] = preferred + used.add(preferred) + + # Keep the ordinary mapping for inputs whose candidate has no recorded + # owner. Anchors run first so a newcomer cannot claim a returning source's + # prior stem merely because it appeared earlier in the invocation. + for i in sorted(range(len(files)), key=lambda j: str(files[j].resolve())): + if i in assigned: + continue + candidate = generated[i] + if candidate not in owners_by_stem and candidate not in used: + assigned[i] = candidate + used.add(candidate) + + # Any remaining candidate collides with recorded ownership. Give it a fresh + # deterministic suffix, avoiding both prior stems and raw basenames that a + # different pending input may need. + pending = [i for i in range(len(files)) if i not in assigned] + unavailable = set(owners_by_stem) | used | {files[i].stem for i in pending} + for i in sorted(pending, key=lambda j: (files[j].stem, str(files[j].resolve()))): + n = 1 + while f"{files[i].stem}-{n}" in unavailable: + n += 1 + assigned[i] = f"{files[i].stem}-{n}" + unavailable.add(assigned[i]) + + return [assigned[i] for i in range(len(files))] + + def process( inputs: list[Path], out_dir: Path, @@ -522,7 +599,7 @@ def process( ) return 0 - stems = _unique_stems(files) + stems = _manifest_aware_stems(files, source_keys, prior_sources) written: set[Path] = set() written_by_source: dict[str, set[Path]] = {source: set() for source in source_keys} seen_inodes: set[int] = set() diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 594a8b0..651fbe0 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -570,11 +570,12 @@ def test_manifest_cleans_dropped_input_orphans(d: Path) -> None: def test_manifest_cleans_changed_collider_stem_orphans(d: Path) -> None: - """--clean removes orphans whose disambiguated stem vanished (#84 case 2). + """--clean removes a dropped collider without renaming the survivor. - Two `take.wav` inputs become stems `take` and `take-1`. Rerun with only one - and `take-1` is not a stem any more, so its clips are orphaned under a name - nothing will overwrite. + Two `take.wav` inputs become stems `take` and `take-1`. Rerunning only the + second source keeps its manifested `take-1` stem so its refresh cannot take + another source's filenames; the manifest removes the dropped first source's + bare-stem clips. """ src = d / "collider_src" _burst_take(src / "a" / "take.wav", 3) @@ -592,8 +593,8 @@ def test_manifest_cleans_changed_collider_stem_orphans(d: Path) -> None: prep.process([src / "b" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) present = sorted(p.name for p in out.glob("*.wav")) check( - not any(n.startswith("take-1_") for n in present), - f"--clean removes orphans of a stem the rerun no longer produces ({present})", + present == ["take-1_000.wav", "take-1_001.wav"], + f"--clean drops the omitted collider without renaming the survivor ({present})", ) From c2be4d5fe6b6eefa8798fc04c4a679605a01e196 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:47:29 -0400 Subject: [PATCH 10/38] test(prep): reproduce manifest recovery diagnostics --- test_prep_wake_samples.py | 60 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 651fbe0..2059c3f 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -836,6 +836,64 @@ def test_manifest_wrong_shape_warns_before_legacy_fallback(d: Path) -> None: ) +def test_manifest_bad_sources_warns_before_ownership_refusal(d: Path) -> None: + """A malformed source map must be diagnosed as corrupt, not legacy.""" + out = d / "bad_sources_manifest_out" + out.mkdir() + (out / prep.MANIFEST_NAME).write_text( + json.dumps( + { + "version": 2, + "label": "positive", + "clips": ["take_000.wav"], + "sources": "not a source map", + } + ) + ) + + errors = io.StringIO() + with redirect_stderr(errors): + prep._read_manifest(out) + check( + "manifest" in errors.getvalue() + and "sources" in errors.getvalue() + and "source ownership" in errors.getvalue(), + "a malformed sources map reports why ownership is unavailable", + ) + + +def test_manifest_replace_failure_describes_authoritative_old_record(d: Path) -> None: + """An atomic replace failure leaves the previous manifest in force.""" + out = d / "manifest_replace_failure_out" + out.mkdir() + old_payload = '{"version": 1, "label": "positive", "clips": []}\n' + manifest = out / prep.MANIFEST_NAME + manifest.write_text(old_payload) + original_replace = prep.os.replace + + def fail_replace(_source: Path, _target: Path) -> None: + raise OSError("simulated replace failure") + + errors = io.StringIO() + try: + prep.os.replace = fail_replace + with redirect_stderr(errors): + prep._write_manifest( + out, + "positive", + {"/recordings/take.wav": {"take_000.wav"}}, + ) + finally: + prep.os.replace = original_replace + + check( + manifest.read_text() == old_payload + and "previous manifest remains in force" in errors.getvalue() + and "unrecorded" in errors.getvalue(), + "a replace failure explains that the old record is still authoritative", + ) + + def test_manifest_without_source_ownership_is_refused(d: Path) -> None: """A readable legacy record cannot prove which dataset owns the output.""" src = d / "legacy_manifest_src" @@ -1092,6 +1150,8 @@ def main() -> int: test_manifest_spares_a_silent_take_alongside_a_good_one(d) test_manifest_unreadable_falls_back_to_stem_rule(d) test_manifest_wrong_shape_warns_before_legacy_fallback(d) + test_manifest_bad_sources_warns_before_ownership_refusal(d) + test_manifest_replace_failure_describes_authoritative_old_record(d) test_manifest_without_source_ownership_is_refused(d) test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d) test_manifest_bootstrap_does_not_adopt_ambiguous_same_stem_audio(d) From e85c8fe972dcbf08a6eb802932567ed79ee6b5b3 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:48:11 -0400 Subject: [PATCH 11/38] fix(prep): report manifest recovery state accurately --- prep_wake_samples.py | 42 +++++++++++++++++++++++++++++++++++------- 1 file changed, 35 insertions(+), 7 deletions(-) diff --git a/prep_wake_samples.py b/prep_wake_samples.py index a793771..0a4c1d9 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -347,10 +347,15 @@ def _read_manifest( """ manifest_path = out_dir / MANIFEST_NAME - def warn_legacy_fallback(reason: object) -> None: + def warn_manifest_problem(reason: object, *, legacy_fallback: bool) -> None: + consequence = ( + "this run will use legacy filename cleanup" + if legacy_fallback + else "this manifest cannot authorize a write" + ) print( f"warning: could not use {manifest_path} ({reason}); source ownership " - "protection is unavailable, so this run will use legacy filename cleanup", + f"protection is unavailable, so {consequence}", file=sys.stderr, ) @@ -359,28 +364,39 @@ def warn_legacy_fallback(reason: object) -> None: except FileNotFoundError: return None, set(), None except (OSError, ValueError) as e: - warn_legacy_fallback(e) + warn_manifest_problem(e, legacy_fallback=True) return None, set(), None if not isinstance(raw, dict): - warn_legacy_fallback("expected a JSON object") + warn_manifest_problem("expected a JSON object", legacy_fallback=True) return None, set(), None clips = raw.get("clips") if not isinstance(clips, list): - warn_legacy_fallback("expected a clips list") + warn_manifest_problem("expected a clips list", legacy_fallback=True) return None, set(), None label = raw.get("label") clip_names = {c for c in clips if isinstance(c, str)} sources_raw = raw.get("sources") if not isinstance(sources_raw, dict): + warn_manifest_problem( + "expected a sources map", legacy_fallback=not isinstance(label, str) + ) return label if isinstance(label, str) else None, clip_names, None sources: dict[str, set[str]] = {} for source, names in sources_raw.items(): if not isinstance(source, str) or not isinstance(names, list): + warn_manifest_problem( + "expected every sources entry to map a path to a clip list", + legacy_fallback=not isinstance(label, str), + ) return label if isinstance(label, str) else None, clip_names, None sources[source] = {name for name in names if isinstance(name, str)} owned_clips = set().union(*sources.values()) if sources else set() if owned_clips != clip_names: + warn_manifest_problem( + "sources do not own exactly the recorded clips", + legacy_fallback=not isinstance(label, str), + ) return label if isinstance(label, str) else None, clip_names, None return label if isinstance(label, str) else None, clip_names, sources @@ -431,9 +447,21 @@ def _write_manifest( os.fsync(tmp.fileno()) os.replace(temporary, out_dir / MANIFEST_NAME) except OSError as e: + manifest_path = out_dir / MANIFEST_NAME + if manifest_path.exists(): + recovery = ( + "the previous manifest remains in force and clips written by this " + "run are unrecorded. Fix the write problem and rerun the full input " + f"set, or remove {manifest_path} to bootstrap ownership again" + ) + else: + recovery = ( + "no manifest was recorded, so clips written by this run have no " + "source ownership protection. Fix the write problem and rerun the " + "full input set" + ) print( - f"warning: could not write {out_dir / MANIFEST_NAME} ({e}); " - f"a later --clean will fall back to matching re-recorded stems only", + f"warning: could not write {manifest_path} ({e}); {recovery}", file=sys.stderr, ) finally: From 0b3165d346fb9b9af6607f8467464306e001f229 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:54:52 -0400 Subject: [PATCH 12/38] test(prep): reproduce silent legacy clean fallback --- test_prep_wake_samples.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 2059c3f..14eeb05 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -971,6 +971,27 @@ def test_manifest_bootstrap_does_not_adopt_ambiguous_same_stem_audio( ) +def test_clean_without_manifest_warns_before_legacy_cleanup(d: Path) -> None: + """A provenance-free clean must announce its filename-only deletion rule.""" + src = d / "legacy_clean_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "legacy_clean_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + (out / prep.MANIFEST_NAME).unlink() + _burst_take(out / "take_007.wav", 1) + + _burst_take(take, 2) + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) + check( + "manifest" in errors.getvalue() + and "legacy filename cleanup" in errors.getvalue(), + "a no-manifest --clean warns before using the provenance-free rule", + ) + + def test_manifest_spares_a_different_dataset_on_stray_output(d: Path) -> None: """A mistyped --output must not let the manifest wipe another dataset. @@ -1155,6 +1176,7 @@ def main() -> int: test_manifest_without_source_ownership_is_refused(d) test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d) test_manifest_bootstrap_does_not_adopt_ambiguous_same_stem_audio(d) + test_clean_without_manifest_warns_before_legacy_cleanup(d) test_manifest_spares_a_different_dataset_on_stray_output(d) test_same_label_shared_stem_stray_is_refused_before_writing(d) test_clean_disarmed_entirely_on_a_label_mismatch(d) From fd7d3cbd26e72cf1d1b5a6353bb0477919c82299 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 20:55:34 -0400 Subject: [PATCH 13/38] fix(prep): warn before legacy clean fallback --- prep_wake_samples.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 0a4c1d9..fb894ca 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -587,7 +587,20 @@ def process( # this directory's take_000.wav with the new dataset's. A label mismatch is # the signature of a mistyped --output, and there is no reading of it where # writing positives into a directory of negatives is what the user meant. + manifest_present = (out_dir / MANIFEST_NAME).exists() prior_label, _prior_clips, prior_sources = _read_manifest(out_dir) + if ( + clean + and not manifest_present + and out_dir.is_dir() + and any(path.suffix.lower() in AUDIO_EXTS for path in out_dir.iterdir()) + ): + print( + f"warning: {out_dir} has no {MANIFEST_NAME}; this --clean will use " + "legacy filename cleanup without source ownership protection and may " + "remove hand-added _NNN.wav files", + file=sys.stderr, + ) if prior_label is not None and prior_label != label: print( f"error: --output {out_dir} holds {prior_label!r} clips, not {label!r}; " From c2969c5d890b1c228f7841d1f1f1c1166011f692 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:09:54 -0400 Subject: [PATCH 14/38] test(prep): reproduce mixed lingering guidance --- test_prep_wake_samples.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 14eeb05..4d5def5 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -755,8 +755,12 @@ def test_manifest_spares_a_silent_take_alongside_a_good_one(d: Path) -> None: _burst_take(flaky, 3) out = d / "mixed_out" prep.process([good, flaky], out, "positive", 0.3, 0.2, 3.0) + _burst_take(out / "custom_001.wav", 1) before = sorted(p.name for p in out.glob("*.wav")) - check(len(before) == 5, f"both takes record on the first run ({before})") + check( + len(before) == 6, + f"both takes record and curated audio is present ({before})", + ) # Re-record both. `good` is fine; `flaky` comes back as one long unbroken # tone -- a mic left running, say -- which the max-duration cap drops @@ -768,13 +772,22 @@ def test_manifest_spares_a_silent_take_alongside_a_good_one(d: Path) -> None: prep.TARGET_SR, subtype="PCM_16", ) - prep.process([good, flaky], out, "positive", 0.3, 0.2, 1.0, clean=True) + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([good, flaky], out, "positive", 0.3, 0.2, 1.0, clean=True) present = sorted(p.name for p in out.glob("*.wav")) check( sorted(n for n in present if n.startswith("flaky_")) == ["flaky_000.wav", "flaky_001.wav", "flaky_002.wav"], f"--clean spares a silent take's only copy while cleaning beside it ({present})", ) + check( + "check the recording" in errors.getvalue() + and "custom_001.wav" in errors.getvalue() + and "remove" in errors.getvalue() + and "by hand" in errors.getvalue(), + "mixed lingering guidance protects silent clips and identifies manual cleanup", + ) def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: From 629d1ecd72883cdd1c5af5046fef4be50e400e25 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:11:25 -0400 Subject: [PATCH 15/38] fix(prep): reconcile clean guidance with ownership --- CHANGELOG.md | 14 ++++++++------ docs/recording-computah.md | 6 ++++++ docs/wake-threshold-tuning.md | 4 ++-- prep_wake_samples.py | 32 +++++++++++++++++++++++++------- 4 files changed, 41 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f426dd..7390cdf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,12 +11,14 @@ All notable changes to computah are recorded here. The format follows just re-recorded (a `_NNN.wav` clip whose stem this run wrote) from `--output`, so re-recording with fewer utterances no longer strands orphaned higher-numbered clips that training and eval would still read. Off by default - (deletion is opt-in) and deliberately narrow: a source recording, an unrelated - take's clips, or a hand-curated clip whose stem this run did not write is named - but never auto-deleted. Without `--clean` the run still warns and names the - leftover count. `test_prep_wake_samples.py` covers the re-run-with-fewer-clips - case with and without `--clean`, and that `--clean` spares files this run did - not record. + (deletion is opt-in). The source manifest below broadens cleanup to the + prep-owned clips of an intentionally omitted take while still sparing source + recordings and hand-curated audio. With no manifest or an unreadable one, the + narrow same-stem rule remains as a warned legacy fallback; readable older + records without source ownership fail closed until removed. Without `--clean` + the run still warns and names the leftover count. `test_prep_wake_samples.py` + covers the re-run-with-fewer-clips case with and without `--clean`, and that + source-aware `--clean` spares files prep did not record. - `prep_wake_samples.py` output manifest (#84): prep records each resolved source path and the clips it owns in `.prep-manifest.json` in the output dir. `--clean` uses that provenance to remove orphans the stem match cannot reach -- diff --git a/docs/recording-computah.md b/docs/recording-computah.md index 964a831..39dab20 100644 --- a/docs/recording-computah.md +++ b/docs/recording-computah.md @@ -76,6 +76,12 @@ reports per-file segment counts and a duration summary, and writes 16 kHz mono i WAVs ready for verifier training. `samples/` is gitignored — the recordings are personal data and stay out of version control. +Once an output directory has a source manifest, an incremental run must include at +least one recording already listed there along with any new recording. Keep added +microphone takes under the same positive-file glob, or list them beside an existing +source explicitly. With `--clean`, pass every source whose clips the dataset should +keep; omitted sources are treated as intentionally dropped. + ## Training Training runs externally (openWakeWord 0.4.0 ships no train submodule) on a GPU host. diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 8d2452e..6faafe1 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -80,8 +80,8 @@ remove that manifest once to bootstrap the source map from the next successful run. ```bash -.venv/bin/python prep_wake_samples.py --input computah_normal.wav \ - --output samples/positive --label positive --clean +.venv/bin/python prep_wake_samples.py --input computah_normal.wav computah_styles.wav \ + computah_distance.wav --output samples/positive --label positive --clean ``` ## 2. Run the sweep diff --git a/prep_wake_samples.py b/prep_wake_samples.py index fb894ca..646ba43 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -42,9 +42,9 @@ TARGET_SR = 16000 AUDIO_EXTS = {".wav", ".flac", ".ogg", ".m4a", ".mp3", ".aac", ".mp4", ".wma"} # prep writes each clip as "_NNN.wav" (see _write_unique; NNN is >=3 digits, -# since :03d is a minimum width). --clean pairs this shape with the current run's -# stems so it only removes a re-recorded take's own leftover clips, never a source -# take, an unrelated take's clips, or hand-added audio a user left in the output dir. +# since :03d is a minimum width). Without source ownership, --clean uses this +# shape plus the current run's stems as its narrow legacy fallback. With a v2 +# manifest, exact recorded clip names authorize cleanup instead. _CLIP_NAME = re.compile(r"(.+)_\d{3,}\.wav$") @@ -779,12 +779,30 @@ def process( ) empty_takes = [p for p in lingering if p.name in silent_names] if empty_takes and clean: + failed_removals = [p for p in lingering if p in removable] + unrecorded = [ + p + for p in lingering + if p not in empty_takes and p not in failed_removals + ] fix = ( - "this run re-recorded " - f"{_summarize(empty_takes)} and got no clips from it, so --clean " - "kept the earlier ones rather than leave you with none; check the " - "recording before removing anything" + f"{_summarize(empty_takes)} are prior clips for a source this run " + "re-recorded but got no clips from, so --clean kept them rather " + "than leave you with none; check the recording before removing " + "anything" ) + if unrecorded: + fix += ( + f". {_summarize(unrecorded)} have no prep ownership record, " + "so --clean leaves them; remove them by hand if they are not " + "wanted" + ) + if failed_removals: + fix += ( + f". {_summarize(failed_removals)} were selected for cleanup " + "but could not be removed; address the earlier errors and rerun " + "--clean" + ) elif clean: fix = ( "prep has no record of creating these, so --clean leaves them; " From bb918d94001c84f78cb8605d4c8a5f6599041b08 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:16:12 -0400 Subject: [PATCH 16/38] test(prep): reproduce corrupt-record recovery gaps --- test_prep_wake_samples.py | 44 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 4d5def5..7ae7cd0 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -809,7 +809,9 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: 0.2, 3.0, ) - (out / prep.MANIFEST_NAME).write_text("{not json") + corrupt_payload = "{not json" + manifest = out / prep.MANIFEST_NAME + manifest.write_text(corrupt_payload) _burst_take(src / "a" / "take.wav", 2) errors = io.StringIO() @@ -827,6 +829,10 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: and "legacy filename cleanup" in errors.getvalue(), "a corrupt manifest warns that ownership protection is unavailable", ) + check( + manifest.read_text() == corrupt_payload, + "a fallback run preserves the unreadable manifest for deliberate recovery", + ) def test_manifest_wrong_shape_warns_before_legacy_fallback(d: Path) -> None: @@ -907,6 +913,41 @@ def fail_replace(_source: Path, _target: Path) -> None: ) +def test_failed_unlink_summary_keeps_recorded_ownership(d: Path) -> None: + """A failed owned deletion must not be summarized as unrecorded audio.""" + src = d / "failed_unlink_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "failed_unlink_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + _burst_take(take, 2) + + original_unlink = prep.Path.unlink + + def fail_one_unlink(path: Path, *args: object, **kwargs: object) -> None: + if path.name == "take_002.wav": + raise OSError("simulated unlink failure") + original_unlink(path, *args, **kwargs) + + errors = io.StringIO() + try: + prep.Path.unlink = fail_one_unlink + with redirect_stderr(errors): + prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) + finally: + prep.Path.unlink = original_unlink + + _label, _clips, sources = prep._read_manifest(out) + owned = sources.get(prep._source_key(take), set()) if sources is not None else set() + check( + "take_002.wav" in owned + and "selected for cleanup" in errors.getvalue() + and "could not be removed" in errors.getvalue() + and "no record of creating" not in errors.getvalue(), + "a failed unlink stays owned and gets accurate recovery guidance", + ) + + def test_manifest_without_source_ownership_is_refused(d: Path) -> None: """A readable legacy record cannot prove which dataset owns the output.""" src = d / "legacy_manifest_src" @@ -1186,6 +1227,7 @@ def main() -> int: test_manifest_wrong_shape_warns_before_legacy_fallback(d) test_manifest_bad_sources_warns_before_ownership_refusal(d) test_manifest_replace_failure_describes_authoritative_old_record(d) + test_failed_unlink_summary_keeps_recorded_ownership(d) test_manifest_without_source_ownership_is_refused(d) test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d) test_manifest_bootstrap_does_not_adopt_ambiguous_same_stem_audio(d) From ed58b180ad67e322e6034ecc8367f00858fed00e Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:17:10 -0400 Subject: [PATCH 17/38] fix(prep): preserve unreadable ownership records --- CHANGELOG.md | 9 ++++++--- docs/wake-threshold-tuning.md | 14 ++++++++------ prep_wake_samples.py | 33 +++++++++++++++++++++++++++------ 3 files changed, 41 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7390cdf..23896c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,9 +36,12 @@ All notable changes to computah are recorded here. The format follows a guess into durable provenance. A no-output run cannot claim an unrecorded directory. Manifest writes use atomic replacement, and an absent or unreadable manifest degrades `--clean` to the narrow stem match with an explicit warning. - That fallback cannot distinguish a prep leftover from a hand-added - `_NNN.wav`. A readable older manifest without source ownership is - refused until the user removes it to bootstrap ownership explicitly. + An unreadable manifest is preserved instead of being replaced by a partial + source map, leaving fallback output explicitly unrecorded until the user + reviews the directory and reboots ownership. That fallback cannot distinguish + a prep leftover from a hand-added `_NNN.wav`. A readable older + manifest without source ownership is refused until the user removes it to + bootstrap ownership explicitly. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 6faafe1..52341e3 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -36,7 +36,7 @@ would linger and the training globs would still read them, so prep warns and names the count. Pass `--clean` to remove those leftovers in the same run. With a readable source-aware manifest, `--clean` only deletes clips prep recorded -making. It removes the now-unused higher-numbered clips of a take this run +as its output. It removes the now-unused higher-numbered clips of a take this run re-recorded, and the clips a previous run recorded for a take that is no longer in the inputs. The `.prep-manifest.json` in the output dir maps each resolved source path to the clips prep wrote for it. A source recording, a hand-curated @@ -72,12 +72,14 @@ then run a full-input `--clean` pass after reviewing the complete source list. Manifest updates use atomic replacement. An absent or unreadable manifest falls back to the narrow same-stem cleanup rule with a warning that source ownership -protection is unavailable. That fallback has no provenance: `--clean` can remove any +protection is unavailable. An unreadable manifest is preserved rather than +replaced with a partial source map, so clips written during that fallback remain +unrecorded. That fallback has no provenance: `--clean` can remove any `_NNN.wav` file, including a hand-added file with that shape. Check the -warning list and move any such file before cleaning. A readable older manifest -without source ownership is refused; after confirming the output directory, -remove that manifest once to bootstrap the source map from the next successful -run. +warning list and move any such file before cleaning. After reviewing the output +directory, remove an unreadable manifest and rerun the full source set to +bootstrap ownership. A readable older manifest without source ownership is +refused and requires the same deliberate removal. ```bash .venv/bin/python prep_wake_samples.py --input computah_normal.wav computah_styles.wav \ diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 646ba43..6ce6eec 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -746,7 +746,17 @@ def process( # A no-output run cannot establish ownership in an unrecorded directory. # Leaving it unclaimed lets a later successful rerun bootstrap the legacy # clips instead of stranding them behind an empty manifest. - if (prior_sources is not None or written) and not ambiguous_legacy: + unusable_manifest = manifest_present and prior_sources is None + if unusable_manifest: + if written: + print( + f"warning: the unreadable {out_dir / MANIFEST_NAME} was preserved; " + "clips written by this run are unrecorded. After reviewing the " + "output directory, remove that manifest and rerun the full input " + "set to bootstrap ownership again", + file=sys.stderr, + ) + elif (prior_sources is not None or written) and not ambiguous_legacy: _write_manifest(out_dir, label, next_sources) elif ambiguous_legacy: print( @@ -778,8 +788,8 @@ def process( ) ) empty_takes = [p for p in lingering if p.name in silent_names] + failed_removals = [p for p in lingering if p in removable] if empty_takes and clean: - failed_removals = [p for p in lingering if p in removable] unrecorded = [ p for p in lingering @@ -804,10 +814,21 @@ def process( "--clean" ) elif clean: - fix = ( - "prep has no record of creating these, so --clean leaves them; " - "remove them by hand if they are not wanted" - ) + unrecorded = [p for p in lingering if p not in failed_removals] + fixes: list[str] = [] + if unrecorded: + fixes.append( + "prep has no record of creating " + f"{_summarize(unrecorded)}, so --clean leaves them; remove " + "them by hand if they are not wanted" + ) + if failed_removals: + fixes.append( + f"{_summarize(failed_removals)} were selected for cleanup " + "but could not be removed; address the earlier errors and " + "rerun --clean" + ) + fix = ". ".join(fixes) else: fix = "rerun with --clean to remove them, or clear the directory by hand" print( From fc60b07a4b3d64c6d5f9f5d47126333edbe74c6d Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:22:10 -0400 Subject: [PATCH 18/38] test(prep): reproduce unsafe silent-take advice --- test_prep_wake_samples.py | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 7ae7cd0..21f32ff 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -719,6 +719,33 @@ def test_manifest_spares_prior_clips_when_rerun_writes_nothing(d: Path) -> None: ) +def test_nonclean_silent_rerun_protects_only_copy_in_warning(d: Path) -> None: + """A warning must not invite deletion of a silent source's prior clips.""" + src = d / "nonclean_silent_src" + take = src / "take.wav" + _burst_take(take, 3) + out = d / "nonclean_silent_out" + prep.process([take], out, "positive", 0.3, 0.2, 3.0) + + sf.write( + take, + np.ones(int(prep.TARGET_SR * 4.0), dtype=np.float32) * 0.5, + prep.TARGET_SR, + subtype="PCM_16", + ) + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([take], out, "positive", 0.3, 0.2, 1.0) + present = sorted(p.name for p in out.glob("*.wav")) + check( + present == ["take_000.wav", "take_001.wav", "take_002.wav"] + and "check the recording" in errors.getvalue() + and "clear the directory" not in errors.getvalue() + and "rerun with --clean" not in errors.getvalue(), + f"a non-clean silent rerun protects its only-copy clips in guidance ({present})", + ) + + def test_manifest_empty_source_remains_refreshable(d: Path) -> None: """An owned source with no clips must not deadlock its output directory.""" src = d / "empty_source_src" @@ -1221,6 +1248,7 @@ def main() -> int: test_new_collider_cannot_overwrite_a_silent_sources_prior_stem(d) test_manifest_never_authorizes_deleting_curated_audio(d) test_manifest_spares_prior_clips_when_rerun_writes_nothing(d) + test_nonclean_silent_rerun_protects_only_copy_in_warning(d) test_manifest_empty_source_remains_refreshable(d) test_manifest_spares_a_silent_take_alongside_a_good_one(d) test_manifest_unreadable_falls_back_to_stem_rule(d) From 1f2ae8f9594c44f7968714e7387e9e8202650f47 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:22:49 -0400 Subject: [PATCH 19/38] fix(prep): protect silent clips in all warnings --- prep_wake_samples.py | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 6ce6eec..f54190f 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -789,24 +789,31 @@ def process( ) empty_takes = [p for p in lingering if p.name in silent_names] failed_removals = [p for p in lingering if p in removable] - if empty_takes and clean: - unrecorded = [ + if empty_takes: + other_lingering = [ p for p in lingering if p not in empty_takes and p not in failed_removals ] fix = ( f"{_summarize(empty_takes)} are prior clips for a source this run " - "re-recorded but got no clips from, so --clean kept them rather " + "re-recorded but got no clips from, so prep kept them rather " "than leave you with none; check the recording before removing " "anything" ) - if unrecorded: - fix += ( - f". {_summarize(unrecorded)} have no prep ownership record, " - "so --clean leaves them; remove them by hand if they are not " - "wanted" - ) + if other_lingering: + if clean: + fix += ( + f". {_summarize(other_lingering)} have no prep ownership " + "record, so --clean leaves them; remove them by hand if " + "they are not wanted" + ) + else: + fix += ( + f". For the other leftovers ({_summarize(other_lingering)}), " + "rerun with --clean or review and remove only those files " + "by hand" + ) if failed_removals: fix += ( f". {_summarize(failed_removals)} were selected for cleanup " From 9f18fab77ace15d2e58fc4f01b322743e7f09dd2 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:41:25 -0400 Subject: [PATCH 20/38] test(prep): reproduce manifest path escape --- test_prep_wake_samples.py | 48 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 21f32ff..c226c53 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -688,6 +688,17 @@ def test_manifest_never_authorizes_deleting_curated_audio(d: Path) -> None: _burst_take(out / "custom_001.wav", 1) _burst_take(out / "my_recording.wav", 1) + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([src / "take.wav"], out, "positive", 0.3, 0.2, 3.0) + check( + "custom_001.wav" in errors.getvalue() + and "remove" in errors.getvalue() + and "by hand" in errors.getvalue() + and "rerun with --clean to remove them" not in errors.getvalue(), + "a non-clean refresh gives manual-only guidance for curated audio", + ) + prep.process([src / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) present = sorted(p.name for p in out.glob("*.wav")) check( @@ -856,6 +867,10 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: and "legacy filename cleanup" in errors.getvalue(), "a corrupt manifest warns that ownership protection is unavailable", ) + check( + "hand-added" in errors.getvalue(), + "a corrupt-manifest clean warns that curated same-stem clips are deletable", + ) check( manifest.read_text() == corrupt_payload, "a fallback run preserves the unreadable manifest for deliberate recovery", @@ -908,6 +923,38 @@ def test_manifest_bad_sources_warns_before_ownership_refusal(d: Path) -> None: ) +def test_manifest_clip_names_cannot_escape_output_directory(d: Path) -> None: + """Manifest ownership must never turn a path into an output stem.""" + src = d / "hostile_manifest_src" + take = src / "take.wav" + _burst_take(take, 2) + out = d / "hostile_manifest_out" + out.mkdir() + escaped = d / "escaped_000.wav" + hostile_name = "../escaped_000.wav" + (out / prep.MANIFEST_NAME).write_text( + json.dumps( + { + "version": 2, + "label": "positive", + "clips": [hostile_name], + "sources": {prep._source_key(take): [hostile_name]}, + } + ) + ) + + errors = io.StringIO() + with redirect_stderr(errors): + total = prep.process([take], out, "positive", 0.3, 0.2, 3.0) + check( + total == 0 + and not escaped.exists() + and "manifest" in errors.getvalue() + and "nothing was written" in errors.getvalue(), + "a manifest path is refused before any write can escape --output", + ) + + def test_manifest_replace_failure_describes_authoritative_old_record(d: Path) -> None: """An atomic replace failure leaves the previous manifest in force.""" out = d / "manifest_replace_failure_out" @@ -1254,6 +1301,7 @@ def main() -> int: test_manifest_unreadable_falls_back_to_stem_rule(d) test_manifest_wrong_shape_warns_before_legacy_fallback(d) test_manifest_bad_sources_warns_before_ownership_refusal(d) + test_manifest_clip_names_cannot_escape_output_directory(d) test_manifest_replace_failure_describes_authoritative_old_record(d) test_failed_unlink_summary_keeps_recorded_ownership(d) test_manifest_without_source_ownership_is_refused(d) From 6babbc5613b38f146cf4d8c5688a5a97d51ca7aa Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:43:03 -0400 Subject: [PATCH 21/38] fix(prep): reject manifest output paths --- CHANGELOG.md | 2 + prep_wake_samples.py | 96 +++++++++++++++++++++++++++++++++++--------- 2 files changed, 78 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 23896c4..ce7984b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,8 @@ All notable changes to computah are recorded here. The format follows (`take-1_000.wav`) that a later run no longer produces. A run must share an exact recorded source path before it may write to that dataset, so a different dataset with the same label and generic basename cannot overwrite or clean it. + Malformed ownership maps, including clip names that are paths instead of safe + output filenames, are refused before any audio is decoded or written. Manifested stems remain reserved for their source while new output names are assigned, so a new same-basename recording cannot overwrite the only good clips from a source whose rerun is silent. A source this run attempted but got diff --git a/prep_wake_samples.py b/prep_wake_samples.py index f54190f..6b2935b 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -374,7 +374,23 @@ def warn_manifest_problem(reason: object, *, legacy_fallback: bool) -> None: warn_manifest_problem("expected a clips list", legacy_fallback=True) return None, set(), None label = raw.get("label") - clip_names = {c for c in clips if isinstance(c, str)} + + def safe_clip_name(name: object) -> bool: + return ( + isinstance(name, str) + and name == Path(name).name + and "/" not in name + and "\\" not in name + and _CLIP_NAME.fullmatch(name) is not None + ) + + if not all(safe_clip_name(name) for name in clips): + warn_manifest_problem( + "clips must contain safe output filenames, not paths", + legacy_fallback=not isinstance(label, str), + ) + return label if isinstance(label, str) else None, set(), None + clip_names = set(clips) sources_raw = raw.get("sources") if not isinstance(sources_raw, dict): warn_manifest_problem( @@ -390,7 +406,13 @@ def warn_manifest_problem(reason: object, *, legacy_fallback: bool) -> None: legacy_fallback=not isinstance(label, str), ) return label if isinstance(label, str) else None, clip_names, None - sources[source] = {name for name in names if isinstance(name, str)} + if not all(safe_clip_name(name) for name in names): + warn_manifest_problem( + "sources must own safe output filenames, not paths", + legacy_fallback=not isinstance(label, str), + ) + return label if isinstance(label, str) else None, clip_names, None + sources[source] = set(names) owned_clips = set().union(*sources.values()) if sources else set() if owned_clips != clip_names: warn_manifest_problem( @@ -589,18 +611,6 @@ def process( # writing positives into a directory of negatives is what the user meant. manifest_present = (out_dir / MANIFEST_NAME).exists() prior_label, _prior_clips, prior_sources = _read_manifest(out_dir) - if ( - clean - and not manifest_present - and out_dir.is_dir() - and any(path.suffix.lower() in AUDIO_EXTS for path in out_dir.iterdir()) - ): - print( - f"warning: {out_dir} has no {MANIFEST_NAME}; this --clean will use " - "legacy filename cleanup without source ownership protection and may " - "remove hand-added _NNN.wav files", - file=sys.stderr, - ) if prior_label is not None and prior_label != label: print( f"error: --output {out_dir} holds {prior_label!r} clips, not {label!r}; " @@ -618,6 +628,23 @@ def process( file=sys.stderr, ) return 0 + if ( + clean + and prior_sources is None + and out_dir.is_dir() + and any(path.suffix.lower() in AUDIO_EXTS for path in out_dir.iterdir()) + ): + manifest_state = ( + f"{out_dir / MANIFEST_NAME} is unusable" + if manifest_present + else f"{out_dir} has no {MANIFEST_NAME}" + ) + print( + f"warning: {manifest_state}; this --clean will use legacy filename " + "cleanup without source ownership protection and may remove hand-added " + "_NNN.wav files", + file=sys.stderr, + ) files = _inputs(inputs) if not files: @@ -788,6 +815,9 @@ def process( ) ) empty_takes = [p for p in lingering if p.name in silent_names] + owned_names = ( + set().union(*prior_sources.values()) if prior_sources is not None else set() + ) failed_removals = [p for p in lingering if p in removable] if empty_takes: other_lingering = [ @@ -809,11 +839,18 @@ def process( "they are not wanted" ) else: - fix += ( - f". For the other leftovers ({_summarize(other_lingering)}), " - "rerun with --clean or review and remove only those files " - "by hand" - ) + cleanable = [p for p in other_lingering if p.name in owned_names] + manual = [p for p in other_lingering if p not in cleanable] + if cleanable: + fix += ( + f". Rerun with --clean to remove prep-owned " + f"{_summarize(cleanable)}" + ) + if manual: + fix += ( + f". {_summarize(manual)} have no prep ownership record; " + "remove only those files by hand if they are not wanted" + ) if failed_removals: fix += ( f". {_summarize(failed_removals)} were selected for cleanup " @@ -837,7 +874,26 @@ def process( ) fix = ". ".join(fixes) else: - fix = "rerun with --clean to remove them, or clear the directory by hand" + if prior_sources is None: + fix = ( + "review these legacy filenames, then rerun with --clean or " + "remove only the unwanted files by hand" + ) + else: + cleanable = [p for p in lingering if p.name in owned_names] + manual = [p for p in lingering if p not in cleanable] + fixes = [] + if cleanable: + fixes.append( + f"rerun with --clean to remove prep-owned " + f"{_summarize(cleanable)}" + ) + if manual: + fixes.append( + f"{_summarize(manual)} have no prep ownership record; " + "remove them by hand if they are not wanted" + ) + fix = ". ".join(fixes) print( f"warning: {len(lingering)} file(s) in {out_dir} are left from an " f"earlier run ({_summarize(lingering)}). Training and eval read every " From 40e1bfe213876dabac21f17d2f3c25c4f8cbcf7e Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:48:00 -0400 Subject: [PATCH 22/38] test(prep): reproduce zero-output ownership grant --- test_prep_wake_samples.py | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index c226c53..15ff84b 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -776,6 +776,40 @@ def test_manifest_empty_source_remains_refreshable(d: Path) -> None: ) +def test_new_zero_output_source_does_not_gain_cleanup_authority(d: Path) -> None: + """A source that contributed no clips cannot authorize a later clean.""" + src = d / "zero_owner_src" + good = src / "good.wav" + junk = src / "junk.wav" + _burst_take(good, 3) + out = d / "zero_owner_out" + prep.process([good], out, "positive", 0.3, 0.2, 3.0) + + sf.write( + junk, + np.ones(int(prep.TARGET_SR * 4.0), dtype=np.float32) * 0.5, + prep.TARGET_SR, + subtype="PCM_16", + ) + prep.process([good, junk], out, "positive", 0.3, 0.2, 1.0, clean=True) + _label, _clips, sources = prep._read_manifest(out) + junk_key = prep._source_key(junk) + check( + sources is not None and junk_key not in sources, + "a first-time zero-output source is not recorded as an owner", + ) + + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([junk], out, "positive", 0.3, 0.2, 1.0, clean=True) + present = sorted(p.name for p in out.glob("*.wav")) + check( + present == ["good_000.wav", "good_001.wav", "good_002.wav"] + and "different source recordings" in errors.getvalue(), + f"a zero-output newcomer cannot later clean the real dataset ({present})", + ) + + def test_manifest_spares_a_silent_take_alongside_a_good_one(d: Path) -> None: """The `attempted` guard, with the manifest rule actually armed. @@ -1297,6 +1331,7 @@ def main() -> int: test_manifest_spares_prior_clips_when_rerun_writes_nothing(d) test_nonclean_silent_rerun_protects_only_copy_in_warning(d) test_manifest_empty_source_remains_refreshable(d) + test_new_zero_output_source_does_not_gain_cleanup_authority(d) test_manifest_spares_a_silent_take_alongside_a_good_one(d) test_manifest_unreadable_falls_back_to_stem_rule(d) test_manifest_wrong_shape_warns_before_legacy_fallback(d) From ab7f50fb37b86c72cf37e9d94c468a5facc79b7d Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 21:48:39 -0400 Subject: [PATCH 23/38] fix(prep): require output before granting ownership --- CHANGELOG.md | 17 +++++++++-------- docs/wake-threshold-tuning.md | 4 +++- prep_wake_samples.py | 10 +++++++--- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ce7984b..9525284 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,14 +36,15 @@ All notable changes to computah are recorded here. The format follows refresh that leaves ambiguous same-stem files does not write a manifest; the next `--clean` remains on the documented filename fallback instead of turning a guess into durable provenance. A no-output run cannot claim an unrecorded - directory. Manifest writes use atomic replacement, and an absent or unreadable - manifest degrades `--clean` to the narrow stem match with an explicit warning. - An unreadable manifest is preserved instead of being replaced by a partial - source map, leaving fallback output explicitly unrecorded until the user - reviews the directory and reboots ownership. That fallback cannot distinguish - a prep leftover from a hand-added `_NNN.wav`. A readable older - manifest without source ownership is refused until the user removes it to - bootstrap ownership explicitly. + directory, and a first-time source that writes no clips cannot gain cleanup + authority over an existing dataset. Manifest writes use atomic replacement, + and an absent or unreadable manifest degrades `--clean` to the narrow stem + match with an explicit warning. An unreadable manifest is preserved instead of + being replaced by a partial source map, leaving fallback output explicitly + unrecorded until the user reviews the directory and reboots ownership. That + fallback cannot distinguish a prep leftover from a hand-added + `_NNN.wav`. A readable older manifest without source ownership is + refused until the user removes it to bootstrap ownership explicitly. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 52341e3..2e95fc1 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -60,7 +60,9 @@ filename rule rather than recording that guess as provenance. A run that produces no clips does not claim an unrecorded directory. Prep also spares the prior clips of a source it tried and got nothing from (a silent or bad re-recording): those are the only copy, and it will not trade them for a run -that produced nothing. +that produced nothing. A new source that produces no clips is not added to the +ownership record, so it cannot authorize a later cleanup until a successful run +actually contributes audio. To add a new recording to an existing dataset without `--clean`, include at least one source already recorded there in the same invocation. This gives prep diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 6b2935b..4ef7320 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -764,10 +764,14 @@ def process( kept = names & lingering_names if kept: next_sources[source] = kept - # Keep current sources in the record even when they wrote nothing. If their - # old clips are already absent, the empty entry still proves the next rerun - # comes from the same source instead of deadlocking the directory. + # Keep an already-owned current source in the record even when it wrote + # nothing. If its old clips are already absent, the empty entry still proves + # the next rerun comes from the same source instead of deadlocking the + # directory. A brand-new source earns no cleanup authority until it actually + # contributes a clip. for source, paths in written_by_source.items(): + if not paths and (prior_sources is None or source not in prior_sources): + continue next_sources.setdefault(source, set()).update(path.name for path in paths) # A no-output run cannot establish ownership in an unrecorded directory. From ff2e153a796579e07881bf5c185adf7a4a93d2c4 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:07:33 -0400 Subject: [PATCH 24/38] test(prep): reproduce manifest reader drift --- test_prep_wake_samples.py | 56 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 15ff84b..fcb3215 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -10,6 +10,7 @@ import io import json +import os import sys import tempfile from contextlib import redirect_stderr @@ -989,6 +990,59 @@ def test_manifest_clip_names_cannot_escape_output_directory(d: Path) -> None: ) +def test_manifest_rejects_unsupported_version_with_v2_shape(d: Path) -> None: + """A version-1 declaration cannot gain v2 cleanup authority by shape.""" + src = d / "unsupported_version_src" + take = src / "take.wav" + _burst_take(take, 2) + out = d / "unsupported_version_out" + out.mkdir() + source_key = prep._source_key(take) + (out / prep.MANIFEST_NAME).write_text( + json.dumps( + { + "version": 1, + "label": "positive", + "clips": ["take_000.wav"], + "sources": {source_key: ["take_000.wav"]}, + } + ) + ) + + errors = io.StringIO() + with redirect_stderr(errors): + total = prep.process([take], out, "positive", 0.3, 0.2, 3.0) + check( + total == 0 + and not list(out.glob("*.wav")) + and "version" in errors.getvalue() + and "nothing was written" in errors.getvalue(), + "an unsupported manifest version is refused despite a v2-shaped map", + ) + + +def test_posix_backslash_basename_round_trips_manifest(d: Path) -> None: + """A POSIX basename the writer accepts must remain readable next run.""" + if os.sep != "/": + check(True, "the POSIX backslash-basename check is not applicable") + return + + src = d / "backslash_basename_src" + take = src / "rec\\take.wav" + _burst_take(take, 2) + out = d / "backslash_basename_out" + first = prep.process([take], out, "positive", 0.3, 0.2, 3.0) + second = prep.process([take], out, "positive", 0.3, 0.2, 3.0) + _label, _clips, sources = prep._read_manifest(out) + check( + first == 2 + and second == 2 + and sources is not None + and prep._source_key(take) in sources, + "a POSIX backslash basename survives its manifest round trip", + ) + + def test_manifest_replace_failure_describes_authoritative_old_record(d: Path) -> None: """An atomic replace failure leaves the previous manifest in force.""" out = d / "manifest_replace_failure_out" @@ -1337,6 +1391,8 @@ def main() -> int: test_manifest_wrong_shape_warns_before_legacy_fallback(d) test_manifest_bad_sources_warns_before_ownership_refusal(d) test_manifest_clip_names_cannot_escape_output_directory(d) + test_manifest_rejects_unsupported_version_with_v2_shape(d) + test_posix_backslash_basename_round_trips_manifest(d) test_manifest_replace_failure_describes_authoritative_old_record(d) test_failed_unlink_summary_keeps_recorded_ownership(d) test_manifest_without_source_ownership_is_refused(d) From deef61d13594aa8fcd0fcb1548d1d4d1385d9a7c Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:08:04 -0400 Subject: [PATCH 25/38] fix(prep): enforce manifest reader contract --- CHANGELOG.md | 4 ++-- docs/wake-threshold-tuning.md | 4 ++-- prep_wake_samples.py | 8 +++++++- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9525284..9168ded 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,8 +43,8 @@ All notable changes to computah are recorded here. The format follows being replaced by a partial source map, leaving fallback output explicitly unrecorded until the user reviews the directory and reboots ownership. That fallback cannot distinguish a prep leftover from a hand-added - `_NNN.wav`. A readable older manifest without source ownership is - refused until the user removes it to bootstrap ownership explicitly. + `_NNN.wav`. Any manifest version other than v2 is refused until the + user removes it to bootstrap ownership explicitly. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 2e95fc1..53ba394 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -80,8 +80,8 @@ unrecorded. That fallback has no provenance: `--clean` can remove any `_NNN.wav` file, including a hand-added file with that shape. Check the warning list and move any such file before cleaning. After reviewing the output directory, remove an unreadable manifest and rerun the full source set to -bootstrap ownership. A readable older manifest without source ownership is -refused and requires the same deliberate removal. +bootstrap ownership. Any manifest version other than v2 is refused, even if it +has similar fields, and requires the same deliberate removal. ```bash .venv/bin/python prep_wake_samples.py --input computah_normal.wav computah_styles.wav \ diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 4ef7320..f4b681a 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -380,7 +380,6 @@ def safe_clip_name(name: object) -> bool: isinstance(name, str) and name == Path(name).name and "/" not in name - and "\\" not in name and _CLIP_NAME.fullmatch(name) is not None ) @@ -391,6 +390,13 @@ def safe_clip_name(name: object) -> bool: ) return label if isinstance(label, str) else None, set(), None clip_names = set(clips) + version = raw.get("version") + if version != 2: + warn_manifest_problem( + f"unsupported manifest version {version!r}; expected 2", + legacy_fallback=not isinstance(label, str), + ) + return label if isinstance(label, str) else None, clip_names, None sources_raw = raw.get("sources") if not isinstance(sources_raw, dict): warn_manifest_problem( From c8f7c496500ee7c9b8e6903d1ba8a85092f3b5e8 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:15:38 -0400 Subject: [PATCH 26/38] test(prep): reproduce fail-open corrupt record --- test_prep_wake_samples.py | 65 ++++++++++++++++++++++++--------------- 1 file changed, 40 insertions(+), 25 deletions(-) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index fcb3215..8b49287 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -562,12 +562,28 @@ def test_manifest_cleans_dropped_input_orphans(d: Path) -> None: "a run records what it wrote in the manifest", ) - prep.process([src / "a" / "take.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) + errors = io.StringIO() + with redirect_stderr(errors): + prep.process( + [src / "a" / "take.wav"], + out, + "positive", + 0.3, + 0.2, + 3.0, + clean=True, + ) present = sorted(p.name for p in out.glob("*.wav")) check( present == ["take_000.wav", "take_001.wav", "take_002.wav"], f"--clean removes a dropped input's orphans ({present})", ) + check( + str(src / "b" / "other.wav") in errors.getvalue() + and "will remove" in errors.getvalue() + and "2 prep-owned clip" in errors.getvalue(), + "a dropped source and its deletion count are warned before cleanup", + ) def test_manifest_cleans_changed_collider_stem_orphans(d: Path) -> None: @@ -863,13 +879,8 @@ def test_manifest_spares_a_silent_take_alongside_a_good_one(d: Path) -> None: ) -def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: - """A corrupt manifest degrades --clean, it does not fail the run. - - The clips are the output that matters. Reading garbage as "no record" costs - precision on the orphans only the manifest can reach; treating it as fatal - would block a run that has real work to do. - """ +def test_manifest_unreadable_is_refused(d: Path) -> None: + """A corrupt manifest cannot disable label and source ownership guards.""" src = d / "corrupt_src" _burst_take(src / "a" / "take.wav", 3) _burst_take(src / "b" / "other.wav", 2) @@ -894,17 +905,20 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: ) present = sorted(p.name for p in out.glob("*.wav")) check( - total == 2 and "take_002.wav" not in present and "other_000.wav" in present, - f"a corrupt manifest leaves the stem rule working ({present})", - ) - check( - "manifest" in errors.getvalue() - and "legacy filename cleanup" in errors.getvalue(), - "a corrupt manifest warns that ownership protection is unavailable", + total == 0 + and present + == [ + "other_000.wav", + "other_001.wav", + "take_000.wav", + "take_001.wav", + "take_002.wav", + ], + f"a corrupt manifest refuses the run before any clip changes ({present})", ) check( - "hand-added" in errors.getvalue(), - "a corrupt-manifest clean warns that curated same-stem clips are deletable", + "manifest" in errors.getvalue() and "nothing was written" in errors.getvalue(), + "a corrupt manifest explains the fail-closed ownership refusal", ) check( manifest.read_text() == corrupt_payload, @@ -912,8 +926,8 @@ def test_manifest_unreadable_falls_back_to_stem_rule(d: Path) -> None: ) -def test_manifest_wrong_shape_warns_before_legacy_fallback(d: Path) -> None: - """Readable JSON with an invalid manifest shape must not degrade silently.""" +def test_manifest_wrong_shape_is_refused(d: Path) -> None: + """Readable JSON with an invalid manifest shape must fail closed.""" src = d / "malformed_manifest_src" take = src / "take.wav" _burst_take(take, 3) @@ -924,11 +938,12 @@ def test_manifest_wrong_shape_warns_before_legacy_fallback(d: Path) -> None: _burst_take(take, 2) errors = io.StringIO() with redirect_stderr(errors): - prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) + total = prep.process([take], out, "positive", 0.3, 0.2, 3.0, clean=True) check( - "manifest" in errors.getvalue() - and "legacy filename cleanup" in errors.getvalue(), - "a wrong-shape manifest warns before ownership protection degrades", + total == 0 + and "manifest" in errors.getvalue() + and "nothing was written" in errors.getvalue(), + "a wrong-shape manifest refuses to disable ownership protection", ) @@ -1387,8 +1402,8 @@ def main() -> int: test_manifest_empty_source_remains_refreshable(d) test_new_zero_output_source_does_not_gain_cleanup_authority(d) test_manifest_spares_a_silent_take_alongside_a_good_one(d) - test_manifest_unreadable_falls_back_to_stem_rule(d) - test_manifest_wrong_shape_warns_before_legacy_fallback(d) + test_manifest_unreadable_is_refused(d) + test_manifest_wrong_shape_is_refused(d) test_manifest_bad_sources_warns_before_ownership_refusal(d) test_manifest_clip_names_cannot_escape_output_directory(d) test_manifest_rejects_unsupported_version_with_v2_shape(d) From 46b77f68cc56a5e717b01eb97c2ca5b321933cc9 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:17:16 -0400 Subject: [PATCH 27/38] fix(prep): fail closed on unusable manifests --- CHANGELOG.md | 21 +++++----- docs/wake-threshold-tuning.md | 27 +++++++------ prep_wake_samples.py | 75 +++++++++++++---------------------- 3 files changed, 52 insertions(+), 71 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9168ded..2e406c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,9 +23,11 @@ All notable changes to computah are recorded here. The format follows path and the clips it owns in `.prep-manifest.json` in the output dir. `--clean` uses that provenance to remove orphans the stem match cannot reach -- clips from an input dropped between runs, and clips under a disambiguated stem - (`take-1_000.wav`) that a later run no longer produces. A run must share an - exact recorded source path before it may write to that dataset, so a different - dataset with the same label and generic basename cannot overwrite or clean it. + (`take-1_000.wav`) that a later run no longer produces. Before a destructive + run, prep names every omitted source and how many of its clips `--clean` will + remove. A run must share an exact recorded source path before it may write to + that dataset, so a different dataset with the same label and generic basename + cannot overwrite or clean it. Malformed ownership maps, including clip names that are paths instead of safe output filenames, are refused before any audio is decoded or written. Manifested stems remain reserved for their source while new output names are @@ -37,14 +39,11 @@ All notable changes to computah are recorded here. The format follows next `--clean` remains on the documented filename fallback instead of turning a guess into durable provenance. A no-output run cannot claim an unrecorded directory, and a first-time source that writes no clips cannot gain cleanup - authority over an existing dataset. Manifest writes use atomic replacement, - and an absent or unreadable manifest degrades `--clean` to the narrow stem - match with an explicit warning. An unreadable manifest is preserved instead of - being replaced by a partial source map, leaving fallback output explicitly - unrecorded until the user reviews the directory and reboots ownership. That - fallback cannot distinguish a prep leftover from a hand-added - `_NNN.wav`. Any manifest version other than v2 is refused until the - user removes it to bootstrap ownership explicitly. + authority over an existing dataset. Manifest writes use atomic replacement. + Only an absent manifest degrades `--clean` to the narrow stem match with an + explicit warning; that fallback cannot distinguish a prep leftover from a + hand-added `_NNN.wav`. An unreadable, malformed, or non-v2 manifest + fails closed until the user removes it to bootstrap ownership explicitly. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 53ba394..0eac2e4 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -69,19 +69,20 @@ least one source already recorded there in the same invocation. This gives prep explicit ownership proof while it adds the new source to the manifest. If you use `--clean`, include every source whose clips the dataset should keep: omitted sources are treated as intentionally dropped and their prep-owned clips are -removed. A safe incremental workflow is to add recordings without `--clean`, -then run a full-input `--clean` pass after reviewing the complete source list. - -Manifest updates use atomic replacement. An absent or unreadable manifest falls -back to the narrow same-stem cleanup rule with a warning that source ownership -protection is unavailable. An unreadable manifest is preserved rather than -replaced with a partial source map, so clips written during that fallback remain -unrecorded. That fallback has no provenance: `--clean` can remove any -`_NNN.wav` file, including a hand-added file with that shape. Check the -warning list and move any such file before cleaning. After reviewing the output -directory, remove an unreadable manifest and rerun the full source set to -bootstrap ownership. Any manifest version other than v2 is refused, even if it -has similar fields, and requires the same deliberate removal. +removed. Before decoding, prep warns with each omitted source path and the +number of its clips scheduled for removal. A safe incremental workflow is to add +recordings without `--clean`, then run a full-input `--clean` pass after +reviewing the complete source list. + +Manifest updates use atomic replacement. Only an absent manifest falls back to +the narrow same-stem cleanup rule, with a warning that source ownership +protection is unavailable. That fallback has no provenance: `--clean` can remove +any `_NNN.wav` file, including a hand-added file with that shape. +Check the warning list and move any such file before cleaning. An unreadable, +malformed, or non-v2 manifest fails closed before any decode or write because +neither its dataset label nor its source ownership can be trusted. After +reviewing the output directory, remove that manifest and rerun the full source +set to bootstrap ownership. ```bash .venv/bin/python prep_wake_samples.py --input computah_normal.wav computah_styles.wav \ diff --git a/prep_wake_samples.py b/prep_wake_samples.py index f4b681a..c99958b 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -347,15 +347,10 @@ def _read_manifest( """ manifest_path = out_dir / MANIFEST_NAME - def warn_manifest_problem(reason: object, *, legacy_fallback: bool) -> None: - consequence = ( - "this run will use legacy filename cleanup" - if legacy_fallback - else "this manifest cannot authorize a write" - ) + def warn_manifest_problem(reason: object) -> None: print( f"warning: could not use {manifest_path} ({reason}); source ownership " - f"protection is unavailable, so {consequence}", + "protection is unavailable, so this manifest cannot authorize a write", file=sys.stderr, ) @@ -364,14 +359,14 @@ def warn_manifest_problem(reason: object, *, legacy_fallback: bool) -> None: except FileNotFoundError: return None, set(), None except (OSError, ValueError) as e: - warn_manifest_problem(e, legacy_fallback=True) + warn_manifest_problem(e) return None, set(), None if not isinstance(raw, dict): - warn_manifest_problem("expected a JSON object", legacy_fallback=True) + warn_manifest_problem("expected a JSON object") return None, set(), None clips = raw.get("clips") if not isinstance(clips, list): - warn_manifest_problem("expected a clips list", legacy_fallback=True) + warn_manifest_problem("expected a clips list") return None, set(), None label = raw.get("label") @@ -384,47 +379,32 @@ def safe_clip_name(name: object) -> bool: ) if not all(safe_clip_name(name) for name in clips): - warn_manifest_problem( - "clips must contain safe output filenames, not paths", - legacy_fallback=not isinstance(label, str), - ) + warn_manifest_problem("clips must contain safe output filenames, not paths") return label if isinstance(label, str) else None, set(), None clip_names = set(clips) version = raw.get("version") if version != 2: - warn_manifest_problem( - f"unsupported manifest version {version!r}; expected 2", - legacy_fallback=not isinstance(label, str), - ) + warn_manifest_problem(f"unsupported manifest version {version!r}; expected 2") return label if isinstance(label, str) else None, clip_names, None sources_raw = raw.get("sources") if not isinstance(sources_raw, dict): - warn_manifest_problem( - "expected a sources map", legacy_fallback=not isinstance(label, str) - ) + warn_manifest_problem("expected a sources map") return label if isinstance(label, str) else None, clip_names, None sources: dict[str, set[str]] = {} for source, names in sources_raw.items(): if not isinstance(source, str) or not isinstance(names, list): warn_manifest_problem( - "expected every sources entry to map a path to a clip list", - legacy_fallback=not isinstance(label, str), + "expected every sources entry to map a path to a clip list" ) return label if isinstance(label, str) else None, clip_names, None if not all(safe_clip_name(name) for name in names): - warn_manifest_problem( - "sources must own safe output filenames, not paths", - legacy_fallback=not isinstance(label, str), - ) + warn_manifest_problem("sources must own safe output filenames, not paths") return label if isinstance(label, str) else None, clip_names, None sources[source] = set(names) owned_clips = set().union(*sources.values()) if sources else set() if owned_clips != clip_names: - warn_manifest_problem( - "sources do not own exactly the recorded clips", - legacy_fallback=not isinstance(label, str), - ) + warn_manifest_problem("sources do not own exactly the recorded clips") return label if isinstance(label, str) else None, clip_names, None return label if isinstance(label, str) else None, clip_names, sources @@ -625,12 +605,13 @@ def process( file=sys.stderr, ) return 0 - if prior_label is not None and prior_sources is None: + if manifest_present and prior_sources is None: print( - f"error: --output {out_dir} has a {MANIFEST_NAME} without source " - "ownership, so this run cannot prove it belongs there; nothing was " - f"written. Remove {out_dir / MANIFEST_NAME} to bootstrap ownership " - "from these inputs, or clear the directory first", + f"error: --output {out_dir} has an unusable {MANIFEST_NAME}, so this " + "run cannot prove its label or source ownership; nothing was written. " + f"After reviewing the directory, remove {out_dir / MANIFEST_NAME} to " + "bootstrap ownership from the full input set, or clear the directory " + "first", file=sys.stderr, ) return 0 @@ -672,6 +653,16 @@ def process( file=sys.stderr, ) return 0 + if clean and prior_sources is not None: + present_names = {path.name for path in _stale_clips(out_dir, set())} + for source in sorted(prior_sources.keys() - set(source_keys)): + count = len(prior_sources[source] & present_names) + if count: + print( + f"warning: source {source} is not in this run; --clean will " + f"remove its {count} prep-owned clip(s) from {out_dir}", + file=sys.stderr, + ) stems = _manifest_aware_stems(files, source_keys, prior_sources) written: set[Path] = set() @@ -783,17 +774,7 @@ def process( # A no-output run cannot establish ownership in an unrecorded directory. # Leaving it unclaimed lets a later successful rerun bootstrap the legacy # clips instead of stranding them behind an empty manifest. - unusable_manifest = manifest_present and prior_sources is None - if unusable_manifest: - if written: - print( - f"warning: the unreadable {out_dir / MANIFEST_NAME} was preserved; " - "clips written by this run are unrecorded. After reviewing the " - "output directory, remove that manifest and rerun the full input " - "set to bootstrap ownership again", - file=sys.stderr, - ) - elif (prior_sources is not None or written) and not ambiguous_legacy: + if (prior_sources is not None or written) and not ambiguous_legacy: _write_manifest(out_dir, label, next_sources) elif ambiguous_legacy: print( From e06013a95f558cc2f01f7196ed49a79cde0fae15 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:36:04 -0400 Subject: [PATCH 28/38] test(prep): reproduce untrusted manifest label --- test_prep_wake_samples.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 8b49287..d515d24 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -1036,6 +1036,35 @@ def test_manifest_rejects_unsupported_version_with_v2_shape(d: Path) -> None: ) +def test_manifest_requires_a_string_label(d: Path) -> None: + """A v2 source map cannot authorize writes without a trusted label.""" + src = d / "missing_label_src" + take = src / "take.wav" + _burst_take(take, 2) + out = d / "missing_label_out" + out.mkdir() + (out / prep.MANIFEST_NAME).write_text( + json.dumps( + { + "version": 2, + "clips": ["take_000.wav"], + "sources": {prep._source_key(take): ["take_000.wav"]}, + } + ) + ) + + errors = io.StringIO() + with redirect_stderr(errors): + total = prep.process([take], out, "positive", 0.3, 0.2, 3.0) + check( + total == 0 + and not list(out.glob("*.wav")) + and "label" in errors.getvalue() + and "nothing was written" in errors.getvalue(), + "a v2 manifest without a string label fails closed", + ) + + def test_posix_backslash_basename_round_trips_manifest(d: Path) -> None: """A POSIX basename the writer accepts must remain readable next run.""" if os.sep != "/": @@ -1407,6 +1436,7 @@ def main() -> int: test_manifest_bad_sources_warns_before_ownership_refusal(d) test_manifest_clip_names_cannot_escape_output_directory(d) test_manifest_rejects_unsupported_version_with_v2_shape(d) + test_manifest_requires_a_string_label(d) test_posix_backslash_basename_round_trips_manifest(d) test_manifest_replace_failure_describes_authoritative_old_record(d) test_failed_unlink_summary_keeps_recorded_ownership(d) From 8113fff8e74a88bf022f2d5f2a064aea2324e62b Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:37:56 -0400 Subject: [PATCH 29/38] fix(prep): close manifest bootstrap contract --- CHANGELOG.md | 2 ++ docs/recording-computah.md | 6 ++++++ docs/wake-threshold-tuning.md | 7 +++++++ prep_wake_samples.py | 39 ++++++++++++++++------------------- test_prep_wake_samples.py | 6 +++--- 5 files changed, 36 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e406c8..6b0b3be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,8 @@ All notable changes to computah are recorded here. The format follows explicit warning; that fallback cannot distinguish a prep leftover from a hand-added `_NNN.wav`. An unreadable, malformed, or non-v2 manifest fails closed until the user removes it to bootstrap ownership explicitly. + Existing pre-manifest directories must bootstrap with the complete source set; + clips under stems omitted from a partial first run remain deliberately unowned. - Configurable request endpointing (#15): the trailing-silence window that ends a captured request and the max-request cap that bounds a runaway are now config keys (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants, diff --git a/docs/recording-computah.md b/docs/recording-computah.md index 39dab20..e1c825c 100644 --- a/docs/recording-computah.md +++ b/docs/recording-computah.md @@ -76,6 +76,12 @@ reports per-file segment counts and a duration summary, and writes 16 kHz mono i WAVs ready for verifier training. `samples/` is gitignored — the recordings are personal data and stay out of version control. +When adding manifests to an existing `samples/` directory, run the first refresh +with the complete source list for that label. A partial bootstrap cannot prove +ownership of legacy clips under omitted stems, so those files remain manual-only +leftovers. Move or clear the old output first if the complete source set is no +longer available. + Once an output directory has a source manifest, an incremental run must include at least one recording already listed there along with any new recording. Keep added microphone takes under the same positive-file glob, or list them beside an existing diff --git a/docs/wake-threshold-tuning.md b/docs/wake-threshold-tuning.md index 0eac2e4..62287d3 100644 --- a/docs/wake-threshold-tuning.md +++ b/docs/wake-threshold-tuning.md @@ -64,6 +64,13 @@ that produced nothing. A new source that produces no clips is not added to the ownership record, so it cannot authorize a later cleanup until a successful run actually contributes audio. +When upgrading an existing pre-manifest `samples/` directory, bootstrap it with +the complete source list for that label. A partial first run can record only the +sources it receives; legacy clips under omitted stems then remain deliberately +unowned and future `--clean` runs will only warn about them. If the complete +source set is unavailable, move or clear the old output before starting a new +partial dataset. + To add a new recording to an existing dataset without `--clean`, include at least one source already recorded there in the same invocation. This gives prep explicit ownership proof while it adds the new source to the manifest. If you diff --git a/prep_wake_samples.py b/prep_wake_samples.py index c99958b..0755b7b 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -338,12 +338,11 @@ def _read_manifest( ) -> tuple[str | None, set[str], dict[str, set[str]] | None]: """The label, clips, and source ownership a previous prep run recorded here. - Fails soft on every read problem -- absent, unreadable, not JSON, wrong - shape -- returning ``(None, set(), None)``. An empty record means "no record - of what prep made," which narrows `--clean` back to the stem rule below. - A version-1 record still returns its label and clips but no source map; the - caller can then refuse to guess ownership and tell the user how to bootstrap - a new record explicitly. + An absent file returns ``(None, set(), None)`` and lets the caller use the + warned legacy stem rule. Every present-but-unreadable, malformed, or + unsupported record returns no source map; the caller sees that the file + exists and refuses to decode or write until ownership is bootstrapped + explicitly. """ manifest_path = out_dir / MANIFEST_NAME @@ -369,6 +368,9 @@ def warn_manifest_problem(reason: object) -> None: warn_manifest_problem("expected a clips list") return None, set(), None label = raw.get("label") + if not isinstance(label, str): + warn_manifest_problem("expected a string label") + return None, set(), None def safe_clip_name(name: object) -> bool: return ( @@ -380,16 +382,16 @@ def safe_clip_name(name: object) -> bool: if not all(safe_clip_name(name) for name in clips): warn_manifest_problem("clips must contain safe output filenames, not paths") - return label if isinstance(label, str) else None, set(), None + return label, set(), None clip_names = set(clips) version = raw.get("version") if version != 2: warn_manifest_problem(f"unsupported manifest version {version!r}; expected 2") - return label if isinstance(label, str) else None, clip_names, None + return label, clip_names, None sources_raw = raw.get("sources") if not isinstance(sources_raw, dict): warn_manifest_problem("expected a sources map") - return label if isinstance(label, str) else None, clip_names, None + return label, clip_names, None sources: dict[str, set[str]] = {} for source, names in sources_raw.items(): @@ -397,16 +399,16 @@ def safe_clip_name(name: object) -> bool: warn_manifest_problem( "expected every sources entry to map a path to a clip list" ) - return label if isinstance(label, str) else None, clip_names, None + return label, clip_names, None if not all(safe_clip_name(name) for name in names): warn_manifest_problem("sources must own safe output filenames, not paths") - return label if isinstance(label, str) else None, clip_names, None + return label, clip_names, None sources[source] = set(names) owned_clips = set().union(*sources.values()) if sources else set() if owned_clips != clip_names: warn_manifest_problem("sources do not own exactly the recorded clips") - return label if isinstance(label, str) else None, clip_names, None - return label if isinstance(label, str) else None, clip_names, sources + return label, clip_names, None + return label, clip_names, sources def _write_manifest( @@ -621,15 +623,10 @@ def process( and out_dir.is_dir() and any(path.suffix.lower() in AUDIO_EXTS for path in out_dir.iterdir()) ): - manifest_state = ( - f"{out_dir / MANIFEST_NAME} is unusable" - if manifest_present - else f"{out_dir} has no {MANIFEST_NAME}" - ) print( - f"warning: {manifest_state}; this --clean will use legacy filename " - "cleanup without source ownership protection and may remove hand-added " - "_NNN.wav files", + f"warning: {out_dir} has no {MANIFEST_NAME}; this --clean will use " + "legacy filename cleanup without source ownership protection and may " + "remove hand-added _NNN.wav files", file=sys.stderr, ) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index d515d24..ac390cb 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -1183,9 +1183,9 @@ def test_manifest_bootstrap_keeps_legacy_leftovers_cleanable(d: Path) -> None: A normal refresh without --clean warns that its higher-numbered leftovers can be removed by rerunning with --clean. When that refresh is also the run - that introduces the manifest, the record has to retain provenance for those - leftovers; otherwise the promised follow-up clean can no longer distinguish - them from hand-added audio. + that would introduce the manifest, prep withholds the record while ambiguous + same-stem files remain. The follow-up clean therefore stays on the legacy + filename rule, removes those leftovers, and only then records v2 ownership. """ src = d / "bootstrap_src" take = src / "take.wav" From 30aff9defb09327fbfd318adc36a127cac7a2072 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:43:28 -0400 Subject: [PATCH 30/38] test(prep): reproduce legacy mixed warning --- test_prep_wake_samples.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index ac390cb..732bca2 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -774,6 +774,34 @@ def test_nonclean_silent_rerun_protects_only_copy_in_warning(d: Path) -> None: ) +def test_legacy_mixed_warning_keeps_cleanable_leftovers_automated(d: Path) -> None: + """Legacy leftovers beside a silent take remain --clean-removable.""" + src = d / "legacy_mixed_src" + good = src / "good.wav" + silent = src / "silent.wav" + _burst_take(good, 2) + _burst_take(silent, 2) + out = d / "legacy_mixed_out" + prep.process([good, silent], out, "positive", 0.3, 0.2, 3.0) + (out / prep.MANIFEST_NAME).unlink() + + _burst_take(good, 1) + sf.write( + silent, + np.ones(int(prep.TARGET_SR * 4.0), dtype=np.float32) * 0.5, + prep.TARGET_SR, + subtype="PCM_16", + ) + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([good, silent], out, "positive", 0.3, 0.2, 1.0) + check( + "check the recording" in errors.getvalue() + and "rerun with --clean to remove prep-owned good_001.wav" in errors.getvalue(), + "legacy mixed guidance protects silent clips and automates true leftovers", + ) + + def test_manifest_empty_source_remains_refreshable(d: Path) -> None: """An owned source with no clips must not deadlock its output directory.""" src = d / "empty_source_src" @@ -1428,6 +1456,7 @@ def main() -> int: test_manifest_never_authorizes_deleting_curated_audio(d) test_manifest_spares_prior_clips_when_rerun_writes_nothing(d) test_nonclean_silent_rerun_protects_only_copy_in_warning(d) + test_legacy_mixed_warning_keeps_cleanable_leftovers_automated(d) test_manifest_empty_source_remains_refreshable(d) test_new_zero_output_source_does_not_gain_cleanup_authority(d) test_manifest_spares_a_silent_take_alongside_a_good_one(d) From 3a20ba3d7d56a3acd0c9c40e5a311117ac6efd6d Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 22:44:36 -0400 Subject: [PATCH 31/38] fix(prep): align legacy cleanup guidance --- CHANGELOG.md | 12 ++++++------ prep_wake_samples.py | 10 +++++++++- test_prep_wake_samples.py | 2 +- 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b0b3be..4b6c7fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,12 +13,12 @@ All notable changes to computah are recorded here. The format follows higher-numbered clips that training and eval would still read. Off by default (deletion is opt-in). The source manifest below broadens cleanup to the prep-owned clips of an intentionally omitted take while still sparing source - recordings and hand-curated audio. With no manifest or an unreadable one, the - narrow same-stem rule remains as a warned legacy fallback; readable older - records without source ownership fail closed until removed. Without `--clean` - the run still warns and names the leftover count. `test_prep_wake_samples.py` - covers the re-run-with-fewer-clips case with and without `--clean`, and that - source-aware `--clean` spares files prep did not record. + recordings and hand-curated audio. With no manifest, the narrow same-stem rule + remains as a warned legacy fallback; any present-but-unusable record fails + closed until removed. Without `--clean` the run still warns and names the + leftover count. `test_prep_wake_samples.py` covers the + re-run-with-fewer-clips case with and without `--clean`, and that source-aware + `--clean` spares files prep did not record. - `prep_wake_samples.py` output manifest (#84): prep records each resolved source path and the clips it owns in `.prep-manifest.json` in the output dir. `--clean` uses that provenance to remove orphans the stem match cannot reach -- diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 0755b7b..f308f91 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -827,7 +827,15 @@ def process( "they are not wanted" ) else: - cleanable = [p for p in other_lingering if p.name in owned_names] + cleanable = [ + p + for p in other_lingering + if ( + p.name in owned_names + if prior_sources is not None + else _clip_stem(p.name) in run_stems + ) + ] manual = [p for p in other_lingering if p not in cleanable] if cleanable: fix += ( diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 732bca2..0c22056 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -797,7 +797,7 @@ def test_legacy_mixed_warning_keeps_cleanable_leftovers_automated(d: Path) -> No prep.process([good, silent], out, "positive", 0.3, 0.2, 1.0) check( "check the recording" in errors.getvalue() - and "rerun with --clean to remove prep-owned good_001.wav" in errors.getvalue(), + and "Rerun with --clean to remove prep-owned good_001.wav" in errors.getvalue(), "legacy mixed guidance protects silent clips and automates true leftovers", ) From 7b3f25935c6f6ec1d27f0f473dbb311a5db60d55 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 23:02:53 -0400 Subject: [PATCH 32/38] test(prep): reproduce warning control injection --- test_prep_wake_samples.py | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 0c22056..49f6baa 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -1001,6 +1001,40 @@ def test_manifest_bad_sources_warns_before_ownership_refusal(d: Path) -> None: ) +def test_manifest_source_warning_escapes_control_characters(d: Path) -> None: + """An untrusted source key cannot forge a destructive-warning line.""" + src = d / "escaped_source_warning_src" + current = src / "current.wav" + _burst_take(current, 2) + out = d / "escaped_source_warning_out" + out.mkdir() + _burst_take(out / "current_000.wav", 1) + _burst_take(out / "old_000.wav", 1) + hostile_source = "old.wav\nwarning: forged" + (out / prep.MANIFEST_NAME).write_text( + json.dumps( + { + "version": 2, + "label": "positive", + "clips": ["current_000.wav", "old_000.wav"], + "sources": { + prep._source_key(current): ["current_000.wav"], + hostile_source: ["old_000.wav"], + }, + } + ) + ) + + errors = io.StringIO() + with redirect_stderr(errors): + prep.process([current], out, "positive", 0.3, 0.2, 3.0, clean=True) + check( + "old.wav\\nwarning: forged" in errors.getvalue() + and hostile_source not in errors.getvalue(), + "a manifest source key is escaped in the pre-clean warning", + ) + + def test_manifest_clip_names_cannot_escape_output_directory(d: Path) -> None: """Manifest ownership must never turn a path into an output stem.""" src = d / "hostile_manifest_src" @@ -1463,6 +1497,7 @@ def main() -> int: test_manifest_unreadable_is_refused(d) test_manifest_wrong_shape_is_refused(d) test_manifest_bad_sources_warns_before_ownership_refusal(d) + test_manifest_source_warning_escapes_control_characters(d) test_manifest_clip_names_cannot_escape_output_directory(d) test_manifest_rejects_unsupported_version_with_v2_shape(d) test_manifest_requires_a_string_label(d) From 4eb483595e5f2d951a7da8e2607a2221d7deeb7c Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 23:03:32 -0400 Subject: [PATCH 33/38] fix(prep): escape manifest source warnings --- prep_wake_samples.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prep_wake_samples.py b/prep_wake_samples.py index f308f91..966c58f 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -656,7 +656,7 @@ def process( count = len(prior_sources[source] & present_names) if count: print( - f"warning: source {source} is not in this run; --clean will " + f"warning: source {source!r} is not in this run; --clean will " f"remove its {count} prep-owned clip(s) from {out_dir}", file=sys.stderr, ) From b860eb4e2e055c7327a6f48505f2b6babc00c019 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 23:24:06 -0400 Subject: [PATCH 34/38] test(prep): reproduce manifest trust gaps --- test_prep_wake_samples.py | 76 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index 49f6baa..c656397 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -579,7 +579,7 @@ def test_manifest_cleans_dropped_input_orphans(d: Path) -> None: f"--clean removes a dropped input's orphans ({present})", ) check( - str(src / "b" / "other.wav") in errors.getvalue() + prep._source_key(src / "b" / "other.wav") in errors.getvalue() and "will remove" in errors.getvalue() and "2 prep-owned clip" in errors.getvalue(), "a dropped source and its deletion count are warned before cleanup", @@ -1001,6 +1001,78 @@ def test_manifest_bad_sources_warns_before_ownership_refusal(d: Path) -> None: ) +def test_manifest_rejects_overlapping_source_ownership(d: Path) -> None: + """Two sources cannot both authorize deletion of the same clip.""" + src = d / "overlapping_owners_src" + current = src / "b" / "take.wav" + _burst_take(current, 2) + out = d / "overlapping_owners_out" + out.mkdir() + _burst_take(out / "take_000.wav", 1) + manifest = out / prep.MANIFEST_NAME + manifest.write_text( + json.dumps( + { + "version": 2, + "label": "positive", + "clips": ["take_000.wav"], + "sources": { + str((src / "a" / "take.wav").resolve()): ["take_000.wav"], + prep._source_key(current): ["take_000.wav"], + }, + } + ) + ) + before = (out / "take_000.wav").read_bytes() + + errors = io.StringIO() + with redirect_stderr(errors): + total = prep.process([current], out, "positive", 0.3, 0.2, 3.0, clean=True) + check( + total == 0 + and (out / "take_000.wav").read_bytes() == before + and "ownership" in errors.getvalue() + and "nothing was written" in errors.getvalue(), + "overlapping source ownership fails closed before cleanup", + ) + + +def test_manifest_reader_requests_utf8(d: Path) -> None: + """Manifest paths must decode with the UTF-8 encoding used by the writer.""" + out = d / "utf8_manifest_out" + out.mkdir() + (out / prep.MANIFEST_NAME).write_text( + json.dumps( + { + "version": 2, + "label": "positive", + "clips": [], + "sources": {"/recordings/José/take.wav": []}, + }, + ensure_ascii=False, + ), + encoding="utf-8", + ) + original_read_text = prep.Path.read_text + requested: list[str | None] = [] + + def require_utf8(path: Path, *args: object, **kwargs: object) -> str: + requested.append(kwargs.get("encoding")) + if kwargs.get("encoding") != "utf-8": + raise UnicodeDecodeError("ascii", b"\x81", 0, 1, "simulated locale") + return original_read_text(path, *args, **kwargs) + + try: + prep.Path.read_text = require_utf8 + label, _clips, sources = prep._read_manifest(out) + finally: + prep.Path.read_text = original_read_text + check( + requested == ["utf-8"] and label == "positive" and sources is not None, + "the manifest reader explicitly decodes UTF-8", + ) + + def test_manifest_source_warning_escapes_control_characters(d: Path) -> None: """An untrusted source key cannot forge a destructive-warning line.""" src = d / "escaped_source_warning_src" @@ -1497,6 +1569,8 @@ def main() -> int: test_manifest_unreadable_is_refused(d) test_manifest_wrong_shape_is_refused(d) test_manifest_bad_sources_warns_before_ownership_refusal(d) + test_manifest_rejects_overlapping_source_ownership(d) + test_manifest_reader_requests_utf8(d) test_manifest_source_warning_escapes_control_characters(d) test_manifest_clip_names_cannot_escape_output_directory(d) test_manifest_rejects_unsupported_version_with_v2_shape(d) From 6379541892d72abbd29792abbaf463cfc8b4b365 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 23:24:39 -0400 Subject: [PATCH 35/38] fix(prep): validate manifest ownership encoding --- CHANGELOG.md | 4 +++- prep_wake_samples.py | 15 +++++++++++---- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b6c7fa..219974b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,9 @@ All notable changes to computah are recorded here. The format follows that dataset, so a different dataset with the same label and generic basename cannot overwrite or clean it. Malformed ownership maps, including clip names that are paths instead of safe - output filenames, are refused before any audio is decoded or written. + output filenames or clips claimed by multiple sources, are refused before any + audio is decoded or written. Manifests are read and written explicitly as + UTF-8 so non-ASCII recording paths round-trip across supported platforms. Manifested stems remain reserved for their source while new output names are assigned, so a new same-basename recording cannot overwrite the only good clips from a source whose rerun is silent. A source this run attempted but got diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 966c58f..8a06547 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -354,7 +354,7 @@ def warn_manifest_problem(reason: object) -> None: ) try: - raw = json.loads(manifest_path.read_text()) + raw = json.loads(manifest_path.read_text(encoding="utf-8")) except FileNotFoundError: return None, set(), None except (OSError, ValueError) as e: @@ -394,6 +394,7 @@ def safe_clip_name(name: object) -> bool: return label, clip_names, None sources: dict[str, set[str]] = {} + claimed_clips: set[str] = set() for source, names in sources_raw.items(): if not isinstance(source, str) or not isinstance(names, list): warn_manifest_problem( @@ -403,9 +404,15 @@ def safe_clip_name(name: object) -> bool: if not all(safe_clip_name(name) for name in names): warn_manifest_problem("sources must own safe output filenames, not paths") return label, clip_names, None - sources[source] = set(names) - owned_clips = set().union(*sources.values()) if sources else set() - if owned_clips != clip_names: + source_clips = set(names) + if claimed_clips & source_clips: + warn_manifest_problem( + "each recorded clip must have exactly one source owner" + ) + return label, clip_names, None + sources[source] = source_clips + claimed_clips.update(source_clips) + if claimed_clips != clip_names: warn_manifest_problem("sources do not own exactly the recorded clips") return label, clip_names, None return label, clip_names, sources From 1f06a24882852fd0b1c15a45da95e52d27f55e45 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 23:43:01 -0400 Subject: [PATCH 36/38] test(prep): reproduce final manifest diagnostics --- test_prep_wake_samples.py | 56 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 4 deletions(-) diff --git a/test_prep_wake_samples.py b/test_prep_wake_samples.py index c656397..6994416 100644 --- a/test_prep_wake_samples.py +++ b/test_prep_wake_samples.py @@ -1139,6 +1139,39 @@ def test_manifest_clip_names_cannot_escape_output_directory(d: Path) -> None: ) +def test_manifest_clip_names_reject_embedded_nul(d: Path) -> None: + """A NUL-bearing manifest stem must fail closed instead of crashing.""" + src = d / "nul_manifest_src" + take = src / "take.wav" + _burst_take(take, 2) + out = d / "nul_manifest_out" + out.mkdir() + hostile_name = "a\u0000b_000.wav" + (out / prep.MANIFEST_NAME).write_text( + json.dumps( + { + "version": 2, + "label": "positive", + "clips": [hostile_name], + "sources": {prep._source_key(take): [hostile_name]}, + } + ) + ) + + errors = io.StringIO() + try: + with redirect_stderr(errors): + total = prep.process([take], out, "positive", 0.3, 0.2, 3.0) + except ValueError: + total = -1 + check( + total == 0 + and "manifest" in errors.getvalue() + and "nothing was written" in errors.getvalue(), + "an embedded NUL is refused before a manifest stem reaches the filesystem", + ) + + def test_manifest_rejects_unsupported_version_with_v2_shape(d: Path) -> None: """A version-1 declaration cannot gain v2 cleanup authority by shape.""" src = d / "unsupported_version_src" @@ -1524,15 +1557,29 @@ def test_same_label_stray_is_refused_before_writing(d: Path) -> None: prep.process([src / "noise.wav"], out, "positive", 0.3, 0.2, 3.0) before = sorted(p.name for p in out.glob("*.wav")) - totals = [ - prep.process([src / "computah.wav"], out, "positive", 0.3, 0.2, 3.0, clean=True) - for _ in range(2) - ] + errors = io.StringIO() + with redirect_stderr(errors): + totals = [ + prep.process( + [src / "computah.wav"], + out, + "positive", + 0.3, + 0.2, + 3.0, + clean=True, + ) + for _ in range(2) + ] present = sorted(p.name for p in out.glob("*.wav")) check( totals == [0, 0] and present == before, f"a repeated same-label stray writes nothing ({present})", ) + check( + repr(prep._source_key(src / "noise.wav")) in errors.getvalue(), + "a no-overlap refusal names an escaped recorded source", + ) def main() -> int: @@ -1573,6 +1620,7 @@ def main() -> int: test_manifest_reader_requests_utf8(d) test_manifest_source_warning_escapes_control_characters(d) test_manifest_clip_names_cannot_escape_output_directory(d) + test_manifest_clip_names_reject_embedded_nul(d) test_manifest_rejects_unsupported_version_with_v2_shape(d) test_manifest_requires_a_string_label(d) test_posix_backslash_basename_round_trips_manifest(d) From 0d1e2c34e213f09432f10ba6922c8d940a8c6313 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Thu, 23 Jul 2026 23:43:38 -0400 Subject: [PATCH 37/38] fix(prep): finish manifest boundary diagnostics --- prep_wake_samples.py | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/prep_wake_samples.py b/prep_wake_samples.py index 8a06547..fb5080c 100644 --- a/prep_wake_samples.py +++ b/prep_wake_samples.py @@ -11,6 +11,9 @@ counts and a duration summary so a bad take (clipped words, no gaps, silence) is obvious before training. +Successful runs also write `.prep-manifest.json` in `--output`; it records the +dataset label and exact source ownership used to guard later refreshes and cleanup. + Recordings are personal data: write them under samples/ (gitignored), not into the tree. @@ -377,6 +380,7 @@ def safe_clip_name(name: object) -> bool: isinstance(name, str) and name == Path(name).name and "/" not in name + and "\x00" not in name and _CLIP_NAME.fullmatch(name) is not None ) @@ -649,11 +653,15 @@ def process( # overwrite take_000.wav before cleanup runs. Add a new source alongside one # recorded source in the same invocation to prove which dataset owns the dir. if prior_sources is not None and not (set(source_keys) & prior_sources.keys()): + recorded = sorted(prior_sources) + preview = ", ".join(repr(source) for source in recorded[:2]) + if len(recorded) > 2: + preview += f", +{len(recorded) - 2} more" print( f"error: --output {out_dir} belongs to a {label!r} dataset with " - "different source recordings; nothing was written. Point --output at " - f"this run's directory, or clear that one (or remove its {MANIFEST_NAME}) " - "to reuse it", + f"different source recordings; nothing was written. Recorded sources " + f"include {preview}. Point --output at this run's directory, or clear " + f"that one (or remove its {MANIFEST_NAME}) to reuse it", file=sys.stderr, ) return 0 @@ -924,7 +932,11 @@ def main() -> int: nargs="+", help="audio file(s), a shell glob, or a folder of them", ) - p.add_argument("--output", required=True, help="output directory for clips") + p.add_argument( + "--output", + required=True, + help=f"output directory for clips and the {MANIFEST_NAME} ownership record", + ) p.add_argument( "--label", choices=("positive", "negative", "background"), @@ -948,8 +960,8 @@ def main() -> int: action="store_true", help="remove prep's own leftover clips in --output: a re-recorded take's " "now-unused higher-numbered clips, plus clips a previous run recorded for a " - "take no longer in the inputs; without a readable manifest, falls back to " - "same-stem filenames", + "take no longer in the inputs; when no manifest exists, falls back to " + "same-stem filenames, while an unusable manifest fails closed", ) args = p.parse_args() From 53120a2b7a4065a26f9cb4224aa44efb5cb87ee9 Mon Sep 17 00:00:00 2001 From: Joe Amditis <6799804+jamditis@users.noreply.github.com> Date: Fri, 24 Jul 2026 00:05:52 -0400 Subject: [PATCH 38/38] chore(prep): ignore ownership manifests --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index f26b60a..ebd613a 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,7 @@ test_audio/* # Wake-word recordings and processed clips — personal voice data, never committed samples/ recordings/ +.prep-manifest.json # Local experiment/training scripts (machine-specific paths), not part of the published project experiments/