Record what prep wrote so --clean can reach the orphans stems cannot (#84)

.gitignore

@@ -20,6 +20,7 @@ test_audio/*
 # Wake-word recordings and processed clips — personal voice data, never committed
 samples/
 recordings/
+.prep-manifest.json
 
 # Local experiment/training scripts (machine-specific paths), not part of the published project
 experiments/

CHANGELOG.md

@@ -11,12 +11,43 @@ All notable changes to computah are recorded here. The format follows
   just re-recorded (a `<stem>_NNN.wav` clip whose stem this run wrote) from
   `--output`, so re-recording with fewer utterances no longer strands orphaned
   higher-numbered clips that training and eval would still read. Off by default
-  (deletion is opt-in) and deliberately narrow: a source recording, an unrelated
-  take's clips, or a hand-curated clip whose stem this run did not write is named
-  but never auto-deleted. Without `--clean` the run still warns and names the
-  leftover count. `test_prep_wake_samples.py` covers the re-run-with-fewer-clips
-  case with and without `--clean`, and that `--clean` spares files this run did
-  not record.
+  (deletion is opt-in). The source manifest below broadens cleanup to the
+  prep-owned clips of an intentionally omitted take while still sparing source
+  recordings and hand-curated audio. With no manifest, the narrow same-stem rule
+  remains as a warned legacy fallback; any present-but-unusable record fails
+  closed until removed. Without `--clean` the run still warns and names the
+  leftover count. `test_prep_wake_samples.py` covers the
+  re-run-with-fewer-clips case with and without `--clean`, and that source-aware
+  `--clean` spares files prep did not record.
+- `prep_wake_samples.py` output manifest (#84): prep records each resolved source
+  path and the clips it owns in `.prep-manifest.json` in the output dir.
+  `--clean` uses that provenance to remove orphans the stem match cannot reach --
+  clips from an input dropped between runs, and clips under a disambiguated stem
+  (`take-1_000.wav`) that a later run no longer produces. Before a destructive
+  run, prep names every omitted source and how many of its clips `--clean` will
+  remove. A run must share an exact recorded source path before it may write to
+  that dataset, so a different dataset with the same label and generic basename
+  cannot overwrite or clean it.
+  Malformed ownership maps, including clip names that are paths instead of safe
+  output filenames or clips claimed by multiple sources, are refused before any
+  audio is decoded or written. Manifests are read and written explicitly as
+  UTF-8 so non-ASCII recording paths round-trip across supported platforms.
+  Manifested stems remain reserved for their source while new output names are
+  assigned, so a new same-basename recording cannot overwrite the only good
+  clips from a source whose rerun is silent. A source this run attempted but got
+  no clips from retains those prior clips. With a readable source map,
+  hand-curated audio stays outside it and is never deleted. A non-clean legacy
+  refresh that leaves ambiguous same-stem files does not write a manifest; the
+  next `--clean` remains on the documented filename fallback instead of turning
+  a guess into durable provenance. A no-output run cannot claim an unrecorded
+  directory, and a first-time source that writes no clips cannot gain cleanup
+  authority over an existing dataset. Manifest writes use atomic replacement.
+  Only an absent manifest degrades `--clean` to the narrow stem match with an
+  explicit warning; that fallback cannot distinguish a prep leftover from a
+  hand-added `<same-stem>_NNN.wav`. An unreadable, malformed, or non-v2 manifest
+  fails closed until the user removes it to bootstrap ownership explicitly.
+  Existing pre-manifest directories must bootstrap with the complete source set;
+  clips under stems omitted from a partial first run remain deliberately unowned.
 - Configurable request endpointing (#15): the trailing-silence window that ends a
   captured request and the max-request cap that bounds a runaway are now config keys
   (`endpoint_silence_ms`, `max_request_ms`, milliseconds) instead of fixed constants,

docs/recording-computah.md

@@ -76,6 +76,18 @@ reports per-file segment counts and a duration summary, and writes 16 kHz mono i
 WAVs ready for verifier training. `samples/` is gitignored — the recordings are
 personal data and stay out of version control.
 
+When adding manifests to an existing `samples/` directory, run the first refresh
+with the complete source list for that label. A partial bootstrap cannot prove
+ownership of legacy clips under omitted stems, so those files remain manual-only
+leftovers. Move or clear the old output first if the complete source set is no
+longer available.
+
+Once an output directory has a source manifest, an incremental run must include at
+least one recording already listed there along with any new recording. Keep added
+microphone takes under the same positive-file glob, or list them beside an existing
+source explicitly. With `--clean`, pass every source whose clips the dataset should
+keep; omitted sources are treated as intentionally dropped.
+
 ## Training
 
 Training runs externally (openWakeWord 0.4.0 ships no train submodule) on a GPU host.

docs/wake-threshold-tuning.md

@@ -33,14 +33,67 @@ repo. The evaluation script below is committed; the audio it reads is not.
 Re-running prep into a populated `samples/` dir refreshes it in place. If a
 re-recording yields fewer clips than last time, the extra clips from the old run
 would linger and the training globs would still read them, so prep warns and
-names the count. Pass `--clean` to remove those leftovers in the same run. It is
-deliberately narrow: it only deletes leftover clips of a take this run
-re-recorded (a `<stem>_NNN.wav` clip whose stem the run wrote), so a source
-recording or a hand-curated clip for a different stem is named but left in place.
+names the count. Pass `--clean` to remove those leftovers in the same run.
+
+With a readable source-aware manifest, `--clean` only deletes clips prep recorded
+as its output. It removes the now-unused higher-numbered clips of a take this run
+re-recorded, and the clips a previous run recorded for a take that is no longer
+in the inputs. The `.prep-manifest.json` in the output dir maps each resolved
+source path to the clips prep wrote for it. A source recording, a hand-curated
+clip, and anything else absent from that map is named but left in place.
+Recorded output stems also remain reserved for their source while the next run
+assigns names. A new same-basename recording therefore cannot overwrite another
+source's manifested clips before `--clean` has a chance to apply its ownership
+rules.
+
+That record proves prep made a file, not that the file belongs to what you are
+refreshing. A `--label` that disagrees with the record already in the directory
+is refused outright, before anything is decoded: that is a mistyped `--output`,
+and waiting until `--clean` would be too late, since the run would already have
+overwritten that dataset's clips. Past that, a run must contain at least one
+exact source path from the record before it may delete from the dataset or write
+to it. A shared basename is not ownership. A directory with no record yet counts
+as yours. If a non-clean legacy refresh leaves same-stem files behind, prep does
+not write a manifest because it cannot prove whether those files are old prep
+output or audio curated by hand. The follow-up `--clean` stays on the legacy
+filename rule rather than recording that guess as provenance. A run that
+produces no clips does not claim an unrecorded directory. Prep also spares the
+prior clips of a source it tried and got nothing from (a silent or bad
+re-recording): those are the only copy, and it will not trade them for a run
+that produced nothing. A new source that produces no clips is not added to the
+ownership record, so it cannot authorize a later cleanup until a successful run
+actually contributes audio.
+
+When upgrading an existing pre-manifest `samples/` directory, bootstrap it with
+the complete source list for that label. A partial first run can record only the
+sources it receives; legacy clips under omitted stems then remain deliberately
+unowned and future `--clean` runs will only warn about them. If the complete
+source set is unavailable, move or clear the old output before starting a new
+partial dataset.
+
+To add a new recording to an existing dataset without `--clean`, include at
+least one source already recorded there in the same invocation. This gives prep
+explicit ownership proof while it adds the new source to the manifest. If you
+use `--clean`, include every source whose clips the dataset should keep: omitted
+sources are treated as intentionally dropped and their prep-owned clips are
+removed. Before decoding, prep warns with each omitted source path and the
+number of its clips scheduled for removal. A safe incremental workflow is to add
+recordings without `--clean`, then run a full-input `--clean` pass after
+reviewing the complete source list.
+
+Manifest updates use atomic replacement. Only an absent manifest falls back to
+the narrow same-stem cleanup rule, with a warning that source ownership
+protection is unavailable. That fallback has no provenance: `--clean` can remove
+any `<same-stem>_NNN.wav` file, including a hand-added file with that shape.
+Check the warning list and move any such file before cleaning. An unreadable,
+malformed, or non-v2 manifest fails closed before any decode or write because
+neither its dataset label nor its source ownership can be trusted. After
+reviewing the output directory, remove that manifest and rerun the full source
+set to bootstrap ownership.
 
 ```bash
-.venv/bin/python prep_wake_samples.py --input computah_normal.wav \
-    --output samples/positive --label positive --clean
+.venv/bin/python prep_wake_samples.py --input computah_normal.wav computah_styles.wav \
+    computah_distance.wav --output samples/positive --label positive --clean
 ```
 
 ## 2. Run the sweep