Add hidden-holdout carving and the leakage gate - #2
Merged
Conversation
S1 (Data & Bench) Week-2 deliverable. The hidden holdout is what ORIGIN-T measures degradation against, so leakage into training would silently inflate every downstream result. This adds both the carve and the guard. - src/ingestion/holdout.py: assigns samples by a stable blake2b hash of sample_id rather than a shuffle, so a sample always lands on the same side of the split even as the corpus grows or is reordered. A reshuffle would leak previously held-out rows into training. Uses blake2b, not the built-in hash, which is per-process randomized. - src/ingestion/check_leakage.py: two independent checks -- id overlap, and identical case/whitespace-folded text under a *different* id, which is the realistic failure once generators start copying and rewriting text. Exits 1 on leakage so it can gate CI before any fine-tune reads data. - tests/test_holdout.py: 11 tests, including the stability-under-growth property and the new-id text-reappearance case. Run: python -m src.ingestion.check_leakage --holdout H.jsonl --train T.jsonl
MONTH1_PLAN.md is internal coordination (per-person weekly targets, hour budgets, mini-viva prep) like the team workplan and execution plan already listed here. Ignoring it keeps a stray `git add -A` from publishing it.
carve_holdout() had no way to actually write a holdout to disk, so S1 could not hand S3 a holdout file without ad-hoc code. Adds main() to the module that already owns the logic rather than a separate CLI module. Writes beside the input (*_kept.jsonl, *_hidden_holdout.jsonl) instead of over it, so re-running is non-destructive. The test asserts the two outputs do not leak into each other.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (S1 — Data & Bench, Week 2)
The hidden holdout is what we measure degradation against — if it leaks into training, every downstream result is silently inflated and GATE 0 becomes meaningless. This adds the carve and the guard.
src/ingestion/holdout.py— assigns samples by a stableblake2bhash ofsample_idrather than a seeded shuffle. A sample therefore always lands on the same side of the split even as the corpus grows or gets reordered; a reshuffle would quietly move previously held-out rows into training. (blake2b, not Python's built-inhash, which is randomized per process.)src/ingestion/check_leakage.py— two independent checks:sample_idappearing in a training fileThe second matters most: once
src/generationstarts copying and paraphrasing text, the realistic leak is the same passage reappearing with a fresh id, which an id-only check would miss entirely. Exits 1 on leakage so it can gate CI before any fine-tune reads data.Verification
pytest -q→ 21 passed (10 existing + 11 new)exit 0, poisoned →exit 1with a precise report naming the leaked idsNote for reviewers
Independent of #1 — this branches off
mainand touches no normalizer code, so the two can merge in either order (both editsrc/ingestion/__init__.py, so the second to merge may need a trivial rebase).Once merged, S3's GATE-0 spike can rely on the holdout being clean.