Skip to content

Add hidden-holdout carving and the leakage gate - #2

Merged
SheeshDarth merged 3 commits into
mainfrom
sp/hidden-holdout
Aug 7, 2026
Merged

Add hidden-holdout carving and the leakage gate#2
SheeshDarth merged 3 commits into
mainfrom
sp/hidden-holdout

Conversation

@SheeshDarth

Copy link
Copy Markdown
Owner

What (S1 — Data & Bench, Week 2)

The hidden holdout is what we measure degradation against — if it leaks into training, every downstream result is silently inflated and GATE 0 becomes meaningless. This adds the carve and the guard.

src/ingestion/holdout.py — assigns samples by a stable blake2b hash of sample_id rather than a seeded shuffle. A sample therefore always lands on the same side of the split even as the corpus grows or gets reordered; a reshuffle would quietly move previously held-out rows into training. (blake2b, not Python's built-in hash, which is randomized per process.)

src/ingestion/check_leakage.py — two independent checks:

  1. id overlap — a holdout sample_id appearing in a training file
  2. text overlap — identical case/whitespace-folded text under a different id

The second matters most: once src/generation starts copying and paraphrasing text, the realistic leak is the same passage reappearing with a fresh id, which an id-only check would miss entirely. Exits 1 on leakage so it can gate CI before any fine-tune reads data.

Verification

  • pytest -q21 passed (10 existing + 11 new)
  • CLI exercised both ways on generated data: clean → exit 0, poisoned → exit 1 with a precise report naming the leaked ids
LEAKAGE DETECTED
  2 sample_id(s) shared with training data
  2 holdout text(s) reappear in training data

Note for reviewers

Independent of #1 — this branches off main and touches no normalizer code, so the two can merge in either order (both edit src/ingestion/__init__.py, so the second to merge may need a trivial rebase).

Once merged, S3's GATE-0 spike can rely on the holdout being clean.

S1 (Data & Bench) Week-2 deliverable. The hidden holdout is what ORIGIN-T
measures degradation against, so leakage into training would silently inflate
every downstream result. This adds both the carve and the guard.

- src/ingestion/holdout.py: assigns samples by a stable blake2b hash of
  sample_id rather than a shuffle, so a sample always lands on the same side of
  the split even as the corpus grows or is reordered. A reshuffle would leak
  previously held-out rows into training. Uses blake2b, not the built-in hash,
  which is per-process randomized.
- src/ingestion/check_leakage.py: two independent checks -- id overlap, and
  identical case/whitespace-folded text under a *different* id, which is the
  realistic failure once generators start copying and rewriting text. Exits 1
  on leakage so it can gate CI before any fine-tune reads data.
- tests/test_holdout.py: 11 tests, including the stability-under-growth
  property and the new-id text-reappearance case.

Run: python -m src.ingestion.check_leakage --holdout H.jsonl --train T.jsonl
MONTH1_PLAN.md is internal coordination (per-person weekly targets, hour
budgets, mini-viva prep) like the team workplan and execution plan already
listed here. Ignoring it keeps a stray `git add -A` from publishing it.
carve_holdout() had no way to actually write a holdout to disk, so S1 could not
hand S3 a holdout file without ad-hoc code. Adds main() to the module that
already owns the logic rather than a separate CLI module.

Writes beside the input (*_kept.jsonl, *_hidden_holdout.jsonl) instead of over
it, so re-running is non-destructive. The test asserts the two outputs do not
leak into each other.
@SheeshDarth
SheeshDarth merged commit 7374655 into main Aug 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant