Add contamination mixing at controlled ratios - #3
Merged
Conversation
First piece of src/generation. Generating contaminated text is type-specific, but blending it into a dataset at 0/25/50/75/100% is the same operation for every type, so the mixer is model-free: deterministic, and testable without a GPU. Each mixed dataset is one cell of the fine-tuning grid. total defaults to len(human) so a ratio sweep holds dataset size fixed -- otherwise a ratio change would confound contamination with training-set size, and the collapse curves would not be attributable to contamination. Every output sample carries contamination_ratio so the mixture is recoverable from the data alone.
The leakage that matters is not the exact copy -- an eval item gets recased, re-wrapped or reformatted somewhere in a pipeline and lands in training as a near-duplicate that exact-match and n-gram checks miss while the model still memorises the answer. The generator has to produce that case for the detector to be tested against it. This is the one contamination type needing no model: perturbations are surface-level and meaning-preserving, so it runs on CPU and stays deterministic. Real paraphrase needs a generation model and stays with the synthetic generators. benchmark_near_score is measured per sample with difflib rather than assumed. The score folds case and whitespace deliberately: an ALL-CAPS copy is a complete leak and must score 1.0 -- comparing raw characters scored it near zero, which would have handed the risk model inverted ground truth. A test caught this. Injection reuses mix() rather than reimplementing ratio logic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (S1 — Data & Bench)
First piece of
src/generation(PRD must-have #2). Generating contaminated text is type-specific, but blending it into a dataset at 0/25/50/75/100% is identical for every type — so the mixer is model-free: deterministic and testable without a GPU. Each mixed dataset is one cell of the fine-tuning grid.The design decision that matters
totaldefaults tolen(human), so a ratio sweep holds dataset size fixed. Without that, raising the contamination ratio would also change training-set size, and the collapse curves wouldn't be attributable to contamination — the whole experiment would be confounded.Every output sample carries
contamination_ratio, so a mixture is recoverable from the data alone rather than depending on a filename convention.API
Verification
pytest -q→ 19 passed (10 existing + 9 new): exact counts per ratio, size held fixed across the sweep, determinism per seed, seed actually changes the draw, inputs not mutated, and clear errors when the pools are too small.Note for reviewers
Branches off
mainand imports..ingestion.schemadirectly, so it does not touchsrc/ingestion/__init__.py— no conflict with #1 or #2, and the three can merge in any order.Skipped deliberately: the model-backed generators (synthetic sampling, recursive Gen-1→3, paraphrase) and benchmark-near injection. Those are Sprint 2 and each needs a model; this is the shared plumbing they all feed into.