Semantic-preserving Transformations for Robust Evaluation of Synthetic-text Screening
STRESS-Test is a research framework for evaluating whether AI-text detectors remain reliable after paraphrasing, translation, grammar correction, human editing, and character-level perturbations. Transformations are evaluated on both machine and human text at a fixed decision threshold, making evasion and false-accusation risk part of the same protocol.
Research status: active, pre-submission work. The current results are preliminary sensitivity analyses, not a finalized benchmark. Semantic-preservation thresholds still require the planned human-validation study, and the present per-domain calibration analysis estimates thresholds from the same clean domain slices used for evaluation. These limitations are stated here because they affect how the numbers should be interpreted.
Paper draft · Generated results · Reproducibility guide · Result provenance
- Do clean-text accuracy and robustness rank detectors differently?
- How do realistic transformation pipelines affect detection at a frozen operating threshold?
- Do benign edits to human writing increase false-accusation rates for particular registers or writer groups?
- Which conclusions change when results are stratified by text difficulty, domain, and detector mechanism?
The table below is generated from the current per-domain calibration sensitivity analysis at a target 1% false-positive rate. Robustness Score (RS) is the mean retained TPR across transformed conditions; worst-case is the minimum TPR among the named transformation pipelines.
| Detector | Clean TPR | RS | Worst-case pipeline TPR | FAR after human editing |
|---|---|---|---|---|
| Fast-DetectGPT | 97.7% | 0.727 | 46.7% | 3.4% |
| Binoculars-lite | 96.9% | 0.634 | 44.0% | 7.4% |
| DeBERTa-v3, HC3 fine-tune | 93.1% | 0.829 | 78.7% | 8.0% |
| GPT-2 perplexity | 93.1% | 0.642 | 62.0% | 6.4% |
| OpenAI RoBERTa detector | 85.9% | 0.554 | 53.3% | 6.7% |
| TF-IDF + logistic regression | 79.4% | 0.961 | 63.3% | 1.1% |
| Stylometric GBM | 55.7% | 0.915 | 50.7% | 1.2% |
What the current analysis suggests:
- Strong clean performance does not guarantee transformation robustness. The two strongest clean detectors fall to 44–47% TPR under the depth-2 laundering pipeline.
- The supervised DeBERTa model retains the highest worst-case pipeline TPR, while TF-IDF has the highest relative robustness at a lower clean TPR.
- False-accusation rates vary substantially by detector, transformation, and writing register. The slice results do not support a simple “native versus non-native” explanation.
- Quality-adjusted evasion results are provisional until the semantic-similarity cutoff has been checked against human judgments.
Full generated tables are available in PER_DOMAIN.md, REPORT.md, FAIRNESS.md, HARDNESS.md, and STATS.md.
- Symmetric transformations: apply each transformation to human and machine text.
- Clean-only training: trainable detectors do not see transformed text during training.
- Frozen thresholds: select a target operating point once and do not retune it for each transformation.
- Document-level grouping: keep a source document and all of its variants in the same split.
- Reliability metrics: report RS, worst-case performance, false-accusation rate, calibration drift, quality-adjusted evasion, and hardness collapse.
- Cluster-aware inference: resample source documents rather than individual rows and correct paired comparisons for multiple testing.
The current draft contains both pooled-threshold and per-domain sensitivity analyses. A separate held-out calibration set is required before the reported operating-point estimates should be treated as final.
The lightweight installation runs the metric, statistics, split, detector, and transformation tests without downloading model weights:
git clone https://github.com/razon1494/stress-test.git
cd stress-test
python -m pip install -e ".[dev]"
python -m pytest -qFor the complete pipeline:
python -m pip install -e ".[models,data,dev]"
python scripts/01_build_dataset.py --help
python scripts/02_apply_transforms.py --help
python scripts/03_run_detectors.py --help
python scripts/04_make_report.py --helpSome fairness corpora require separate downloads and cannot be redistributed in this repository. See the reproducibility guide before attempting a full run.
| Path | Purpose |
|---|---|
src/stress_test/ |
Transformations, detectors, metrics, statistics, data utilities, and report generation |
scripts/ |
Numbered experiment and analysis stages |
tests/ |
Unit tests for core research logic |
results/ |
Committed generated summaries and structured outputs |
paper/ |
LaTeX manuscript draft and generated figure |
docs/ |
Reproduction instructions, annotation protocol, and related-work notes |
- The machine-text core currently uses one generator family and one broad source dataset (HC3).
- Threshold calibration and final evaluation are not yet separated in the committed result set.
- Semantic preservation is currently embedding-scored; the planned three-annotator validation has not been completed.
- The DeBERTa fine-tune uses one training seed, and Binoculars-lite is not the original Falcon-7B model pair.
- Several subgroup estimates have wide uncertainty because the cells are small.
- Raw licensed corpora, model weights, and score caches are intentionally not committed.
Until a preprint is released, cite the software repository:
@software{rahman2026stresstest,
author = {Rahman, Mohammad Arifur},
title = {STRESS-Test: Semantic-preserving Transformations for Robust
Evaluation of Synthetic-text Screening},
year = {2026},
url = {https://github.com/razon1494/stress-test}
}Mohammad Arifur Rahman · rahman.arif.cse@gmail.com · Portfolio
The framework code is released under the MIT License. External datasets and pretrained models remain subject to their original licenses and terms of use.