Skip to content

feat(replay): add deterministic counterfactual core - #351

Open
PhilipJohnBasile wants to merge 2 commits into
youssofal:mainfrom
PhilipJohnBasile:feat/deterministic-replay
Open

feat(replay): add deterministic counterfactual core#351
PhilipJohnBasile wants to merge 2 commits into
youssofal:mainfrom
PhilipJohnBasile:feat/deterministic-replay

Conversation

@PhilipJohnBasile

Copy link
Copy Markdown
Contributor

Splits the deterministic counterfactual replay core out of #336.

This PR contains only the offline replay system:

  • ordered, bounded candidate execution
  • private execution deduplication and redacted public fingerprints
  • mutation-isolated candidate and evaluator inputs
  • candidate and evaluator timeouts
  • deterministic regression policy decisions
  • content-redacted report serialization
  • a standalone no-model measurement

It does not change live serving, request capture, replay orchestration, model selection, promotion state, dashboard code, or workflows.

Measurement on commit f96817c, macOS 27.0 arm64, Python 3.14.6, 10 repeats with 128 cases and 4 evaluators:

Metric Result
Mean suite time 26.916 ms
Suite p50 19.964 ms
Suite p95 59.264 ms
Mean throughput 4,755.6 cases/s
Candidate executions 64
Deduplicated cases 64
Unique report digests 1
5 ms timeout caller wall 7.112 ms

Reproduce:

python scripts/bench_deterministic_replay.py \
  --cases 128 \
  --evaluators 4 \
  --repeats 10 \
  --output /tmp/deterministic-replay.json

Validation:

python -m pytest tests/test_deterministic_replay.py tests/test_trace_parity.py
17 passed in 0.19s

python -m compileall -q mtplx/deterministic_replay.py tests/test_deterministic_replay.py scripts/bench_deterministic_replay.py
PASS

ruff check mtplx/deterministic_replay.py tests/test_deterministic_replay.py scripts/bench_deterministic_replay.py
PASS

git diff --check
PASS

This measurement covers offline control-plane cost and determinism. It makes no live throughput or quality claim.

@PhilipJohnBasile
PhilipJohnBasile force-pushed the feat/deterministic-replay branch from f96817c to fa5ce1c Compare September 1, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant