You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Expected Enterprise/evaluation value: exceptional Proxy-level fit: very high Implementation leverage vs complexity: high once #509/#519/#521 exist; replay safety and data governance are the main design challenges
Summary
Add an Enterprise production dataset + regression replay capability: curate retained production requests/sessions into immutable dataset revisions, then replay safe model-bound inputs against candidate AIProxer routes/models/configurations and compare quality, latency and cost before rollout.
Helicone Datasets turn selected production requests into curated evaluation/training sets, using filters and scores to identify useful examples. Its former Experiments workflow also demonstrated the valuable pattern of testing prompt/model variants against production or curated rows and comparing evaluator scores side-by-side. Helicone has since deprecated that specific Experiments UI, so AIProxer should borrow the workflow, not its old spreadsheet implementation.
production AIProxer traffic
↓
#519 human/CI/evaluator quality evidence
↓
filter interesting cases
↓
Dataset: coding-regressions@17
↓
replay against:
current route
candidate model
new provider
new reasoning policy
new compatibility/profile generation
↓
#521 evaluators + deterministic checks
↓
compare quality / cost / latency
↓
approve rollout or reject regression
This makes the proxy's heterogeneous model/router position especially valuable: the same production corpus can test multiple candidate backends through one canonical execution layer.
Critical safety principle
Replay is model inference only. It must never automatically re-execute historical external side effects.
Coding-agent traces contain shell commands, writes, MCP calls, GitHub mutations and other actions that are unsafe to replay.
V1 replays only the model-bound canonical input and observes the candidate model response/tool-call proposal. It does not execute generated tool calls.
1. Dataset definition vs dataset revision
A dataset is a named logical collection with immutable revisions.
Conceptually:
dataset id/name/description
owner/org/project scope
revision
created_at/by
selection provenance
member references
content snapshot/reference policy
schema version
Adding/removing examples creates a new revision or explicit draft->publish lifecycle.
An evaluation run always records the exact dataset revision.
2. Dataset sources
Allow examples to come from:
manually selected production requests
#519 quality-score filters
#520 observational-property filters
failed/retried requests
specific project/route/model/config generation
session/agent examples
API/CSV import where authorized
A useful curation query might be:
workflow=pr-review
AND tests_passed=true
AND human_quality>=4
or intentionally gather failures:
route=coding-fast
AND task_success=false
3. Reference instead of copy where possible
Large coding prompts may be many MiB.
Do not blindly duplicate every prompt/output into every dataset row.
Prefer dataset membership references to retained immutable/content-addressed #509 objects where retention guarantees permit it.
If dataset longevity must exceed ordinary request-content retention, explicitly snapshot the necessary redacted content into a dataset-owned retention class with its own consent/governance.
Never silently extend sensitive source-code retention merely because a request was added to a dataset.
4. Content privacy and residency
Dataset creation is a privileged data-retention action.
An evaluator requiring raw content cannot force a redacted dataset to reveal originals.
5. Canonical replay input
Store/materialize a portable canonical model-bound envelope, not only raw frontend JSON.
Useful replay facts may include:
messages/items/instructions
model-independent tool definitions
structured output contract
reasoning intent where portable
relevant content/media refs subject to policy
original public route/config metadata as context
The replay runner can then choose a candidate route and pass through ordinary #495 compatibility/capability adaptation.
Do not preserve provider-specific transient auth headers or credentials.
6. Historical semantics vs current semantics
A replay run needs explicit modes:
Current-stack evaluation
Use the current AIProxer runtime/policy/capability adapters to ask:
How would this production input behave if sent today through candidate route X?
This is the primary V1.
Pinned historical-stack reproduction
Potentially useful later, but requires retaining historical config/plugin generations and security implications.
Do not retain arbitrary old executable runtime graphs indefinitely just to reproduce history.
7. Candidate matrix
A run may compare a bounded set of candidate configurations:
route/model
reasoning effort/policy
selected safe generation parameters
compatibility/profile revision
prompt/skill revision only where AIProxer owns that artifact
Avoid arbitrary provider Cartesian explosion.
Require explicit candidates and hard caps:
max rows/run
max candidates/run
max total inference calls
max estimated/authorized spend
8. Run execution through ordinary AIProxer core
Every replay call uses the same canonical executor as normal inference with origin:
Some coding behaviors only appear over several turns.
Support a dataset row that references a bounded session/transcript projection when #509 retains the required content.
Replays still operate on model-bound transcript inputs and may advance through a fixture/simulation, not real tool execution.
V1 can prioritize single-turn/portable continuation inputs before full simulated sessions.
11. Evaluator integration
A replay run attaches one or more #521 evaluator revisions and/or #519 known ground-truth metrics.
Examples:
correct expected tool
structured output valid
code-review quality judge
known test result fixture
similarity to accepted production answer
Record evaluator failures separately from candidate quality failures.
12. Comparison metrics
For each candidate and dataset revision expose:
pass rate / score distribution
paired score delta vs baseline
provider/model errors
latency/TTFT distribution
input/output usage
provider cost
AIProxer overhead
capability incompatibility/skips
For paired datasets, show row-level wins/losses and aggregate deltas.
Do not reduce every comparison to one synthetic scalar automatically.
13. Statistical caution
Evaluation dashboards should display:
N
missing/skipped rows
score distribution
paired deltas
confidence intervals where meaningful
A 3-row dataset should not produce a green "+12% quality" badge that implies statistical certainty.
LLM-as-judge scores are noisy and must be identified as such.
14. Durable run lifecycle
Regression runs may contain thousands of model calls.
Use a durable bounded run/job authority rather than keeping one browser connection alive.
#500 async-job primitives may be reusable for individual/background executions; the SDD should decide whether an evaluation-run coordinator composes #500 rather than building another generic job queue.
Potential provider-native batch acceleration through #512 is optional and only valid when semantics/cost/ordering match the run requirements.
15. Concurrency/QoS
Eval replays should default to background/best-effort capacity via #493.
Hard limits:
concurrent runs/org
concurrent model calls/run
max queued rows
max run wall time
max cost
Production interactive coding traffic should not be starved by a 50k-row regression run.
16. Dataset export
Allow protected exports such as:
JSONL canonical evaluation rows
CSV metadata/score view
provider fine-tuning format only through explicit adapters
Export itself is governed by #506/#508 and content-residency policy.
Do not claim every canonical coding transcript can be losslessly transformed into every provider's fine-tuning schema.
17. Model/provider rollout gate
A future enterprise workflow can make an evaluation run an advisory/pre-deployment gate:
new model metadata/config proposed
-> run selected regression suites
-> display pass/fail policy
-> human approves publish
Do not automatically modify production routes in V1; keep rollout decision explicit and auditable.
18. Open-Core boundary
This is an excellent closed Enterprise EvalOps feature:
OSS core supplies the canonical inference executor and compatibility machinery;
Enterprise owns production dataset curation, long-lived quality corpus, replay orchestration, comparisons and organization-level UI;
production inference remains usable without any dataset subsystem.
automatically pushing a winning candidate to production in V1;
copying Helicone's deprecated spreadsheet UI.
Why 9.5/10
Continuous online scores detect that quality may be changing; regression datasets tell engineers why and whether a candidate change is safe before rollout. AIProxer's canonical cross-provider request model makes it unusually well suited to replay the same real coding inputs across heterogeneous models/routes, creating a strong Enterprise evaluation moat around the observability data it already collects.
Assessment: 9.5/10
Expected Enterprise/evaluation value: exceptional
Proxy-level fit: very high
Implementation leverage vs complexity: high once #509/#519/#521 exist; replay safety and data governance are the main design challenges
Summary
Add an Enterprise production dataset + regression replay capability: curate retained production requests/sessions into immutable dataset revisions, then replay safe model-bound inputs against candidate AIProxer routes/models/configurations and compare quality, latency and cost before rollout.
Helicone Datasets turn selected production requests into curated evaluation/training sets, using filters and scores to identify useful examples. Its former Experiments workflow also demonstrated the valuable pattern of testing prompt/model variants against production or curated rows and comparing evaluator scores side-by-side. Helicone has since deprecated that specific Experiments UI, so AIProxer should borrow the workflow, not its old spreadsheet implementation.
References:
Product goal
Create a closed feedback loop:
This makes the proxy's heterogeneous model/router position especially valuable: the same production corpus can test multiple candidate backends through one canonical execution layer.
Critical safety principle
Coding-agent traces contain shell commands, writes, MCP calls, GitHub mutations and other actions that are unsafe to replay.
V1 replays only the model-bound canonical input and observes the candidate model response/tool-call proposal. It does not execute generated tool calls.
1. Dataset definition vs dataset revision
A dataset is a named logical collection with immutable revisions.
Conceptually:
Adding/removing examples creates a new revision or explicit draft->publish lifecycle.
An evaluation run always records the exact dataset revision.
2. Dataset sources
Allow examples to come from:
A useful curation query might be:
or intentionally gather failures:
3. Reference instead of copy where possible
Large coding prompts may be many MiB.
Do not blindly duplicate every prompt/output into every dataset row.
Prefer dataset membership references to retained immutable/content-addressed #509 objects where retention guarantees permit it.
If dataset longevity must exceed ordinary request-content retention, explicitly snapshot the necessary redacted content into a dataset-owned retention class with its own consent/governance.
Never silently extend sensitive source-code retention merely because a request was added to a dataset.
4. Content privacy and residency
Dataset creation is a privileged data-retention action.
Requirements:
An evaluator requiring raw content cannot force a redacted dataset to reveal originals.
5. Canonical replay input
Store/materialize a portable canonical model-bound envelope, not only raw frontend JSON.
Useful replay facts may include:
The replay runner can then choose a candidate route and pass through ordinary #495 compatibility/capability adaptation.
Do not preserve provider-specific transient auth headers or credentials.
6. Historical semantics vs current semantics
A replay run needs explicit modes:
Current-stack evaluation
Use the current AIProxer runtime/policy/capability adapters to ask:
This is the primary V1.
Pinned historical-stack reproduction
Potentially useful later, but requires retaining historical config/plugin generations and security implications.
Do not retain arbitrary old executable runtime graphs indefinitely just to reproduce history.
7. Candidate matrix
A run may compare a bounded set of candidate configurations:
Avoid arbitrary provider Cartesian explosion.
Require explicit candidates and hard caps:
8. Run execution through ordinary AIProxer core
Every replay call uses the same canonical executor as normal inference with origin:
It gets:
But replay policy may deliberately disable normal production sidecars/evaluators to prevent recursion unless explicitly part of what is under test.
No hidden direct provider client.
9. Tool semantics
If historical input declares tools, replay may allow the candidate model to propose tool calls so tool-use quality can be evaluated.
Do not dispatch those calls to:
Instead:
10. Multi-turn/session datasets
Some coding behaviors only appear over several turns.
Support a dataset row that references a bounded session/transcript projection when #509 retains the required content.
Replays still operate on model-bound transcript inputs and may advance through a fixture/simulation, not real tool execution.
V1 can prioritize single-turn/portable continuation inputs before full simulated sessions.
11. Evaluator integration
A replay run attaches one or more #521 evaluator revisions and/or #519 known ground-truth metrics.
Examples:
Record evaluator failures separately from candidate quality failures.
12. Comparison metrics
For each candidate and dataset revision expose:
For paired datasets, show row-level wins/losses and aggregate deltas.
Do not reduce every comparison to one synthetic scalar automatically.
13. Statistical caution
Evaluation dashboards should display:
A 3-row dataset should not produce a green "+12% quality" badge that implies statistical certainty.
LLM-as-judge scores are noisy and must be identified as such.
14. Durable run lifecycle
Regression runs may contain thousands of model calls.
Use a durable bounded run/job authority rather than keeping one browser connection alive.
#500 async-job primitives may be reusable for individual/background executions; the SDD should decide whether an evaluation-run coordinator composes #500 rather than building another generic job queue.
Potential provider-native batch acceleration through #512 is optional and only valid when semantics/cost/ordering match the run requirements.
15. Concurrency/QoS
Eval replays should default to background/best-effort capacity via #493.
Hard limits:
Production interactive coding traffic should not be starved by a 50k-row regression run.
16. Dataset export
Allow protected exports such as:
Export itself is governed by #506/#508 and content-residency policy.
Do not claim every canonical coding transcript can be losslessly transformed into every provider's fine-tuning schema.
17. Model/provider rollout gate
A future enterprise workflow can make an evaluation run an advisory/pre-deployment gate:
Do not automatically modify production routes in V1; keep rollout decision explicit and auditable.
18. Open-Core boundary
This is an excellent closed Enterprise EvalOps feature:
Suggested V1
Acceptance criteria
Non-goals
Why 9.5/10
Continuous online scores detect that quality may be changing; regression datasets tell engineers why and whether a candidate change is safe before rollout. AIProxer's canonical cross-provider request model makes it unusually well suited to replay the same real coding inputs across heterogeneous models/routes, creating a strong Enterprise evaluation moat around the observability data it already collects.