You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Expected Enterprise/observability value: exceptional Proxy-level fit: exceptional Implementation leverage vs complexity: very high because AIProxer already has stable request/session/trace identities and planned metadata storage
Summary
Add an Enterprise quality-evidence layer that lets humans, coding harnesses, CI systems and future automated evaluators attach structured outcome/quality signals to an already-completed AIProxer request, turn, tool call or session.
Helicone treats feedback and named scores as first-class observability data. It supports request/session feedback and a score store where named boolean/integer scores are attached to request IDs and then aggregated/queryable. Its documentation explicitly positions feedback as the production signal used to detect regressions and build better evaluation datasets.
This issue answers: was the result useful/correct/successful, according to whom or what?
Why this matters especially for coding-agent observability
For agentic coding, HTTP 200 is a very weak definition of success.
Useful production signals include:
user accepted/rejected result
patch applied successfully
unit tests passed after response
build succeeded
PR review comment accepted/resolved
agent had to retry/regenerate
human reviewer rated result 1..5
session completed intended task
harness aborted after bad output
Once linked to provider/model/route/cost/latency evidence, Enterprise can answer much more valuable questions than "which model had the lowest TTFT?":
Which route has best task-success / dollar?
Did quality drop after model/provider change?
Which teams have high retry/rejection rates?
Does low TTFT actually correlate with successful coding sessions?
Which production examples should seed a regression dataset?
Architecture principle
Quality evidence is an append-only/provenanced observation about a completed object; it is not authentication, billing authority, or an untrusted way to alter routing policy.
1. Canonical quality observation
Define a stable record conceptually containing:
observation_id
target_type: request | session | tool_call | async_job | batch_item
stable target id
metric/score name
value type
value
source/provenance
actor/evaluator identity where applicable
created_at
evaluator/version/source revision where applicable
optional bounded reason/comment metadata
success rate by route/model
tests_passed by model version
quality vs latency
quality vs cost
feedback by project/team
score distribution before/after rollout
9. Storage model
Do not store one wide row with a new nullable column for every user-created score.
Prefer normalized append/event storage plus indexed projections appropriate to the analytics store.
Requirements:
bounded score count per target/source window;
indexed target ID / metric ID / timestamp / trusted scope;
batch ingestion for CI/evaluator results;
pagination for history;
retention policy independent from raw prompt content where appropriate.
A quality score can remain useful after raw content expires.
10. Privacy and sensitive annotations
Default scores should contain no prompt/completion content.
Do not accept arbitrary annotation keys into Prometheus labels.
11. Authorization
#506 controls who may read/write which observations.
Potential policies:
user may score own request/session
CI service may score requests for bound project
team lead may annotate team data
operator/evaluator service may write configured metric classes
Possession of a request UUID alone is never sufficient authorization.
12. Hot-path performance
Quality observations normally arrive after the model request.
Hard constraints:
no DB read/write added to inference routing merely because quality tracking is enabled;
no synchronous score API dependency before client response;
batch evaluator writes use bounded async persistence;
inference remains correct if score storage is temporarily unavailable.
If an integration supplies a cheap feedback marker inline, it can be emitted as bounded post-turn evidence, not an extra critical-path transaction.
13. Explainability linkage
#456 should optionally show a compact quality section:
Observability becomes much more valuable when it measures outcomes, not only latency and cost. For coding agents, test success, patch acceptance, retries and human approval are often stronger measures of model quality than HTTP status. This feature creates the quality-data substrate required for production evaluators, regression detection and model/route ROI analysis across the Enterprise product.
Assessment: 9.6/10
Expected Enterprise/observability value: exceptional
Proxy-level fit: exceptional
Implementation leverage vs complexity: very high because AIProxer already has stable request/session/trace identities and planned metadata storage
Summary
Add an Enterprise quality-evidence layer that lets humans, coding harnesses, CI systems and future automated evaluators attach structured outcome/quality signals to an already-completed AIProxer request, turn, tool call or session.
Helicone treats feedback and named scores as first-class observability data. It supports request/session feedback and a score store where named boolean/integer scores are attached to request IDs and then aggregated/queryable. Its documentation explicitly positions feedback as the production signal used to detect regressions and build better evaluation datasets.
References:
This is not another request log or trace store.
Why this matters especially for coding-agent observability
For agentic coding, HTTP 200 is a very weak definition of success.
Useful production signals include:
Once linked to provider/model/route/cost/latency evidence, Enterprise can answer much more valuable questions than "which model had the lowest TTFT?":
Architecture principle
1. Canonical quality observation
Define a stable record conceptually containing:
Useful value types:
Free-form text annotations may be supported separately with strict byte limits and privacy controls; do not turn score columns into arbitrary blobs.
2. Named score registry
Helicone allows arbitrary named score keys. AIProxer should preserve flexibility while making Enterprise analytics manageable.
Introduce a score/metric definition registry with fields such as:
Examples:
Do not create SQL columns dynamically for every metric.
3. Explicit feedback API
Provide protected APIs conceptually like:
The exact API should support idempotency and clear source ownership.
A coding harness/wrapper can submit a signal after it learns the actual outcome, e.g. tests completed 20 seconds after model output.
4. Implicit feedback
Helicone highlights the value of implicit feedback in addition to thumbs-up/down.
AIProxer should support ingestion from integrations such as:
But implicit signals need named semantics and provenance; do not pretend every follow-up request is a negative rating.
The harness/integration producing the signal owns its interpretation.
5. Multiple observers and conflicts
The same request may receive:
Do not overwrite all of these into one
scorefield.Each observation retains source/metric identity.
For repeated writes of the same logical source + metric, define explicit semantics:
Never use timestamp last-write-wins across unrelated observers.
6. Provenance classes
At minimum distinguish:
For automated evaluators, store evaluator definition + immutable revision used to produce the score.
A score of 0.8 from evaluator-v3 is not directly interchangeable with evaluator-v7 unless the metric contract says so.
7. Session-level quality
Request-level quality is necessary but coding tasks frequently span many turns.
Allow session/workflow observations such as:
#511 supplies native coding-agent session correlation where available; secure-session authority remains separate.
A caller-supplied session hint does not grant permission to annotate another principal's session.
8. Analytics projection
Scores should become dimensions/measures available to:
Examples:
9. Storage model
Do not store one wide row with a new nullable column for every user-created score.
Prefer normalized append/event storage plus indexed projections appropriate to the analytics store.
Requirements:
A quality score can remain useful after raw content expires.
10. Privacy and sensitive annotations
Default scores should contain no prompt/completion content.
Text comments/annotations, if enabled:
Do not accept arbitrary annotation keys into Prometheus labels.
11. Authorization
#506 controls who may read/write which observations.
Potential policies:
Possession of a request UUID alone is never sufficient authorization.
12. Hot-path performance
Quality observations normally arrive after the model request.
Hard constraints:
If an integration supplies a cheap feedback marker inline, it can be emitted as bounded post-turn evidence, not an extra critical-path transaction.
13. Explainability linkage
#456 should optionally show a compact quality section:
The request explanation itself remains routing/lifecycle evidence authority.
14. Alerting
#516 can alert on aggregate quality degradation, e.g.:
Alerts consume aggregates; they do not redefine score semantics.
15. Open-Core boundary
This is a strong Enterprise observability layer.
A reasonable boundary:
Suggested V1
Acceptance criteria
Non-goals
Why 9.6/10
Observability becomes much more valuable when it measures outcomes, not only latency and cost. For coding agents, test success, patch acceptance, retries and human approval are often stronger measures of model quality than HTTP status. This feature creates the quality-data substrate required for production evaluators, regression detection and model/route ROI analysis across the Enterprise product.