You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Expected Enterprise/agent-observability value: exceptional Proxy-level fit: very high as an observability ingestion surface Implementation leverage vs complexity: high because #502/#511 supply trace/session correlation, while the missing problem is visibility outside the proxy boundary
Summary
Add an Enterprise external agent-span ingestion API/SDK contract so coding harnesses, wrappers and internal applications can report tool execution, retrieval/vector DB work, shell/filesystem steps and other non-LLM workflow stages into the same AIProxer request/session trace tree.
Helicone's Sessions can group LLM calls, vector database queries, tool calls and arbitrary logged requests into one hierarchical workflow. Its manual Logger SDK lets application code wrap external tool/API work and submit the result so it appears alongside LLM telemetry.
But a stock coding harness may execute this locally:
model -> tool call
|
+-> shell command 4.2s
+-> grep filesystem 0.3s
+-> GitHub API 1.1s
+-> test suite 42.0s
+-> local MCP 0.8s
then next model request
AIProxer can observe the model's tool-call envelope and later tool result, but often cannot know the actual execution timing, intermediate retries, external dependency, or structured outcome.
The Enterprise trace should be capable of showing the whole agent task rather than only its LLM portions.
Product goal
A complete coding-agent session might render conceptually as:
Session: fix failing auth tests
turn 1
AIProxer inference 3.2s
route resolution 0.4ms
provider claude 3.1s
external shell: rg 0.12s
external file read 0.02s
external shell: go test 17.4s FAILED
turn 2
AIProxer inference 4.6s
external edit 0.03s
external shell: go test 18.1s PASSED
session outcome: task_success=true (#519)
That is substantially more useful to a team lead/SRE than a list of provider calls alone.
Core trust principle
Externally submitted spans are observational evidence from an authenticated producer; they can never masquerade as AIProxer-owned routing, billing, authorization or security events.
Every span has an immutable origin/provenance such as:
harder to enforce AIProxer agent-specific semantic contracts.
B. Typed AIProxer AgentSpan API
Pros:
strict size/kind/privacy validation;
better session/request association.
A good design may accept both, translating OTLP through a bounded allowlisted projection rather than storing arbitrary OTLP structures verbatim.
5. Producer authentication and scope
External span producers require a scoped Enterprise credential/policy.
Examples:
coding wrapper may write spans only for its authenticated principal/session
CI service may write spans for project Apollo
central harness service may write spans for its organization
#506 controls management/read access; #480/#507 or a dedicated telemetry-write credential can authenticate ingestion.
Possession of a trace/session ID alone is never write authorization.
6. Anti-spoofing / namespaces
Reserve namespaces for AIProxer-owned facts:
aiproxer.*
proxy.*
billing.*
auth.*
security.*
External spans cannot set internal routing outcome, billed cost, guardrail decision, provider usage authority or principal identity.
If an external producer reports its own estimated cost, store it as producer evidence such as external.estimated_cost, not as AIProxer billing truth.
7. Session workflow hierarchy
Helicone's path-based sessions are useful inspiration, but AIProxer already has proper span/trace identities.
Allow a human-readable workflow path/stage as an optional bounded observational property:
For RAG/agent systems support safe metadata such as:
retrieval backend class
query duration
documents requested/returned
result bytes/count
cache hit
safe collection/index alias
Do not log embedding vectors by default.
This mirrors Helicone's ability to include vector DB requests in sessions without requiring AIProxer to become the vector database gateway.
11. Clock and timestamp handling
External producers have different clocks.
Requirements:
receive timestamp + observed/ingested timestamp;
allowed skew bound;
preserve negative/overlapping timing without silently falsifying chronology;
diagnostics flag excessive skew;
duration can be producer-reported when trustworthy enough for observation.
Do not use external timestamps for security ordering or billing authority.
12. Bounded ingestion
Enterprise telemetry endpoints must resist accidental/hostile floods.
Hard bounds:
batch count
span bytes
attributes/span
attribute name/value bytes
content bytes
write rate per principal/org
concurrent ingest requests
late-arrival age
retention
Use batch writes and bounded async persistence.
No goroutine or DB transaction per tiny external span under high-volume traffic.
13. Orphan/late spans
External spans may arrive before/after corresponding request metadata.
Store correlation references independently and reconcile/query by stable IDs later.
Do not reject useful late tool/test spans merely because the model request completed several minutes ago, but cap maximum accepted age according to policy.
Unknown parent should remain an orphan/link rather than fabricating a parent.
agent wall time = LLM vs tools vs tests
p95 test duration after specific model
shell failure rate by workflow
retrieval latency vs task success
average tool calls per completed coding task
15. Sampling
For huge organizations, support telemetry sampling independent from inference.
Potential rules:
retain all failed tool/test spans
sample successful high-volume filesystem reads
retain full spans for sessions selected by #521/#519 quality investigation
Sampling occurs at ingestion/producer configuration, with explicit dropped-count evidence where possible.
An authenticated coding-harness integration can submit an external shell/test/tool span and see it correlated with the same AIProxer session/request trace.
External spans are visibly marked external and cannot spoof proxy routing, billing, auth or security evidence.
Metadata-only ingestion works without retaining command/tool contents.
one giant mandatory trace covering an unlimited months-long session;
running arbitrary customer instrumentation code inside the proxy.
Why 9.6/10
For coding agents, a model request is often only a fraction of task wall time. A gateway-only trace can tell you the model took 3 seconds but miss the 45-second test run or failing shell command that determined the actual experience. Helicone's session + manual logger design demonstrates the value of correlating those external steps. AIProxer can turn that into a particularly strong Enterprise feature by combining external agent spans with its unusually rich routing, policy, cost and session evidence.
Assessment: 9.6/10
Expected Enterprise/agent-observability value: exceptional
Proxy-level fit: very high as an observability ingestion surface
Implementation leverage vs complexity: high because #502/#511 supply trace/session correlation, while the missing problem is visibility outside the proxy boundary
Summary
Add an Enterprise external agent-span ingestion API/SDK contract so coding harnesses, wrappers and internal applications can report tool execution, retrieval/vector DB work, shell/filesystem steps and other non-LLM workflow stages into the same AIProxer request/session trace tree.
Helicone's Sessions can group LLM calls, vector database queries, tool calls and arbitrary logged requests into one hierarchical workflow. Its manual Logger SDK lets application code wrap external tool/API work and submit the result so it appears alongside LLM telemetry.
References:
Why #502 alone cannot solve this
#502 instruments work AIProxer itself owns or sees:
But a stock coding harness may execute this locally:
AIProxer can observe the model's tool-call envelope and later tool result, but often cannot know the actual execution timing, intermediate retries, external dependency, or structured outcome.
The Enterprise trace should be capable of showing the whole agent task rather than only its LLM portions.
Product goal
A complete coding-agent session might render conceptually as:
That is substantially more useful to a team lead/SRE than a list of provider calls alone.
Core trust principle
Every span has an immutable origin/provenance such as:
External callers cannot submit
proxy_internalorigin.1. Canonical AgentSpan envelope
Define a small versioned span/event contract conceptually containing:
Initial span kinds could include:
Do not attempt to encode every vendor/tool in the kind enum; details belong in bounded attributes.
2. Prefer standard trace correlation
Where integrations already use OpenTelemetry/W3C Trace Context, accept/link standard:
#502 remains the internal tracing authority.
Also allow correlation through #511's normalized coding-harness session ID plus a request/turn ID when a harness cannot propagate W3C context.
Do not invent a second unrelated session identity system.
3. Dedicated ingestion API
Provide a protected endpoint/stream conceptually like:
or an OTLP-compatible receiver/profile where that cleanly satisfies the contract.
A thin helper SDK for wrappers/harness plugins may provide:
But the wire contract, not a specific SDK language, is authoritative.
4. OTLP compatibility vs AIProxer semantic envelope
Evaluate two ingestion modes during SDD:
A. OTLP receiver
Pros:
Cons:
B. Typed AIProxer AgentSpan API
Pros:
A good design may accept both, translating OTLP through a bounded allowlisted projection rather than storing arbitrary OTLP structures verbatim.
5. Producer authentication and scope
External span producers require a scoped Enterprise credential/policy.
Examples:
#506 controls management/read access; #480/#507 or a dedicated telemetry-write credential can authenticate ingestion.
Possession of a trace/session ID alone is never write authorization.
6. Anti-spoofing / namespaces
Reserve namespaces for AIProxer-owned facts:
External spans cannot set internal routing outcome, billed cost, guardrail decision, provider usage authority or principal identity.
If an external producer reports its own estimated
cost, store it as producer evidence such asexternal.estimated_cost, not as AIProxer billing truth.7. Session workflow hierarchy
Helicone's path-based sessions are useful inspiration, but AIProxer already has proper span/trace identities.
Allow a human-readable workflow path/stage as an optional bounded observational property:
Use it for grouping/visualization via #520, not as the actual parent-child authority when real span IDs are available.
This keeps session visualization stable while avoiding one giant synthetic trace requirement.
8. Content capture
By default external spans store metadata only.
Optional fields such as:
are highly sensitive in coding environments.
If supported:
A useful first version needs timing/outcome metadata, not raw source code.
9. Tool-call correlation
When AIProxer observed a canonical model tool call ID, an external harness can reference that ID on the execution span.
This enables:
But tool-call ID is correlation only, not authorization.
#478/#343 policy decisions remain proxy-owned facts.
10. Retrieval/vector observability
For RAG/agent systems support safe metadata such as:
Do not log embedding vectors by default.
This mirrors Helicone's ability to include vector DB requests in sessions without requiring AIProxer to become the vector database gateway.
11. Clock and timestamp handling
External producers have different clocks.
Requirements:
Do not use external timestamps for security ordering or billing authority.
12. Bounded ingestion
Enterprise telemetry endpoints must resist accidental/hostile floods.
Hard bounds:
Use batch writes and bounded async persistence.
No goroutine or DB transaction per tiny external span under high-volume traffic.
13. Orphan/late spans
External spans may arrive before/after corresponding request metadata.
Store correlation references independently and reconcile/query by stable IDs later.
Do not reject useful late tool/test spans merely because the model request completed several minutes ago, but cap maximum accepted age according to policy.
Unknown parent should remain an orphan/link rather than fabricating a parent.
14. Query and visualization
External spans should appear through:
Useful aggregate questions:
15. Sampling
For huge organizations, support telemetry sampling independent from inference.
Potential rules:
Sampling occurs at ingestion/producer configuration, with explicit dropped-count evidence where possible.
16. Relation to #502 exporters
The AIProxer Enterprise UI/store may ingest external spans even when the customer's OTLP exporter is disabled.
When #502 OTLP export is enabled, external accepted spans can optionally be re-exported/linked according to policy.
Avoid export loops:
Mark source/export provenance and suppress loops.
17. Open-Core boundary
This is a strong closed Enterprise agent-observability feature.
Possible split:
Using open trace standards in OSS should not be intentionally crippled.
Suggested V1
Acceptance criteria
Non-goals
Why 9.6/10
For coding agents, a model request is often only a fraction of task wall time. A gateway-only trace can tell you the model took 3 seconds but miss the 45-second test run or failing shell command that determined the actual experience. Helicone's session + manual logger design demonstrates the value of correlating those external steps. AIProxer can turn that into a particularly strong Enterprise feature by combining external agent spans with its unusually rich routing, policy, cost and session evidence.