feat(enterprise-export): Add governed asynchronous event sinks for Kafka and enterprise data platforms

[matdev83]

Assessment: 8.9/10

Expected enterprise/data-platform value: very high
Proxy-level fit: very high
Implementation leverage vs complexity: high if built as one normalized asynchronous export contract rather than direct sink calls scattered through request handling

Summary

Add an Enterprise event/data export plane that publishes completed AIProxer request/trace/usage/security lifecycle projections asynchronously to customer-owned data infrastructure, starting with Kafka and leaving a clean adapter path for Pub/Sub, data warehouses and SIEM-oriented sinks.

Bifrost Enterprise provides direct observability connectors such as Kafka, Pub/Sub and BigQuery. Its Kafka connector publishes completed request traces as JSON, keyed by trace ID for partition ordering, with asynchronous batching, compression, TLS/SASL, content suppression and selective span/header export.

Reference:

• https://github.com/maximhq/bifrost/blob/dev/docs/features/observability/kafka.mdx

Why this is distinct from existing AIProxer work

#502 — OpenTelemetry

Provides standard distributed tracing/OTLP semantics and should remain the default interoperability mechanism for observability backends.

#509 — Enterprise request log/object storage

Provides AIProxer-owned searchable metadata plus optional retained content/object references.

#508 — Administrative audit

Provides tamper-evident security/control-plane history and archival.

This issue

Provides a customer-owned streaming data integration plane:

AIProxer completed lifecycle event
        -> bounded enterprise projection/redaction
        -> async exporter
        -> Kafka / PubSub / warehouse ingestion
        -> customer's own analytics/SIEM/data lake

Enterprises frequently want raw normalized gateway events in their existing data platform without scraping AIProxer databases or operating an OTel collector solely to obtain application-specific economic/policy fields.

Architecture principle

Exporters consume completed normalized events after request processing; they never participate in routing, accounting or request correctness.

Exporter outage must not become inference outage unless a future explicit compliance profile deliberately chooses a stronger retention requirement.

1. One normalized export envelope

Define a stable versioned envelope suitable for external processing, containing bounded safe facts such as:

schema_version
event_id / request_id / trace_id
occurred_at / completed_at
principal/org/team/project attribution where authorized
operation/frontend class
requested public route
surfaced provider/model identity
attempt/fallback summary
latency / TTFT / AIProxer-overhead split (#502)
usage/cost/cache evidence
policy/guardrail intervention reason codes
async/batch origin where applicable
runtime/config/pricing generations
content projection state

Do not export internal Go structs directly as a de facto permanent public schema.

2. Event classes

Start with a deliberately small registry:

inference.completed
inference.failed
mcp.completed               #470
async_job.completed          #500
batch.settled                #512

Potential later classes:

guardrail.intervention
credential lifecycle events
alert transitions (#516)
selected administrative/audit events (#508) through a separately approved projection

Do not dump every internal debug event onto enterprise buses by default.

3. Metadata-only default

Default export posture should contain no raw prompt/completion/tool content.

Content modes may include:

none
redacted_only
selected_fields

and must reuse #474/#509's content/redaction projection rather than independently sanitizing inside each exporter.

Reversible-redaction mappings from #513/#474 must never be exported with placeholderized content.

4. Kafka first adapter

Kafka is a strong first validation target because enterprise data stacks commonly consume it.

Support:

broker list
topic
message key strategy
TLS + private CA
SASL PLAIN/SCRAM where required
compression
batch size / linger/flush interval
write timeout
bounded retries

A useful default message key is trace/request/session-safe identity when ordering is beneficial, but key cardinality/partition implications should be documented.

Do not auto-create topics by default; that often requires excessive broker privileges.

5. Adapter contract

Generic enterprise code should know only something like:

Exporter
  ValidateConfig
  Start
  Enqueue(Event)
  Flush(ctx)
  Close(ctx)
  Health

Sink-specific protocol/auth/config belongs in adapters.

Later candidates:

Google Pub/Sub
AWS Kinesis/EventBridge where justified
Azure Event Hubs
BigQuery / warehouse batch sink
generic signed HTTPS collector

Do not make support for every sink a V1 acceptance criterion.

6. Bounded asynchronous pipeline

No sink call in inference hot path.

Use process-owned bounded batching/workers with explicit limits:

queue capacity
max event bytes
max batch events/bytes
flush interval
worker count
retry budget
shutdown drain deadline

On overload/outage, behavior is configured and observable:

drop_export_keep_request
spill to bounded durable outbox when enabled
reject only under an explicit compliance-required profile

No goroutine per event.

7. Optional durable outbox

Best-effort streaming is sufficient for many analytics use cases.

For customers needing stronger delivery, support a durable bounded outbox pattern:

request/event authority commits normal result
 -> enqueue durable export record asynchronously/transactionally where applicable
 -> exporter claims/delivers
 -> mark delivered / retry with backoff
 -> TTL/dead-letter after explicit policy

Do not turn #509 request log rows into a mutable export queue.

Exactly-once delivery to external systems is not required; stable event IDs permit consumer deduplication.

8. Delivery semantics

At-least-once when durable mode is enabled; best-effort in lightweight mode.

Requirements:

• stable event ID across retries;
• payload immutable for one event/schema version;
• retry/backoff bounded;
• dead-letter/failure evidence;
• one bad oversized event cannot permanently block following exports;
• sink acknowledgment interpreted correctly.

9. Security / credentials

Exporter credentials/config are Enterprise secrets:

• persist through feat(enterprise-secrets): Add envelope-encrypted secret persistence with rotation and KMS integration #513 encrypted secret authority or feat(secrets): Add external secret-manager references with cached rotation-safe resolution #483 secret references;
• prefer workload identity/default chains where adapters support them;
• TLS required for non-local production brokers/endpoints;
• custom CA bounded/validated;
• no raw auth values in diagnostics.

Generic HTTPS-style sinks require the same SSRF/redirect/private-network posture as #516/#500.

10. Tenant/data scoping

A global organization exporter may receive all permitted workspace events; a team/project exporter may be scoped to a subtree.

Configuration uses trusted #515/#411 hierarchy IDs, not caller-supplied labels.

#506 controls who may configure/view exporter destinations.

Do not create one producer connection per user/project automatically; exporter count and resource usage must remain bounded.

11. Schema evolution

External data pipelines depend on stable schemas.

Requirements:

explicit schema_version
backward-compatible field addition rules
stable enums/reason codes
unknown field tolerance
migration/change notes
contract fixtures in CI

Do not repurpose a field's meaning silently.

Potential Avro/Protobuf/JSON Schema support can follow after the JSON contract is stable; V1 JSON is sufficient.

12. Relationship to OTel

Do not duplicate every generic trace exporter OTel already provides.

This feature is justified only for AIProxer-specific completed event/economic/policy projections and operational simplicity.

Where an OTLP collector can satisfy a customer's requirement equally well, documentation should say so.

#502 remains the distributed-tracing authority; this feature may include trace IDs and selected span summaries but not invent a second tracing model.

13. Data warehouse sinks

Warehouse direct writers should batch aggressively and remain outside request latency.

A future BigQuery-like adapter should:

• use append/batch APIs;
• partition by time;
• keep stable schema;
• reject uncontrolled prompt-derived column names;
• expose partial-row failures safely;
• avoid one insert RPC per inference.

For long-term large payload retention, #509 object storage is likely cheaper than putting raw source-code prompts into warehouse rows.

14. Alerting and observability

Expose bounded exporter health:

queue depth
events exported/dropped/retried
oldest pending age
batch latency
sink healthy/degraded
last success/failure category

#516 can alert on persistent exporter failure/backlog.

Do not label Prometheus metrics with topic names, URLs, event IDs or arbitrary tenant strings unless carefully bounded.

15. Open-Core boundary

This is a good closed-Enterprise integration feature:

• OSS retains Prometheus and feat(tracing): Add semantic GenAI spans for routing, attempts, sidecars and tool lifecycles #502 standard OTel interoperability;
• Enterprise adds managed direct data-platform sinks, governed scoping, optional durable delivery, sink administration and Enterprise UI;
• shared core exposes only a narrow completed-event projection seam if needed.

This creates Enterprise convenience/value without intentionally crippling open standards in OSS.

Suggested implementation order

1. Versioned metadata-only EnterpriseExportEvent contract.
2. Bounded async exporter manager.
3. Kafka TLS/SASL adapter.
4. Export health/metrics and feat(enterprise-rbac): Add control-plane RBAC and row-level data access scopes #506 management authorization.
5. feat(guardrails): Add pluggable ingress/egress guardrail provider pipeline #474 redacted content projection opt-in.
6. Optional durable outbox.
7. Add one heterogeneous second sink (Pub/Sub or generic HTTPS) to prove abstraction.
8. Add warehouse adapters based on customer demand.

Acceptance criteria

• A completed inference can produce one versioned metadata-only export event without blocking provider response delivery.
• Kafka exporter batches and publishes events with bounded memory and no goroutine per request/event.
• Broker outage does not fail ordinary inference in default mode and produces observable backlog/drop state.
• Raw prompts/tool arguments are absent unless explicitly enabled through an authorized redacted content policy.
• Kafka TLS/SASL credentials are never emitted in logs/read APIs and can use feat(enterprise-secrets): Add envelope-encrypted secret persistence with rotation and KMS integration #513/feat(secrets): Add external secret-manager references with cached rotation-safe resolution #483.
• Stable event IDs allow consumer-side deduplication after retries.
• feat(enterprise-rbac): Add control-plane RBAC and row-level data access scopes #506 limits who can configure/read enterprise exporter destinations.
• feat(enterprise-alerting): Add policy-aware alerts, incident channels and delivery history #516 can observe persistent exporter failure/backlog.
• Existing feat(tracing): Add semantic GenAI spans for routing, attempts, sidecars and tool lifecycles #502 OTLP and feat(enterprise-logs): Add metadata-first request log store with object-storage payload offload #509 logging behavior remain independent when exporters are disabled.

Non-goals

• replacing OpenTelemetry collectors for generic tracing;
• exporting every internal debug event;
• raw content by default;
• exactly-once guarantees across arbitrary external brokers;
• one network write per inference request;
• making feat(enterprise-logs): Add metadata-first request log store with object-storage payload offload #509 database rows a work queue;
• supporting every cloud/data warehouse in V1.

Why 8.9/10

This ranks below core Enterprise pillars such as encryption, HA, governance and identity, but it is still commercially meaningful: mature organizations want AI gateway usage, cost and policy data in the analytics/SIEM platform they already operate. A governed direct-export plane gives them that without coupling AIProxer's request path to their data infrastructure.