audit(performance): Reality check for high volume, highly concurrent proxy use

[matdev83]

audit(performance): Reality check for high volume, highly concurrent proxy use

Perform a thorough analysis of performance with focus on supporting at least thousands of concurrent connections per server, with an ambitious goal of tens of thousands. Analyze hot paths for lock contention, allocation/copy amplification, persistence/query amplification, cache opportunities, transport topology, and other concurrency/scalability limits.

---

Implementation tracking

This issue is the authoritative implementation tracker for the approved, implementation-ready Kiro SDD originally introduced by #396 and refreshed on current architecture by merged spec-only PR #447:

.kiro/specs/high-concurrency-performance-hardening/

The canonical spec remains fully approved and ready_for_implementation: true. #407 was a redundant implementation-task issue and is closed as a duplicate of this tracker.

Dependencies / current measurement baseline

• No active hard blocker inside OSS Base. Evidence-first work can proceed against current main.
• Phase 1/2 must capture a fresh baseline on current main; do not reuse older benchmark assumptions.
• Material production/runtime baseline changes include fix(json): harden JSON trust boundaries with bounded preflight policies #434, feat: add non-forwardable conversation content #435/fix: harden conversation-view anchors and identity validation #437, feat(reasoningpreservation): add semantic compression #444, feat(runtime): harden A-leg cancellation and B-leg termination #446, feat(runtime): add pluggable agent loop breach prevention #448, feat(database): enforce SQLite/PostgreSQL dialect parity #529, completed extension-plane work through docs(spec): archive extension plane consolidation #547 and fix(extension-planes): apply review corrections #555/docs(spec): archive extension plane review corrections #556, the pre-OSS slimming ownership waves feat(pre-oss): closed-plane contract on generated planes (1.x+2.x) #576/feat(pre-oss): move tool-call repair into feature (3.x) #582/feat(pre-oss): move secret-guard into feature + compose adapter (4.x) #578/feat(pre-oss): relocate compaction detector to infra (5.x) #579/feat(pre-oss): runtimebundle adapters + zero feature imports (6.x) #580/feat(pre-oss): ownership ratchets, SDK fixture, docs, evidence (7.x+8.x) #581, and the subsequent race/canonical-policy corrections fix(runtime): synchronize concurrent terminal state #589/fix(feature): make generated plane policy canonical #591. Re-inventory current owners and synchronization rather than measuring historical paths.
• ci: add pre-OSS exact race gate #588 introduced an exact scoped Linux race workflow and test(runtime): isolate parallel stream assembly state #590 repaired test-only parallel state exposed by it. Exact run 33891386913 is green on fix(feature): make generated plane policy canonical #591's merged SHA. This is useful correctness/concurrency evidence for the slimming scope, not audit(performance): Reality check for high volume, highly concurrent proxy use #394 load/HOLD/soak capacity certification.
• Verification/test-harness changes affecting how baselines are captured but not production capacity by themselves include perf(test-infra): faster local test/lint loops for agents and devs #449, fix(quality): clear repository lint baseline #544, perf(test): optimize test suite and quality-checks execution duration #553, feat(test): add Windows test-cost performance ratchet #558/perf(test): bound tagged QA connector builds #560/perf(test): restore bounded tagged QA connector builds #562/perf(test): reduce tagged QA subprocess contention #563/perf(test): parallelize isolated runtimebundle suites #564/fix(test): make performance baseline suites deterministic #566/fix(test): stabilize authority store concurrency proof #567/feat(test): ratchet tagged QA hotspot cost #568/fix(quality): clear repository and connector modules lint debt #570/feat(quality): add parallel multi-module linting to local pre-commit and quality gates #574 and chore(pre-oss): pin inventory SHAs to merged main #584/fix(pre-oss): baseline SHA validated as mainline ancestor #585/fix(pre-oss): hermetic git identity in inventory SHA test #586/ci: stabilize Windows test-cost ratchet #587. Use the current harness and ratchets, but do not count faster QA or test stability as proof that production request capacity improved.
• Re-inventory the spec(performance): evidence-driven high-concurrency hardening #396/spec(perf): refresh post-#446 high-concurrency baseline #447 hot-path findings against current main and measure current owners under the canonical HOLD/DELTA/START/BODY/OUTPUT/COMPLETE/RETRY/OBSERVE/AUTHZ/B2BUA/DISCONNECT/SOAK scenarios.
• feat(performance): Add large-payload streaming fast path to avoid full request materialization #503/task: Implement Kiro spec large-payload-streaming-fast-path #532 remains a concrete large-request allocation/copy optimization under this performance umbrella, not a prerequisite for audit(performance): Reality check for high volume, highly concurrent proxy use #394 evidence/harness work. If audit(performance): Reality check for high volume, highly concurrent proxy use #394 signs off relevant large-request ingress/allocation/GC evidence before feat(performance): Add large-payload streaming fast path to avoid full request materialization #503 lands, refresh only those affected measurements on the post-feat(performance): Add large-payload streaming fast path to avoid full request materialization #503 tree. Do not make task: Implement Kiro spec large-payload-streaming-fast-path #532 wait for audit(performance): Reality check for high volume, highly concurrent proxy use #394.
• task: Implement Kiro spec pre-oss-core-slimming #571 does not block audit(performance): Reality check for high volume, highly concurrent proxy use #394. Its production implementation and exact scoped Linux race are effectively complete through fix(feature): make generated plane policy canonical #591; only canonical Task 8.4/spec archive closeout remains. Measure the current tree and refresh only evidence affected by any final closeout commit.
• Merged spec: define full core feature ownership closure #559 / post-split task: Implement Kiro spec core-feature-ownership-full-closure #572 remains follow-up architecture completion and is not an OSS Base dependency for this issue.
• feat(agent-loops): Add agent loop breach prevention #426/feat(tests): Create tests to prove that A-leg cancellation is successfully causing all B-legs termination #431/feat(database): Enforce database type-related code parity #438/task: Implement Kiro spec database-dialect-parity-enforcement #443/task: Implement Kiro spec extension-plane-declaration-consolidation #451/task: Implement Kiro spec extension-plane-review-corrections #549/sdk(feature): Decide support contract for dynamically declared planes #554 are completed baseline work, not future blockers.
• Blocks feat(split): OSS / Enterprise split [ENDS OSS Base Milestone] ============================================ #398, the terminal OSS Base split gate.

Keep dependency relationships strictly within OSS Base; adjacent post-split work is coordination/milestone sequencing only, never a cross-milestone hard dependency.

[matdev83]

Audit result — executive summary

I audited the standard runtime/request/streaming path on current main at 962e1546e29a6df0f6d19f2ae195a9999cdf1b64 with #394's 1k / 10k+ simultaneous-stream goal in mind. This is a static code audit, not a load-test certification; where I call something a blocker below, that means its complexity/synchronization/I/O shape is intrinsically incompatible with the target unless measurements prove an unexpectedly benign operating mode.

Bottom line

The Go goroutine-per-request/stream model is not the main problem. Several concurrency primitives are correctly scoped per stream/request, generation pinning is atomic, the model catalog uses atomic snapshot publication, HTTP clients are shared, and decode admission already exists.

The material risks are instead shared synchronous side effects, unbounded/duplicated per-stream retained state, admission occurring after expensive materialization, and whole-request stream timeouts.

I would not certify 1,000 simultaneous active streams from the current evidence, and I do not consider 10,000+ active streams credible on current main when secure-session stream recording is enabled in its current form. The latter has direct synchronous store work on every client-facing event, and the in-memory implementation adds global-lock + linear-scan behavior to that event path.

Priority findings

Priority	Finding	Why it matters at high concurrency
P0	Secure-session recorder performs synchronous store work for every client-visible stream event	A normal transcript-enabled delta can drive TouchActivity, transcript sequence lookup + append, audit sequence lookup + append (roughly 5 store calls before considering usage/control-plane projection). Durable DB latency is therefore in the token/delta forwarding path.
P0	In-memory secure-session store globally serializes mutations and scans transcript/audit slices for next sequence numbers	Unrelated sessions contend on one RWMutex; per-event NextTranscriptSeq / NextAuditSeq are O(events-so-far), creating O(n²) work over a long stream/session.
P0	Stream runtime retains already-forwarded output in multiple representations until terminalization	retryRecvStream retains seenEvents and visibleText; customerEvidenceAccumulator separately accumulates text/reasoning/tool arguments and content events. Token accounting later reconstructs from the full event history. Memory therefore scales with cumulative output per live stream, not merely the number of live streams.
P0	Request body is fully ReadAll'd and JSON-preflighted before decode admission	Existing 32-decode / 64 MiB limits do not bound body materialization. A burst of 1,000 maximum-size 8 MiB requests can theoretically allocate ~7.8 GiB just for body backing arrays before decode admission; 10,000 is ~78 GiB, before decompression/JSON/DTO overhead.
P0 correctness	Default inbound and outbound whole-request timeouts are 120 s	Healthy LLM streams lasting >2 minutes can be terminated by absolute request/write deadlines. This must be fixed before meaningful long-stream scale testing.
P1	Auth event delivery holds one process-wide mutex while invoking the sink	Default auth/session-start logging becomes a globally serialized synchronous critical section across otherwise independent requests.
P1	B2BUA in-memory continuity store converts ordinary reads into global write-lock operations	FetchALeg/ResolveALeg refresh activity timestamps and therefore take a single store-wide write lock; create can also sweep expirations while holding it.
P1 / conditional	In-memory concurrency-authority admission uses one mutex and scans the lease set during Acquire	Reclaim + live-count scans occur under the same lock. There is a useful 100-contender benchmark, but not evidence for 1k–10k admission concurrency.
P1	Turn start/finish persistence has avoidable extra reads/writes	Secure session begin/resume and B2BUA preparation perform several synchronous store operations before first token. This can dominate burst admission even after steady-state event recording is fixed.
P1	Durable terminal billing spool is deliberately serialized/durable but expensive per completion	Owned SQLite uses one connection + FULL synchronous durability; each append performs replay lookup, pending COUNT/SUM, disk-capacity checks, insert, commit. This is terminal-only rather than per-token, but can bottleneck completion storms.
P2	Deep lipapi.Call clones amplify large-prompt allocations across preparation/retries	CloneCall recursively copies messages, raw JSON, tool schemas/options/extensions; retries clone the baseline again. Large prompts × multi-attempt failover create substantial GC pressure.
P2 / feature-dependent	Traffic/audit observation can add synchronous marshal/copy/sink work	When enabled, traffic emission redacts/clones payloads and calls sinks synchronously. Some paths marshal before testing whether emission is a no-op.
P2	A few small shared executor/credential/state locks remain	These should be cleaned up/profiled, but they are lower ROI than the P0/P1 items above.

High-confidence positive findings

A few things should not be rewritten merely because they use synchronization:

• Generation request retain/release is implemented with atomic/CAS refcounting rather than a process-wide lock (internal/infra/runtimehost/generation_refcount.go).
• modelcatalog.CatalogRuntime.Active() is an atomic.Pointer read (internal/core/modelcatalog/runtime.go).
• streampump.EventPump synchronization is per stream, not process-wide.
• leglifecycle.Coordinator keeps expensive B-leg cancel/close work outside its top-level map lock; most mutable B-leg state is scoped to one A-leg.
• The process-owned decode limiter is shared across generations, so config reload does not multiply the admission budget.
• The standard outbound transport is shared and attempts HTTP/2.

So I would avoid a broad "remove all mutexes / add worker pools" refactor. The right work is much narrower: remove DB/sink work from event cadence, bound retained bytes, move admission earlier, shard only truly shared mutable stores, and establish end-to-end load gates.

Main source locations inspected

• internal/core/runtime/secure_session_stream_record.go
• internal/core/securesession/app/recorder.go
• internal/core/securesession/adapters/memory/store.go
• internal/core/securesession/adapters/bunstore/store_transcript.go
• internal/core/securesession/adapters/bunstore/store_audit.go
• internal/core/runtime/executor_retry_stream.go
• internal/core/runtime/customer_evidence.go
• internal/core/runtime/executor_settlement.go
• internal/plugins/frontends/frontendpipe/pipe.go
• internal/plugins/frontends/reqbody/body.go
• internal/plugins/frontends/decodeqos/*
• internal/stdhttp/generation_host.go
• internal/infra/httpclient/standard.go
• internal/core/auth/events.go
• internal/core/b2bua/store.go
• internal/infra/concurrencyauthority/leasestore/memory.go
• internal/infra/billingspool/spool.go
• pkg/lipapi/call_clone.go
• pkg/lipsdk/traffic/*
• internal/infra/runtimehost/generation*.go

I detail the P0/P1 mechanics and recommended changes in follow-up comments.

[matdev83]

Detailed audit — persistence, synchronization, and write amplification

This is the highest-ROI part of the audit. The code currently has a few operations whose frequency changes their performance class completely: they look cheap in unit tests, but they execute once per stream delta or once per request while holding process-shared state.

1. P0 — secure-session persistence is directly in the client-event hot path

internal/core/runtime/secure_session_stream_record.go::beforeEmitClientFacing runs before a client-facing canonical event is released. It marshals a wire representation and synchronously invokes SecureSessionRecorder.RecordPostHookStreamEvent.

The normal transcript-enabled path in internal/core/securesession/app/recorder.go then performs, for a typical event:

1. TouchActivity
2. NextTranscriptSeq
3. AppendTranscript
4. NextAuditSeq
5. AppendAudit
6. and, for usage-bearing events, potentially AddUsage

This is event cadence, not turn cadence. If an upstream emits 10–30 deltas/s, multiply that by the number of active streams. At 10k streams, even 10 events/s is 100k events/s; a design requiring several synchronous store operations for each event cannot reasonably be expected to scale by tuning a DB pool.

Durable store compounds the problem

The Bun implementations currently make the explicit Next*Seq calls partly redundant:

• internal/core/securesession/adapters/bunstore/store_transcript.go
  • NextTranscriptSeq checks session existence and executes MAX(seq).
  • AppendTranscript starts a transaction, updates/locks the parent session, checks transcript policy, computes MAX(seq)+1 again, inserts the row, and records the turn.
• internal/core/securesession/adapters/bunstore/store_audit.go
  • NextAuditSeq similarly reads MAX(seq).
  • AppendAudit locks the parent session and computes the sequence again inside the transaction before insert.

So the application-level sequence lookup does not eliminate the transactional sequence lookup; it adds another round trip/query.

Recommended architecture

The important objective is not merely fewer queries. It is: no remote/durable DB round trip on every token/delta.

I recommend:

• Make sequence allocation part of the append operation. Remove NextTranscriptSeq / NextAuditSeq from the event path.
• Store O(1) monotonic sequence counters per session/stream (e.g. atomic DB update / UPDATE ... RETURNING) rather than MAX(seq) scans.
• Coalesce transcript deltas into bounded chunks (for example bytes-or-time based) rather than one row per canonical delta.
• Coalesce TouchActivity (e.g. update at bounded intervals and at terminalization), not once per delta.
• Write transcript chunk + audit metadata + activity update in one transaction/batch where durable ordering requires them together.
• For best-effort recording, use a bounded asynchronous batch/outbox path with explicit overload semantics.
• For mandatory "durable before visible" recording, a remote SQL transaction per delta is still the wrong primitive. Use a local durable append/WAL/outbox or bounded group commit, acknowledge the durable append, then asynchronously normalize/materialize into query tables.
• Define loss/backpressure policy by evidence class. Audit evidence and full transcript payloads do not necessarily need identical persistence semantics.

This is the single change most likely to move the project from hundreds toward thousands of active streams.

---

2. P0 — in-memory secure-session sequencing is globally serialized and O(n²)

internal/core/securesession/adapters/memory/store.go has one store-wide sync.RWMutex. More importantly, stream recording uses write-locked operations and:

• NextTranscriptSeq scans the transcript slice to find the highest sequence.
• NextAuditSeq scans the audit slice to find the highest sequence.

Because those functions run repeatedly as a session grows, total sequence-discovery work for an n-event stream is approximately 1 + 2 + ... + n = O(n²). Because the scan occurs behind one store-wide lock, unrelated sessions are serialized while it happens.

This means switching from SQL to the memory store is not a valid scale workaround.

Recommended fix:

• Put nextTranscriptSeq / nextAuditSeq counters in the session record/state and increment in O(1).
• Partition mutable session storage by session ID (sharded maps are sufficient; no need for a complex lock-free map).
• Use per-session locking for transcript/audit append invariants instead of one global lock.
• Add explicit retention bounds for memory mode: max sessions, max transcript/audit bytes per session, TTL/expiry cleanup, and metrics. I did not find a request-path eviction API that would make historical session accumulation obviously bounded.
• Do not perform full-store sweeps while holding a request-path global lock.

---

3. Conditional P0/P1 — control-plane recording can duplicate event-path persistence

When control-plane observation wraps the secure-session store (internal/infra/controlplane/observers/securesession_store.go):

• TouchActivity delegates, then records a control-plane session event.
• AppendAudit delegates, then records a control-plane audit event.
• AddUsage delegates, then records a control-plane usage event.

internal/core/controlplane/recorder.go::prepareAppend synchronously calls store.Append.

Therefore, when enabled, an already expensive secure-session stream event can cause additional synchronous ledger appends. Best-effort changes error policy; it does not make the operation asynchronous.

Recommendation: derive control-plane records from the same chunked/turn-level durable fact stream rather than independently persisting another event for every low-level delta.

---

4. P1 — B2BUA memory store turns reads into store-wide writers

internal/core/b2bua/store.go::MemoryStore uses a single RWMutex, but normal operations call lockForOperation, which takes the write lock because reads refresh LastSeenAt.

That makes FetchALeg, ResolveALeg, and related lookups serialize globally. CreateALeg can also sweep expired legs under this lock.

At scale, activity timestamps should not force canonical lineage/state lookups into a global write critical section.

Recommended fix:

• Separate immutable/stable lineage state from activity metadata.
• Shard by A-leg/session ID.
• Make activity refresh lazy/coalesced or atomic/per-entry.
• Move expiry cleanup off the foreground path, or make it incremental/amortized with a tiny bounded budget per request.

---

5. P1 — auth/session-start event delivery is globally serialized around sink I/O

internal/core/auth/events.go::EventDispatcher has one mutex. Both DispatchAuthDecision and DispatchSessionStart acquire it and hold it while invoking the sink.

internal/infra/runtimebundle/auth_events.go installs a synchronous slog event sink in the default delivery mode. Thus otherwise independent requests can queue on one mutex while logging/sink I/O executes.

Recommended fix:

• Define whether EventSink must itself be concurrency-safe.
• Do not wrap the standard concurrency-safe sink in a process-wide ordering mutex.
• If an injected sink is not concurrency-safe, adapt that sink specifically rather than serializing every possible sink.
• If ordering is a semantic requirement, order by session/principal key, not globally.
• If logging/audit is best effort, use a bounded queue/batcher with explicit overflow telemetry; if it is mandatory, use a scalable durable append primitive rather than one in-process mutex held over I/O.

---

6. P1 / conditional — concurrency-authority memory store has O(total leases) admission under one mutex

internal/infra/concurrencyauthority/leasestore/memory.go::Acquire takes one MemoryState.mu, then:

• scans leases to reclaim expired entries for the rule/dimension;
• can scan again to count live entries;
• replay/remaining-slot paths can scan again.

The repo has contention_bench_test.go exercising 100 contenders for 5 slots, which is useful correctness/perf groundwork, but the benchmark topology is far below #394's target and does not address growth in the backing lease map.

If this mode participates in production request admission, make capacity accounting O(1) per dimension/rule and keep lease lookup indexed by lease ID. Expiry should use indexed time buckets/heap/wheel or bounded lazy cleanup rather than scanning the entire store on every acquire.

---

7. P1 — secure-session turn start has avoidable store chatter before TTFT

internal/core/securesession/app/manager.go::BeginTurn performs several store operations before upstream execution:

• new session: A-leg creation, session Create, then TouchActivity;
• resumed session: lookup, then TouchActivity;
• executor preparation subsequently needs A-leg data again.

The Bun session Create path inserts and reloads the record. These are not per-token operations, but they matter enormously during a 1k/10k connection-open burst.

Recommended fix:

• Prefer INSERT ... RETURNING / construct the newly created immutable result directly instead of insert + select.
• Carry stable A-leg/session state returned from BeginTurn; avoid immediately reading it again through another store authority.
• Coalesce activity updates.
• Instrument DB queries per request before first upstream byte, DB pool wait duration, and transaction duration. Query count should become an explicit performance contract.

---

8. P1 — billing terminal spool is safe/durable, but completion throughput needs its own scale class

internal/infra/billingspool/spool.go deliberately prioritizes durability. When it owns SQLite it sets MaxOpenConns(1), uses WAL + synchronous=FULL, and each append roughly performs:

• record seal + JSON marshal;
• transaction begin;
• existing-record lookup;
• COUNT + SUM(payload_bytes) over pending/delivering records for capacity;
• disk-capacity checks;
• insert;
• synchronous commit.

That is much better than doing billing DB work on every token, but a terminal storm of thousands of completions/s can still serialize here.

I would preserve the correctness/durability contract but optimize the mechanism:

• maintain pending count/bytes incrementally instead of aggregate scans on every append;
• reconcile counters periodically/crash-safely;
• batch/group terminal appends where exact durability allows;
• expose a production scale profile where PostgreSQL/external durable queue is expected above a documented terminal rate;
• benchmark completion storms separately from steady active-stream concurrency.

---

9. Lower-priority locks: profile before redesigning

I also checked several other shared synchronizers:

• Generation lease/refcounting is atomic/CAS and looks appropriate.
• Affinity memory store is an RW-locked map with short operations; shard if mutex profiles demonstrate pressure.
• Credential pool has one mutex and linear credential scans; likely small in normal deployments, but large credential pools/high attempt rates should be benchmarked.
• Executor lazy dependency/RNG locks are small, but immutable/eager process dependencies could remove some request-time mutex acquisitions cheaply.
• Generic in-memory feature state uses one mutex and can JSON-decode while locked; feature authors should not use it at high event cadence without sharding/bounding.

These are not where I would start. Fix event-path persistence, secure-session O(n²), auth sink serialization, and B2BUA write-lock reads first.

[matdev83]

Detailed audit — memory growth, allocations, transport lifetime, and ingress admission

The second major class of problems is memory/lifetime behavior. At 10k connections, even a few hundred KiB of avoidable live state per stream becomes GiBs, so the important question is not only allocation rate but what remains reachable for the full duration of a stream.

1. P0 — already-delivered output remains live in multiple representations

internal/core/runtime/executor_retry_stream.go::retryRecvStream currently retains, among other state:

• seenEvents []lipapi.Event
• visibleText strings.Builder
• completion/gate queues and drains
• baseline/request state needed for recovery/terminal work

Separately, internal/core/runtime/customer_evidence.go::customerEvidenceAccumulator retains:

• accumulated visible text
• accumulated reasoning text
• accumulated tool argument text
• content []lipapi.Event

Then terminal token accounting (internal/core/runtime/executor_settlement.go) copies/uses the full seen-event history and released output text for StreamUsage.Reconstruct.

So a long stream may have several live copies/representations of bytes that have already been sent to the client. This creates a much worse capacity function than "one goroutine + small structs per socket":

live memory ~= fixed stream state + f(all output emitted so far)

For an ambitious proxy, that is a structural problem. Ten thousand 1 MiB responses should not require keeping ten thousand MiB-scale histories resident until all streams terminate.

Recommended change

• Make token accounting incremental. Maintain bounded counters/tokenizer state as events pass rather than reconstructing from the entire history at terminalization.
• Replace seenEvents with only the small amount of state needed for recovery/dedup/terminal semantics. If a feature genuinely needs history, give that feature an explicit bounded history budget.
• Make customer evidence incremental/aggregated. Do not retain both concatenated strings and the same semantic content as an event slice unless required by a proven contract.
• After output commitment makes retry/failover illegal, aggressively release large recovery-only objects/baseline copies that no longer affect behavior.
• Add a benchmark that keeps a stream open while delivering 1, 10, 100 MiB and checks heap slope per MiB already forwarded. The desired slope should approach zero except for deliberately retained bounded state.

There is currently an active structural spec under .kiro/specs/turn-recv-terminal-ownership-simplification/. Its requirements explicitly preserve current streaming/accounting behavior while moving responsibilities into cleaner owners. That refactor is useful, but #394 should be coordinated with it: merely moving seenEvents/evidence accumulators into a new response-pipeline owner would preserve the scalability defect behind better architecture. The new owner should support incremental/bounded evidence as part of the target shape.

---

2. P0 — decode admission happens too late to bound request-body memory

The standard frontend pipeline currently does:

1. reqbody.ReadAll
2. JSON structural preflight
3. decodeqos.TryAdmit
4. decode into the frontend DTO

See internal/plugins/frontends/frontendpipe/pipe.go and internal/plugins/frontends/reqbody/body.go.

The existing defaults are good after admission (32 concurrent decodes / 64 MiB inflight decode bytes), but the code/comments explicitly place body ReadAll and preflight before that gate. Maximum request body defaults to 8 MiB.

This allows a connection/request burst to allocate independently of the decode budget. The simple theoretical upper bound illustrates the risk:

• 1,000 × 8 MiB = ~7.8 GiB of body data
• 10,000 × 8 MiB = ~78 GiB

Those are not measured resident-memory numbers; they are the raw payload-array upper bounds and intentionally exclude gzip expansion, JSON preflight buffers/objects, frontend DTOs, cloned canonical calls, and GC overhead.

The existing decode limiter should remain, but it needs an earlier ingress/materialization gate.

Recommended two-stage admission:

• Stage A, before body read/decompression: cap concurrent materializations and reserve a byte budget.
  • Use trustworthy bounded Content-Length when present.
  • For chunked/unknown/gzip bodies, reserve progressively or pessimistically up to a configured cap.
  • Reject overload before allocating the entire body.
• Stage B: keep the existing weighted decode admission to bound CPU/decoded-object memory.

Also test gzip expansion/bombs explicitly; the current request limit is applied to decompressed bytes, which is correct for safety but means compressed network size is not a useful memory reservation proxy by itself.

PR #164 intentionally introduced "decode-only weight after ReadAll/preflight". #394 is the natural follow-up rather than evidence that #164 was wrong; it solved a narrower resource class.

---

3. P0 correctness — 120-second absolute timeouts conflict with long LLM streams

Inbound server construction (internal/stdhttp/generation_host.go) installs ReadTimeout, WriteTimeout, and IdleTimeout; the effective/default write timeout is 120 s.

The shared outbound HTTP client (internal/infra/httpclient/standard.go) defaults to:

• http.Client.Timeout = 120s
• ResponseHeaderTimeout = 60s
• shared transport
• ForceAttemptHTTP2 = true

http.Client.Timeout covers the whole request, including reading the response body. A server WriteTimeout is likewise not a semantic "no progress for N seconds" stream timer. A healthy model producing output for >120 seconds can therefore be killed solely because it is long-lived.

Before scale testing, separate absolute request lifetime from idle/progress timeouts:

• outbound streaming: no whole-request Client.Timeout; retain connect/TLS/response-header timeouts and request-context cancellation, plus an explicit idle-between-upstream-events watchdog if desired;
• inbound streaming: do not apply a whole-response write deadline to SSE/streaming routes; enforce a progress/idle write deadline if required;
• preserve finite header/body admission timeouts for non-streaming request ingress.

Required regression: a healthy 10–30 minute synthetic stream must survive without special provider behavior. This is also essential for 10k soak tests; otherwise the test simply churns every connection after two minutes.

---

4. P1/P2 — lipapi.Call deep cloning multiplies large prompt allocations

pkg/lipapi/call_clone.go::CloneCall recursively copies mutable call state: messages/parts, raw JSON bytes/items, tool schemas, options, extensions, semantic extensions, session metadata, etc.

Preparation keeps defensive baselines, and attempt opening/retry can clone again. This is semantically understandable but expensive for large coding-agent prompts containing long history/tool schemas.

At high request burst rates, this becomes GC pressure even when streams themselves are mostly idle.

Recommended direction:

• Freeze canonical request data into an immutable request snapshot at the appropriate boundary.
• Share immutable subtrees across retries/attempts.
• Apply transforms with copy-on-write overlays or clone only fields a transform actually mutates.
• Keep the public semantic contract defensive, but do not require repeated physical deep copies of a 1 MiB prompt inside trusted runtime ownership.

Benchmark matrix should include prompt sizes 1 KiB / 64 KiB / 1 MiB and 1 / 3 attempts, reporting bytes allocated/op and peak live heap during a connection burst.

---

5. P1/P2 — traffic observation can make the data plane synchronous and allocation-heavy

pkg/lipsdk/traffic/emit.go already has EmitIsNoop, which is the right mechanism: expensive hot-path payload preparation should be skipped when no sink/redactor/observer consumes it.

However, when emission is enabled:

• ApplyRedactors begins by cloning the body;
• event payloads are JSON-marshaled;
• sinks/redactors/observers run synchronously;
• an existing runtime traffic benchmark explicitly notes that event JSON marshaling dominates BTP/PTC CPU/allocation in that path.

Also inspect call sites for unconditional marshaling before EmitIsNoop; secure-request/CTP preparation contains opportunities to avoid work when traffic output is disabled.

Recommendations:

• Require no-op checks before every payload marshal/copy.
• Avoid cloning if there are no redactors and the sink ownership contract allows immutable borrowed bytes.
• Make diagnostic payload capture a clearly documented scale trade-off: sampling/aggregation/metadata-only modes should exist for production high-volume use.
• Best-effort observers should not be able to stall a customer's stream indefinitely; bounded asynchronous export is appropriate where semantics permit.
• Benchmark traffic disabled vs metadata-only vs full payload/redaction modes independently.

---

6. P2 / conditional — generic memory state can serialize feature traffic

internal/core/state/mem.go uses one process-wide mutex for all keys. Get holds it while decoding the stored value, including JSON marshal/unmarshal paths.

This is fine for low-rate feature state. It is not suitable as a hidden per-delta state bus for future features. The SDK should document the performance contract, and high-frequency users should get sharded/per-scope stores or purpose-built owners.

---

7. Network/socket capacity must be part of the target, not just Go heap

The direct lipstd data listener is a normal net/http cleartext server; I did not find an h2c wrapper in the standard server construction. Behind an ingress/LB that is often completely acceptable, but direct 10k streaming clients then imply roughly 10k inbound TCP connections, plus upstream connections/HTTP2 streams, DB connections, diagnostics, etc.

The standard outbound transport attempts HTTP/2, which is positive, but provider-side HTTP/2 MAX_CONCURRENT_STREAMS and connection-pool behavior still need to be measured per backend family.

A production scale claim therefore needs to record:

• open file descriptors/handles;
• inbound TCP connections;
• outbound TCP connections and HTTP/2 stream fan-in;
• socket memory;
• reverse-proxy/LB idle and response-duration timeouts;
• provider connection/stream limits;
• reconnect behavior during upstream GOAWAY/network churn.

Do not "optimize Go" while an LB, OS handle limit, or provider H2 stream cap is the actual ceiling.

---

8. Existing bounded mechanisms worth preserving

There are good foundations that should remain:

• request-body hard size limits;
• weighted decode admission;
• process-scoped shared limiter across reload generations;
• bounded completion-gate concepts;
• shared outbound HTTP transport and HTTP/2 attempt;
• per-stream event-pump ownership rather than a giant global stream dispatcher;
• atomic generation snapshot/pin paths.

The issue is mostly where the bounds are placed and which data remains live, not an absence of resource-control concepts.

[matdev83]

Recommended implementation order and performance-certification plan

I would handle #394 as an evidence-driven scalability program, not as a broad optimization/refactor ticket. The order matters: there is little value micro-optimizing maps/formatting while the stream hot path still performs durable persistence and retains whole histories.

Phase 0 — add measurement before changing semantics

Build one local synthetic upstream/load harness that can deterministically control:

• TTFT;
• stream duration;
• events/second;
• bytes/event and total response bytes;
• upstream HTTP/1.1 vs HTTP/2 where practical;
• failures before first output / after commitment;
• slow/stalled upstream;
• slow/stalled downstream client;
• large request bodies and gzip expansion;
• 1 vs N failover attempts.

Use the local/reference backend rather than a real inference provider so provider rate limits/cost do not contaminate proxy measurements.

Capture on every run:

• throughput and concurrent active streams;
• proxy-added TTFT;
• per-event forwarding delay, p50/p95/p99;
• request/error/429/503/timeout rates;
• process RSS + Go heap/live objects;
• alloc bytes/op or bytes/request where meaningful;
• goroutine count;
• file descriptor/handle count;
• Go mutex profile and block profile;
• CPU + heap profiles;
• GC CPU/pause/frequency;
• DB queries/request and queries/stream-event;
• DB transaction latency and connection-pool wait;
• secure-session/control-plane rows or bytes generated per response MiB;
• outbound connection count / HTTP2 connection and stream reuse;
• billing spool append latency/backlog during completion storms.

Commit the harness/scenarios to the repo so later features cannot silently regress the target.

---

Phase 1 — fix correctness/resource blockers before throughput tuning

1. Streaming timeout semantics

   • Remove the 120 s whole-stream ceiling from streaming inbound/outbound paths.
   • Add explicit idle/progress watchdog semantics instead.
   • Regression: healthy stream survives 10–30 min.

2. Early body/materialization admission

   • Add a gate before ReadAll, decompression and JSON preflight.
   • Keep current decode admission as a second stage.
   • Regression: a burst of maximum-size/gzip requests is rejected/bounded without an RSS cliff.

3. Bound stream-retained output state

   • Convert terminal token accounting/evidence to incremental state.
   • Remove full seenEvents/duplicate text retention where not semantically required.
   • Regression: keeping a stream open while emitting additional MiBs should not cause proportional live-heap growth after already-forwarded content is no longer needed.

These are prerequisites for a meaningful 1k/10k soak.

---

Phase 2 — remove per-event persistence from the critical path

This is the largest throughput change.

Target architecture:

canonical event -> bounded in-memory accounting/state -> client emit

with persistence becoming something closer to:

bounded transcript/audit chunk -> local durable batch/outbox (if required) -> batched materialization/export

Specific changes:

• O(1) per-session sequence counters;
• one append API allocates sequence + persists, no NextSeq round trip;
• chunk transcript deltas by size/time;
• coalesce activity timestamps;
• batch transcript/audit/session metadata;
• avoid duplicate control-plane persistence at raw-delta cadence;
• make best-effort recording explicitly asynchronous/bounded;
• for fail-closed durability, retain durability without remote SQL RTT per delta (local WAL/outbox/group commit).

Performance invariant: the normal streaming hot path should perform zero remote/durable SQL transactions per text/reasoning delta.

This should be an architecture/performance test, not only a benchmark convention.

---

Phase 3 — eliminate process-wide serialization on ordinary independent requests

In priority order:

1. Secure-session memory store: per-session/sharded locking + O(1) sequences.
2. Auth event dispatcher: no global lock held across sink I/O.
3. B2BUA memory store: shard stable state; decouple activity refresh and expiry sweeps.
4. Concurrency-authority memory mode: indexed O(1) live-count accounting per dimension/rule; bounded expiry cleanup.
5. Generic state/affinity/credential pools: shard/index only if profiles show material wait.
6. Executor lazy dependency/RNG locks: remove opportunistically after the large bottlenecks, not as a headline change.

Gate: mutex/block profiles during a 1k+ active-stream run should not show a single process-wide application lock dominating wait time.

---

Phase 4 — reduce large-request allocation amplification

• Freeze immutable canonical request snapshots.
• Share immutable message/tool/schema subtrees between baseline and attempts.
• Copy-on-write only where transforms mutate.
• Audit traffic/observer call sites for marshal-before-noop and unnecessary payload clones.
• Make production observability modes explicit: disabled / metadata / sampled payload / full payload.

Benchmark prompt-size × retry-count combinations and run with GC/memory profiles.

---

Phase 5 — terminal throughput

Steady active-stream capacity and completion rate are different dimensions.

Optimize/certify the billing terminal spool separately:

• O(1) capacity bookkeeping instead of COUNT/SUM on every append;
• batch/group commit where durability semantics permit;
• explicit SQLite scale envelope;
• production recommendation for a higher-throughput durable store/queue when the completion rate exceeds that envelope.

Run a completion-storm scenario with many short streams ending simultaneously.

---

Proposed benchmark matrix

A. Idle/slow-stream concurrency

Run 1k -> 5k -> 10k simultaneously open streams for at least 10–30 minutes with sparse heartbeats.

Purpose: fixed per-connection cost, socket/FD limits, goroutine footprint, timeout correctness, leaks.

B. Active streaming

Run 1k / 5k (then 10k if hardware allows) with at least two event rates, e.g. low and high delta cadence.

Test configurations independently:

• minimal/no secure persistence;
• secure-session memory persistence;
• durable secure-session persistence;
• control plane off/on;
• traffic payload observation off/on;
• one backend vs multiple failover candidates.

Purpose: distinguish core networking capacity from optional persistence/observability cost.

C. Long-output memory slope

Keep a fixed number of streams open and progressively emit large responses.

Plot live heap against bytes already delivered. The important result is the slope, not merely peak allocation rate.

D. Connection-open burst

Start 1k and then 10k requests as simultaneously as possible using bodies of approximately:

• 1 KiB
• 64 KiB
• 1 MiB
• max-body size
• compressed high-expansion equivalent

Purpose: body admission, prompt clones, DB pool/startup persistence, auth logging, request-generation binding.

E. Slow downstream

Upstream produces faster than the client reads.

Purpose: prove queue/backpressure behavior is bounded and that no hidden event backlog grows indefinitely.

F. Failure/failover storm

Pre-commit upstream failures with 1 vs 3 attempts, including large prompts.

Purpose: clone amplification, routing/credential locks, attempt cleanup, authority lifecycle.

G. Completion storm

Thousands of short streams terminate close together.

Purpose: billing spool, terminal DB work, session audit finalization.

H. Reload while loaded

Keep thousands of streams pinned to an old generation while publishing a new configuration and admitting new requests.

Purpose: prove generation lifetime/refcounts remain cheap and old generations drain without resource duplication/leak.

---

Suggested release gates

I would not invent arbitrary latency/RSS numbers before a baseline run, because hardware and feature profile matter. Instead establish the baseline first and then ratchet reproducible budgets. However, several qualitative gates can be stated now:

1. No whole-request timeout kills a healthy >120 s stream.
2. No normal text/reasoning delta requires a remote/durable SQL transaction.
3. No O(events-so-far) sequence scan occurs per stream event.
4. No process-wide mutex is held while performing default sink/log I/O.
5. Request overload is bounded before full body materialization.
6. Live heap does not grow proportionally with already-forwarded response bytes except for explicitly bounded/declared feature state.
7. All queues/buffers used for asynchronous diagnostics/persistence have explicit byte/item bounds and overload policy.
8. A 10–30 minute steady-load run has stable heap/goroutine/FD/backlog trends after warm-up.
9. Mutex/block profiles show no dominant global application critical section under the certified load profile.
10. The certified profile documents which optional features were enabled; "10k streams" without persistence/traffic/control-plane configuration is not a meaningful claim.

After the first baseline, add numeric ratchets for proxy-added TTFT, p99 event forwarding delay, RSS/stream, bytes allocated/request, DB pool wait, and accepted completion throughput.

---

Coordination with existing work

The ongoing turn-recv-terminal-ownership-simplification spec is a good architectural opportunity, but its current mandate is primarily structural/behavior-preserving. I recommend explicitly feeding the #394 constraints into that implementation:

• the response-pipeline owner should hold bounded incremental state, not simply inherit all current history buffers;
• terminal accounting APIs should accept incremental evidence/counters;
• secure-session recording should be factored so chunking/batched persistence has one owner;
• the refactor should leave no new "universal mutable turn bag" and no background goroutine per stream just to move state around.

In other words: use the ownership refactor to make the performance fix easier, but do not mistake ownership cleanup itself for a scalability fix.

Overall recommended priority

If only four changes are funded initially, I would do them in this order:

1. Remove event-cadence durable secure-session persistence / O(n²) sequencing.
2. Stop retaining complete output history for terminal reconstruction.
3. Move admission before body materialization.
4. Fix long-stream timeout semantics and then build the 1k/10k soak harness.

Everything else should be driven by the resulting CPU/heap/mutex/block/DB profiles.

[matdev83]

Tracking update

The audit now has a governing implementation-ready Kiro SDD in #396 / .kiro/specs/high-concurrency-performance-hardening/.

Implementation is tracked in #407. Phases 1–2 (performance laboratory + untouched baseline) are intentionally not blocked by #403/#401; production changes that touch recv/response/request ownership must coordinate with the then-current #403 → #401 architecture.

[matdev83]

Status sync — prior runtime-refactor coordination baselines have landed

The issue remains open and implementation-ready. Its native dependency summary currently reports no active blockers, and the Kiro SDD is still ready_for_implementation: true.

The body’s future-looking coordination note around #403 → #401 is now historical: those runtime/request-pipeline refactors have landed on main. Performance hardening should therefore target the current authoritative owners/seams directly. Phases 1–2 (performance lab + baseline evidence) remain independently actionable as specified.