feat(llm-bench): route generation sweeps and sustained load

[ishaan-shivhare]

Summary

• split generation-sweep batches across num_servers * num_gens and send one explicitly targeted n > 1 completions request per selected worker
• require full shareable-prefix cache hits for gen_load_test.py measurements so cache misses cannot silently become retries
• share generator-worker routing helpers with load_test.py
• add sustained targeted traffic to load_test.py: Locust users are assigned round-robin to generator workers and continuously replenish requests
• add --reuse-prompt to materialize and reuse one exact prompt across all load-test requests
• add --warmup-prompt-cache to prime each targeted worker once before its measured traffic begins
• keep cache misses non-fatal in load_test.py; cached-token counts remain visible through existing logs and metrics
• support multiple OpenAI completion choices in load-test responses and account for n when validating output-token counts

Why

gen_load_test.py should remain a latency-per-forward sweep tool. Its routed mode now measures the intended backend batching path by splitting a requested global batch into one n > 1 request per generator worker.

Continuous profiling traffic belongs in load_test.py, which already owns concurrency, request replenishment, Ctrl-C/runtime termination, and Locust metrics. With -u 16 --num-servers 8 --num-gens 1 -n 128, two Locust users target each generator service. One request can remain queued while the active request drains, keeping occupancy much steadier than globally repeating sweep rounds.

Fixed-prompt mode makes cache reuse explicit. Each worker receives one max_tokens=0 warmup with the same prompt and stable session id used by its subsequent traffic. Load testing records cache behavior without aborting the run on a cache miss.

Validation

./llm_bench/benchmark/bin/python -m pytest -q
44 passed

The only warning is the existing Locust/gevent monkey-patch warning produced by the test import order.

---

Note

Low Risk
Changes are confined to benchmark/load-test clients and docs; they do not alter production inference paths, though misconfigured routing headers could skew load-test results.

Overview
Generation benchmark (gen_load_test) no longer fans a routed global batch into many concurrent n=1 calls. It splits the batch across targeted generator workers via new shared generator_routing helpers and sends one concurrent n > 1 request per worker, with per-worker usage and forward-pass math. Measured runs must prove a full shareable-prefix prompt-cache hit (header/body cached token counts); misses raise PromptCacheVerificationError and are not retried. The old 64-request sequential pre-warmup hack is removed in favor of one warmup per worker when routing is on.

Locust load test (load_test) adds --reuse-prompt, --warmup-prompt-cache (one max_tokens=0 warmup per targeted worker, deduplicated), and --num-servers / --num-gens so users round-robin to generator targeting headers with a shared user id per worker for session-aware caching. Response parsing aggregates multiple choices when n > 1, and token-count warnings expect max_tokens * n.

README documents the sustained cached-load workflow and an example command.

^{Reviewed by Cursor Bugbot for commit 14a7672. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale comment

Risk: medium. Left a non-blocking comment (not approved): Bugbot completed with an unresolved soft-stop/SIGINT finding, and this routed-batch/cache/repeat change exceeds the low-risk auto-approve threshold. Assigning reviewers for human review of measurement correctness and interrupt handling.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 14a7672. Configure here.}

[cursor]

Warmup skips transient request retries

Medium Severity

_warm_prompt_cache issues a single requests.post and lets any RequestException (TLS/network close, HTTP 408/429 via raise_for_status) escape. warm_worker_once then stores that failure in the shared AsyncResult, so every later user on that worker fails immediately and on_start exits the Locust process. This breaks the llm_bench warmup retry rule that transient transport failures should use a bounded backoff loop.

Additional Locations (1)

• llm_bench/load_test.py#L1303-L1318

 

^{Triggered by learned rule: llm_bench warmup: retry transient requests.exceptions.RequestException with bounded backoff}

^{Reviewed by Cursor Bugbot for commit 14a7672. Configure here.}

[cursor]

Warmup timeout too short

Medium Severity

Non-streaming cache warmup hard-codes timeout=60 while forcing stream=False. The README’s sustained-load example warms ~128k-token prompts, so the full prefill must finish before any bytes return. That commonly exceeds 60s and fails startup even when the deployment is healthy.

Additional Locations (1)

• llm_bench/README.md#L141-L158

 

^{Reviewed by Cursor Bugbot for commit 14a7672. Configure here.}

[cursor]

Risk: medium. Left a non-blocking comment (not approved): Bugbot ended as skipping with unresolved medium warmup findings, and this routed-batch/cache change exceeds the low-risk auto-approve threshold. Two reviewers are already assigned for human review.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}