fix(llm-bench): stop reading embeddings/rerank responses through iter_lines

[jonathancaevans]

Problem

Embeddings and rerank load tests have been reporting endpoint latency that is almost entirely client-side CPU time.

Both are single-response JSON APIs, but their responses were drained through response.iter_lines(delimiter=b"\n\n"). A JSON document never contains \n\n, so requests' iter_lines never finds a delimiter: it accumulates the whole body in pending and, for every 512-byte chunk (ITER_CHUNK_SIZE), does pending + chunk and then chunk.split(delimiter) across the entire accumulated buffer. That is quadratic in body size.

Embeddings bodies are big — a batch of 50 inputs at 4096 dims is ~2.7 MB — so the scan costs seconds per response. And because it's a tight C-level scan over Python bytes with no I/O, it never yields to gevent, so it serializes across greenlets. The load generator, not the endpoint, becomes the bottleneck.

Measured cost of the old read path by batch size:

batch	body	old read+parse	new
8	0.43 MB	100 ms	0.12 ms
25	1.33 MB	960 ms	0.43 ms
50	2.66 MB	3,852 ms	0.63 ms
64	3.41 MB	5,814 ms	0.84 ms

Fix

Read the body once for these two endpoints and leave the chunk loop to the streaming paths. When --show-response is off, skip decoding the float arrays altogether — usage.prompt_tokens is scanned out of the raw bytes, with a fallback to a real parse if the scan misses.

Verification

Stub /v1/embeddings returning a 2.7 MB body, --embeddings-batch-size 50, 8 users, 30 s:

	requests completed	total_latency p50
before	14	18,000 ms
after	8,366	22 ms

The stub's fireworks-server-processing-time was a constant 120 ms in both runs, so the entire difference was client-side. Unit tests added for the usage extraction (exact-match against a full parse, absent field, malformed body, whitespace).

Also

Surfaces response_bytes and server_side_total_latency in the embeddings and rerank summary rows, so a client-side ceiling shows up in the results rather than being read as server latency.

Impact

Any embeddings or rerank benchmark run to date understates endpoint throughput and overstates latency, increasingly so with larger batch sizes and embedding dimensions. Streaming and completions paths are untouched.

---

Note

Low Risk
Benchmark-only client path change with tests; streaming/completions untouched. Regex scan for prompt_tokens has a safe orjson fallback.

Overview
Fixes embeddings and rerank load tests that were bottlenecked on the Locust client, not the API. Those endpoints return a single JSON body, but the client drained them via iter_lines(delimiter=b"\n\n"), causing quadratic buffering/scans on multi‑MB responses and blocking gevent greenlets—so reported latency and throughput were mostly client CPU.

Rerank/embeddings now use response.content once and skip the line iterator; streaming/completions paths are unchanged. When --show-response is off for embeddings, extract_prompt_tokens scans raw bytes for usage.prompt_tokens instead of parsing huge float arrays, with JSON parse fallback. response_bytes is recorded, and summary CSV rows for embeddings/rerank add response_bytes and server_side_total_latency (with percentiles where applicable). Unit tests cover the token extractor.

^{Reviewed by Cursor Bugbot for commit d3d39fa. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d3d39fa. Configure here.}

[cursor]

Invalid bodies skip failure recording

Medium Severity

The new embeddings/rerank branch sets t_first_token before validating the body and no longer records parse errors via response.failure(). Empty bodies skip the later empty-response check and succeed; malformed JSON with --show-response off is swallowed by extract_prompt_tokens; rerank/--show-response parse errors raise into runner.exceptions instead of failure stats, so fail_ratio / --max-fail-ratio miss broken responses.

 

^{Triggered by learned rule: Locust load tests: record HTTP errors via response.failure(), not RuntimeError}

^{Reviewed by Cursor Bugbot for commit d3d39fa. Configure here.}

[cursor]

Risk: medium. Left a non-blocking comment — Cursor Bugbot finished as skipping and reported an unresolved medium-severity finding on failure recording for invalid embeddings/rerank bodies, so this is above the low-risk auto-approve threshold. Assigned reviewers for human review of the llm_bench response path.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}