feat(load_test): add --session-mode for progressively growing prompts

[nithiyn]

The default workload re-sends one fixed prefix, so a prompt cache is either warm for that whole block or not warm at all. Agent traffic instead grows a single conversation a turn at a time, and DeepSeek-V4 only serves a cheap hit when a request is an exact continuation of the previous one, so the two patterns produce very different TTFT.

In session mode each Locust user owns one conversation. A turn appends (-p minus -pcml) tokens of fresh dataset text plus the previous response, and the conversation restarts once the prompt passes 2x -p, which keeps the average request at the requested -p. -pcml only sizes the turns here and is no longer sent to the server, since capping the cacheable prefix at -pcml would leave the later turns re-prefilling what they just cached. Every request in a session carries the same user id so consecutive turns land on the same generator.

---

Note

Low Risk
Benchmark-only changes with unit tests; default workload is unchanged unless --session-mode is enabled.

Overview
Adds --session-mode to llm_bench so each Locust user runs one growing chat conversation instead of repeating a fixed prompt prefix—closer to agent clients and to cache behavior that only hits on exact continuation (e.g. DSv4).

Each turn sends prior messages plus a new user chunk sized as -p minus -pcml, echoes visible assistant content only (not reasoning) into history, reuses a stable user id for Fireworks session affinity, and resets the thread once estimated prompt size reaches 2× -p. -pcml is not sent to Fireworks in this mode (it only sizes turns). Dataset drawing drops the shared prefix; session_turn is recorded as a metric.

README documents usage; test_session_mode.py covers limits, growth, restart, failed-turn rollback, and session id stability.

^{Reviewed by Cursor Bugbot for commit dd35067. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit dd35067. Configure here.}

[cursor]

History counts reasoning tokens

Medium Severity

complete_turn grows _history_tokens by full completion_tokens / num_tokens, but the committed assistant message is only assistant_content (visible text, reasoning stripped). On reasoning models those totals include thinking tokens that never enter messages, so sessions hit the 2 * -p restart threshold too early and average prompt length drifts below -p.

Additional Locations (1)

• llm_bench/load_test.py#L1870-L1875

 

^{Reviewed by Cursor Bugbot for commit dd35067. Configure here.}

[cursor]

Risk: low. Not approving: Cursor Bugbot completed as skipped, and there is an unresolved finding on reasoning-token counting in session history. Assigned reviewers for human review.

  

_{Sent by Cursor Approval Agent: Pull Request Approver}