Solving Inference Bottlenecks (Bad TPS, ITL, TTFT, etc.)

[code259]

Problem: Low TPS & Low Compute/GPU Utilization

• TPS is constrained by inter-GPU bandwidth, causing low compute utilization.
• TTFT = load duration + prompt eval duration
• $1 / \text{eval rate}$
• TPS = eval rate

Example command to get TPS and other stats (use the --verbose flag): bash ollama run qwen3.8 "Write a short summary of how photosynthesis works." --verbose

• Current stats (after a warmup run):

ex. when running qwen3.8 27b:

Solution: Speculative Decoding

• Goal: 60+ t/s (Instantaneous): Feels nearly instant, which is great for rapid coding or continuous prompting.

How it works:

• generate cheap, low-compute draft tokens
• target model will accept/reject the draft tokens
  • we'll need to ensure that the target model accepts enough draft tokens to make a performance benifit truly

Problem: High TTFT

Problem: Cold start

• If model is not in use for a few minutes next user is subjected to cold start which is approximately 75x slower TTFT
• When run the model on cold start the TTFT in total is around 2m 27 seconds compared to a TTFT of 2s otherwise

Potential Problem: 256k Context Window

• Solution: YaRN