Conversation
Add NVTX ranges across the multimodal pipeline (hash branches, processor stages, gpu_model_runner vit_forward, ec_connector load/save) so per-stage cost is visible in nsys reps. Add scheduler-level encoder cache hit/miss accounting via a new EncoderCacheStats aggregate, with misses recorded only after allocate() so budget-deferred inputs don't pollute the miss count. Surfaces through LoggingStatLogger and Prometheus (vllm:encoder_cache_queries, vllm:encoder_cache_hits). Adds VLLM_MM_CACHE_PROBE env for per-invocation HIT/MISS probe logging.
furionw
changed the title
…_meta The previous vit_forward NVTX range wrapped the entire _execute_mm_encoder call, which meant the per-cold-miss save loop (maybe_save_ec_to_connector -> save_d2h) was nested inside it. In nsys analysis the marker reported real_ViT + save_d2h, requiring manual subtraction to see the cache-hit benefit. Split the markers so they are disjoint siblings under preprocess: preprocess |- ec_connector: load (incl. load_h2d) |- vit_forward <-- now ONLY the encoder GPU work |- ec_connector: save (incl. save_d2h) |- gather_mm_embeddings <-- NEW marker |- embed_input_ids / _prepare_mm_inputs Side effect: vit_forward count drops from "every step" to "only steps that ran the encoder" (no-mm-input and all-cache-hit steps no longer emit the marker). This is the desired semantics; analysis scripts that assumed count == step count need updating. Also add per-step token telemetry to correlate forward duration with prefill/decode batch shape: - Python log line: 'step_tokens step=N prefill=P decode=D total=T num_reqs=R' emitted at DEBUG every step, INFO every 50 steps. - Zero-duration NVTX range 'step_meta: prefill=P decode=D total=T reqs=R' joinable with the forward marker by globalTid + timestamp. Mechanism mirrors the existing prefill-detection pattern in this file (num_computed_tokens_cpu < num_prompt_tokens_cpu); no change to scheduler or model behavior. Co-authored-by: Claude
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
We profile multimodal serving with nsys but the rep had no way to separate
per-stage cost: hashing, HF processor, cache lookup, encoder forward, and
ec_connectorload/save all collapsed into one opaque region. Independently,mm_caching_metricsonly covered the host-side processor cache, so we hadno way to verify the GPU encoder cache (post-ViT embedding residency) was
working under sweep load. This PR adds NVTX ranges across the multimodal
pipeline and adds scheduler-level encoder-cache accounting so both surfaces
show up in nsys and in our metrics/log lines. Misses are recorded only
after
allocate(), so inputs deferred by the encoder-compute budget don'tpollute the miss count.
Effect
LoggingStatLoggerline, before / after:New Prometheus counters:
vllm:encoder_cache_queries,vllm:encoder_cache_hits.What Change
vit_forward, andec_connectorload/save.EncoderCacheStats+record_miss()on the encoder cache manager; scheduler emits per step.VLLM_MM_CACHE_PROBEenv for per-invocation HIT/MISS probe logging.Test Plan
mm:*,gpu_model_runner: vit_forward,ec_connector: *ranges visible.