test(vllm): restore KV-transfer migration coverage

[alec-flowers]

Summary

• restore the vLLM disaggregated KV-transfer migration test on top of test(vllm): restore decode migration coverage #13693
• pass the explicit NIXL --kv-transfer-config now required by vLLM 0.27.1
• replace the inherited eight migration combinations with two KV-specific cases while the aggregate test continues to own shared migration-policy coverage
• reduce the functional workload from an approximately 8K-token prompt and 8,192-token context to an 823-token transferred prefix, 1,024-token context, and 192-token output cap
• start the independent prefill and two decode workers concurrently and wait on the frontend's structured discovery view before injecting the fault
• add the measured 6.9 GiB VRAM marker so the pre-merge GPU scheduler selects the test during qualification
• apply a measured 350-second pytest timeout and leave a TODO to return the restored matrix to nightly after no-retry pre-merge CI passes

Scope and traceability

The target now retains two cases:

Case	Fault lifecycle	API	Response	Request plane
worker-failure-chat-stream-nats	SIGKILL	chat	streaming	NATS
graceful-shutdown-chat-unary-tcp	SIGTERM	chat	unary	TCP

• Chat is retained because the historical defect involved chat-template prefix-cache hits and reuse of the request ID.
• Migration-disabled and sequence-cap combinations cannot enter successful KV transfer; they remain covered by the aggregate migration test.
• Completion differs only at the renderer boundary for this defect. Streaming/unary are paired across the retained cases rather than multiplied with every other axis.
• Both fault lifecycles and both request planes remain represented.

On the stacked base, this changes the target from eight skipped outcomes to two executing outcomes. The module changes from 20 outcomes (12 pass / 8 skip) to 14 executing outcomes with zero function-level migration skips. The original pre-#13684 Cartesian target had 96 combinations.

Root cause and restored contract

The first forced execution failed before migration because current vLLM rejects disaggregated startup without an explicit KV connector. After configuring NIXL, the old 8K workload did transfer approximately 878.5 MB, but nearly filled the functional test's cache and failed to complete within the 350-second test timeout (377.97 seconds including teardown).

The smaller workload still transfers an 823-token prefix through NIXL and exercises the historical same-request-ID/chat-prefix path. The test now proves that the request is still in flight when the worker fault is injected, the user response completes after migration, and the frontend reports exactly one ongoing-request migration.

Profiling also exposed an independent startup race: three concurrently healthy engines did not guarantee that the frontend had observed both decode instances. The test now condition-polls /health until one prefill and two decode generate instances are present; it does not add sleeps or retries.

Validation

Exact image:

210086341041.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo:10d71514e9c9c1c6a1b902be4b692378e78bff8a-vllm-runtime-nightly-test

All qualification runs used the local GPU, the exact nightly image, the mounted model cache, and zero pytest/OpenAI retries.

• readiness-hardened NATS fault case: 1/1 passed in 100.31s (96.63s call)
• repository NVML profiler: 2/2 passed; calls 96.53s and 77.19s; 177.74s pytest / 184.7s profiler wall
• profile: 6.9 GiB peak, 668 MiB baseline and final, 0 MiB measured leak
• CI-equivalent --max-vram-gib=41 -n 1: scheduler selected both cases at 6.9 GiB / 0.31 GiB requested KV and passed 2/2 in 216.05s (128s and 88s including scheduler-managed setup/teardown)
• exact collection: two KV-transfer cases, neither skipped nor deselected
• py_compile, git diff --check, isort, Black 23.1.0, flake8, codespell, Ruff, and file-integrity hooks passed

The repo-wide marker-report hook also reported this module at 14 collected tests with no missing marker sets, then failed elsewhere during full-repository collection because the local port pool was exhausted in tests/serve; that unrelated collection failure is not hidden by a retry.

Stack navigation

1. #13684 — Aggregate migration
2. #13692 — Prefill removal · DYN-4118
3. #13693 — Decode migration · DYN-4119
4. #13694 — KV-transfer migration · DYN-4120

This PR is layer 4 and targets codex/vllm-decode-migration. Review and merge the stack in order.

---

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.