[RFC]: Upgrade the GPU backend to vLLM 0.28.0

[jiangkuaixue123]

Motivation

The GPU backend currently targets vLLM 0.26.0 at commit 568afb3a13806beb53bb2e6bd518269357b237c0, while roadmap #155 identifies the 0.28 line as the next mirrored compatibility target. The target v0.28.0 tag resolves to commit 2cf0a6915ce544dc493a0990f2ea38d81601128a.

The upgrade crosses plugin-owned workers and model runners, model construction and weight loading, compatibility patches, EngineCore lifecycle, connector metadata and topology, CUDA Graph/DBO behavior, profiling, packaging, and E2E qualification. A version-pin-only update would not prove that the existing AFD contracts still map to the target upstream architecture.

This RFC therefore treats 0.26.0 to 0.28.0 as a coordinated GPU runtime migration. Implementation starts with an exact-source architecture audit and proceeds only when the current AFD invariants can be preserved through native vLLM extension points, inheritance/composition, or small exact-version adapters.

Goals

• Upgrade the supported GPU runtime from exactly vLLM 0.26.0 to exactly vLLM 0.28.0.
• Freeze the current and target vLLM source identities (v0.26.0 and v0.28.0) and the immutable pre-upgrade AFD revision before implementation.
• Inventory and preserve the currently supported CUDA contract, including:
  • DeepSeek V2/V3/V3.2, Qwen3 MoE, and the Qwen3.5/Qwen3.6 adapter family;
  • P2pNcclAFDConnector Attention-to-FFN and FFN-to-Attention ordering;
  • eager and FULL_DECODE_ONLY CUDA Graph execution;
  • native two-ubatch DBO where currently supported;
  • profiling and the represented TP/DP/EP topologies;
  • Attention ownership of requests, scheduling, KV cache, sampling, and output;
  • FFN ownership of connector-driven MoE execution without request-facing or KV-cache ownership.
• Audit every copied upstream method, compatibility patch, private vLLM dependency, and worker/model-runner override against the exact v0.28.0 source.
• Prefer target-native vLLM behavior and stable extension points; retain only narrow AFD-specific differences with focused AFD and non-AFD tests and a removal/upstream plan.
• Make exact-runtime native control, the complete GPU E2E suite, full repository-owned GPU accuracy coverage, and clean teardown release gates.

Non-goals

• This RFC does not upgrade the NPU or vLLM-Ascend runtime.
• This RFC does not claim support for a vLLM version range; the target is exactly 0.28.0.
• This RFC does not add new model families, connector protocols, or unrelated GPU features.
• ModelRunnerV2 support remains tracked by [RFC]: Add GPU ModelRunnerV2 Support for AFD #223 and is not implied by this upgrade unless that work is separately completed and included in the frozen feature matrix.
• Async GPU connector work remains tracked by [RFC]: Async GPU Connector #233 and is not a prerequisite for restoring the currently supported synchronous GPU contract.
• The native MoERunner refactor remains tracked by [RFC]: Refactor AFD MoE forward around native MoERunner injection #225. This upgrade should consume an approved native seam where available, but it must not hide a broad architecture redesign inside a version port.
• This RFC does not modify the vLLM source tree.

Proposed change

1. Freeze identity and current behavior

Record the immutable AFD base revision, current vLLM v0.26.0 commit, target vLLM v0.28.0 commit, intended Python/PyTorch/CUDA/C++ ABI/toolchain tuple, target GPU topology, and validation model paths.

Build the affected-surface and feature inventories before editing production code. The inventory must cover version/package contracts, plugin registration and CPU-safe imports, all patches and copied upstream methods, model wrappers and weight policy, Attention/FFN workers and model runners, EngineCore/KV/scheduler ownership, connectors and process groups, CUDA Graph, DBO/ubatching, TP/DP/EP/PP, profiling, request output, failure propagation, shutdown, tests, recipes, and documentation.

2. Architecture gate against vLLM v0.28.0

Compare the exact v0.26.0 and v0.28.0 source revisions and classify affected contracts as mechanical, behavioral, or architectural.

Implementation must stop for an explicit maintainer decision if a runner generation or lifecycle owner was replaced, a required seam disappeared, a supported execution cell was removed, or preserving behavior would require a new public protocol, broad upstream copies, reflection-heavy compatibility, silent feature deletion, or a validation bypass.

For every copied or patched function, reconstruct from the exact v0.28.0 upstream skeleton and replay only still-required AFD differences. Match decorators, signature, return type, call order, ownership, resource lifetime, graph/compile behavior, and shutdown semantics.

3. Version, packaging, and compatibility baseline

Update package/runtime pins, lock metadata, compatibility checks, documentation, test fixtures, CI, and active recipes from 0.26.0 to 0.28.0 only after the architecture gate passes.

For each compatibility seam, choose one explicit disposition:

• remove because target upstream absorbed the behavior;
• delegate/inherit through a target-native extension point;
• port as an exact v0.28.0 adapter;
• retain temporarily while proposing a concrete upstream hook;
• stop because no narrow invariant-preserving adaptation exists.

Any retained patch must follow the repository patching rules: copy the target implementation where practical, mark only AFD-specific deltas with # ### PATCH START / # ### PATCH END, cover AFD and non-AFD paths, document performance implications, and state a removal or upstream plan.

4. Model and MoE integration

Audit the registered GPU model families against their exact v0.28.0 native implementations.

• Preserve native construction, ordinary forward behavior, parameter naming, quantization hooks, and weight loading wherever the target exposes a usable seam.
• Keep AFD ownership limited to role-aware construction/composition, remote-expert handoff, connector communication, gate/expert placement, and role-specific weight filtering.
• Reconcile this work with [RFC]: Refactor AFD MoE forward around native MoERunner injection #225 rather than creating a second model-specific MoE abstraction.
• Keep non-AFD model resolution and execution native and isolated from AFD registration.
• Fail closed for model, quantization, pipeline, multimodal, or runner combinations outside the frozen supported matrix.

5. GPU workers, model runners, and EngineCore

Adapt AFDAttentionWorker, AFDFFNWorker, AFDAttentionModelRunner, GPUFFNModelRunner, and the AFD EngineCore integration to the v0.28.0 lifecycle.

The review must cover construction and class selection; device/distributed initialization; model loading; scheduler, request, KV-cache, and sampling ownership; forward-context installation and cleanup; eager and graph execution; DBO/microbatch state; dummy/warmup/profile paths; FFN daemon startup and failure propagation; output processing; memory accounting; and clean shutdown.

The FFN path must remain minimal and connector-driven. It must not acquire resources or responsibilities that belong only to the Attention/request-facing path.

6. Connector, metadata, and topology contracts

Audit P2pNcclAFDConnector and all GPU-side control/tensor payloads against target vLLM distributed and forward-context contracts.

Preserve producer/wire/consumer ordering, stage and layer identity, tensor shape/dtype/device ownership, CUDA stream synchronization, process-group lifetime, physical rank/device mapping, and graph replay behavior. Validate represented symmetric and asymmetric Attention:FFN layouts rather than relying only on equal-rank topologies.

Do not add a new public connector method or protocol unless the architecture gate proves that the existing AFD boundary cannot carry the required invariant.

7. Staged migration

1. Identity and inventory: freeze exact revisions, runtime tuple, current feature matrix, patches, and private seams.
2. Upstream diff and architecture gate: map target changes to AFD consumers and approve or stop each affected boundary.
3. Package and compatibility update: update pins/checks and remove, delegate, or port compatibility seams.
4. Model integration: adapt model construction, MoE boundary, weight ownership, and native lifecycle reuse.
5. Runtime integration: adapt workers, model runners, EngineCore, connector metadata, topology, graph/DBO, profiling, and cleanup.
6. Focused validation: pass formatting, imports, package, signature/patch, unit, contract, and non-AFD isolation tests.
7. GPU qualification: pass target-native control, the complete non-accuracy GPU E2E matrix, then every full GPU accuracy case.
8. Documentation and final audit: reconcile support claims, recipes, CI, patch inventory, release metadata, and cleanup evidence with the frozen final revision.

Each implementation stage should be independently reviewable and revertible. Keep one upstream cause and its focused regression tests together.

Plugin boundary

Plugin-owned:

• Attention/FFN role selection and resource ownership;
• AFDAttentionWorker and AFDFFNWorker orchestration;
• AFD-specific behavior in AFDAttentionModelRunner and the minimal GPUFFNModelRunner;
• connector send/yield/receive semantics, payload ordering, and topology mapping;
• role-aware model composition, remote-expert handoff, gate/expert placement, and weight filtering;
• AFD configuration, control metadata, failure propagation, cleanup, and GPU E2E coverage.

Compatibility helpers or patches:

• only v0.28.0 seams that cannot be expressed through supported inheritance, composition, or target-native hooks;
• exact-version constructor, lifecycle, scheduling, forward-context, graph/DBO, or distributed differences, with narrowly marked AFD deltas and removal/upstream plans.

Upstream-owned and unchanged where usable:

• native vLLM model, worker, model-runner, EngineCore, scheduler, sampling/output, graph/compile, distributed, and weight-loading behavior outside explicit AFD hooks;
• GPU operators and native runtime artifacts;
• non-AFD execution and model registration.

Primary class paths:

• afd_plugin.v1.worker.attention_worker.AFDAttentionWorker;
• afd_plugin.v1.worker.ffn_worker.AFDFFNWorker;
• afd_plugin.v1.worker.attention_model_runner.AFDAttentionModelRunner;
• afd_plugin.v1.worker.ffn_model_runner.GPUFFNModelRunner.

Validation plan and acceptance criteria

The upgrade is complete only when all of the following are true:

• current and target vLLM tags resolve to recorded full SHAs, the AFD base/final revisions are frozen, and the Python/PyTorch/CUDA/C++ ABI/native-artifact tuple is reproducible;
• package metadata, lock data, runtime checks, CI, and active GPU documentation target exactly vLLM 0.28.0;
• the architecture gate records a narrow invariant-preserving adaptation for every affected upstream surface;
• every copied or patched function matches the exact target signature and patch requirements, with focused AFD/non-AFD coverage and a removal/upstream plan;
• model construction, loading, gate/expert ownership, Attention/FFN worker and runner responsibilities, connector ordering/topology, graph/DBO, profiling, failure, and cleanup contracts match the frozen supported matrix;
• non-AFD vLLM behavior remains native and isolated from AFD registration/runtime changes;
• repository formatting, compile/import, package, version, patch/signature, focused, and complete CPU/unit gates pass;
• the exact target vLLM runtime serves a real eager GPU request with AFD disabled before AFD parity testing;
• every repository-owned non-accuracy GPU E2E case passes on the frozen final AFD revision with no unexplained skip, including feature, model, serving, eager/graph, DBO, topology, and profiling scenarios represented by the suite;
• every repository-owned GPU accuracy case passes at full scope after the complete basic GPU gate;
• evidence records the exact AFD/vLLM revisions, runtime and native artifact identity, image/host, GPU topology, model paths, commands, actual collected/pass/fail/error/skip counts, metrics, logs, and process/port/GPU/IPC cleanup;
• active documentation and release metadata distinguish GPU-validated, shared-code, unit-only, unvalidated, unsupported, and intentionally historical claims;
• no NPU support claim is inferred from shared-code or GPU-only evidence.

A subset, launch smoke, limited accuracy run, configuration flag, or evidence from an earlier commit is diagnostic only and cannot qualify the release.

Risks and alternatives

Risks

• Upstream changes between 0.26.0 and 0.28.0 may cross worker/model-runner generations, EngineCore ownership, model/MoE construction, graph/DBO behavior, or distributed metadata at multiple layers.
• Carrying copied 0.26.0 bodies forward can preserve obsolete lifecycle assumptions even when tests import successfully.
• Model refactoring, compatibility patch refreshes, and runtime migration can mask one another's regressions.
• Equal-rank or eager-only tests can miss asymmetric topology, replay, DBO-stage, synchronization, and cleanup bugs.
• A GPU-only change can regress shared code used by NPU without proving NPU runtime support.
• The target wheel, PyTorch build CUDA, toolkit/driver, ABI, NCCL, and target compute capability may not form one reproducible validation cell.

Mitigations are exact-source identity, a pre-edit architecture gate, atomic adaptations, independent patch review, focused contract tests, native control, complete E2E/accuracy gates, explicit shared-code regression evidence, and recorded teardown.

Alternatives considered

• Only update the version pin and fix failures reactively: smaller initial diff, but does not audit silent lifecycle, ownership, graph, or distributed contract changes.
• Copy complete v0.28.0 model/worker/runner methods: expedites a mechanical port but increases private-code drift and conflicts with the repository patching policy.
• Support 0.26.0 and 0.28.0 simultaneously: requires version branching and multiplies the matrix; the project currently uses exact mirrored compatibility.
• Upgrade GPU and NPU together: couples independent runtime stacks and validation environments; NPU alignment should have its own RFC and evidence.
• Bundle ModelRunnerV2, async connector, or broad MoERunner redesign: expands the architecture and acceptance surface beyond restoring the current GPU contract; those workstreams remain separately tracked.

Feedback requested

1. Does the current supported GPU feature matrix above match what must be preserved on v0.28.0?
2. Which v0.28.0 changes, if any, should trigger an architecture stop rather than an exact-version adapter?
3. Which current compatibility patches can be removed or replaced by target-native extension points?
4. Should any part of [RFC]: Refactor AFD MoE forward around native MoERunner injection #225 land before the version migration, or should the upgrade first preserve the current MoE boundary and refactor separately?
5. Which asymmetric Attention:FFN and TP/DP/EP cells are required for release qualification?
6. Which shared-code changes require explicit NPU regression evidence without making an NPU 0.28.0 support claim?

Feedback period

Proposed feedback period: one week, through 2026-09-11. Read-only identity, inventory, and upstream-diff work may proceed during review, but production adaptation should begin only after the affected architecture boundaries are agreed.

Related issues

• [RFC]: afd-plugin project roadmap #155 — project roadmap, mirrored versioning, and vLLM alignment policy.
• [RFC]: Upgrade the GPU backend to vLLM 0.26.0 #167 — previous GPU backend upgrade from vLLM 0.19.1 to 0.26.0.
• [RFC]: Add GPU ModelRunnerV2 Support for AFD #223 — GPU ModelRunnerV2 support.
• [RFC]: Refactor AFD MoE forward around native MoERunner injection #225 — native MoERunner forward refactor.
• [RFC]: Async GPU Connector #233 — async GPU connector.
• [Tracker]: afd-plugin active development #263 — active development tracker.

CC

@jiaran-king @specture724 @yujuancao07