Skip to content

Update run-e2e skill fixing inconsistencies for the Qwen suite #306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jiangkuaixue123 merged 3 commits into from
Sep 7, 2026
+31 −20
Merged

Update run-e2e skill fixing inconsistencies for the Qwen suite #306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d71ee75
fix e2e skill and e2e readme
ronenkat Sep 3, 2026
9dd76cb
Merge branch 'main' into fix-e2e-skill
ronenkat Sep 6, 2026
c8e29c8
update review comments on deepseek-v2-lite
ronenkat Sep 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 18 additions & 9 deletions .agents/skills/run-e2e/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,16 @@ Run one of the model suites:
Each suite contains four gate scenarios:

- baseline-graph
- afd-eager-2a2f
- afd-graph-2a2f
- afd-graph-dbo-2a2f
- afd-eager-2a2f (DeepSeek-V2-Lite gate only; Qwen3 MoE/Qwen3.6 use afd-eager-2a1f)
- afd-graph-2a2f (DeepSeek-V2-Lite gate only; Qwen3 MoE/Qwen3.6 use afd-graph-2a1f)
- afd-graph-dbo-2a2f (DeepSeek-V2-Lite gate only; Qwen3 MoE/Qwen3.6 use afd-graph-dbo-2a1f)

Each gate scenario evaluates the first 7 GSM8K samples. The AFD gate scenarios
use 2 Attention ranks and 2 FFN ranks. `baseline-graph` uses native
DP4/TP1/EP4. The 2A1F cases (`afd-eager-2a1f`, `afd-graph-2a1f`,
`afd-graph-dbo-2a1f`) are local-only scenarios.
Each gate scenario evaluates the first 7 GSM8K samples. The 2A2F AFD gate
scenarios (DeepSeek-V2-Lite only) use 2 Attention ranks and 2 FFN ranks.
`baseline-graph` uses native DP4/TP1/EP4. The 2A1F cases (`afd-eager-2a1f`,
`afd-graph-2a1f`, `afd-graph-dbo-2a1f`) use 2 Attention ranks and 1 FFN rank.
Note that for DeepSeek-V2-Lite 2A1F are local-only scenarios, while for Qwen3 MoE and
Qwen3.6 MoE they are the suite's gate scenarios.

@jiangkuaixue123 jiangkuaixue123 Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also align the device-order paragraph below (lines 89–90), which still calls these “The local 2A1F scenarios.” Since this skill covers Qwen suites whose gate scenarios use this same topology, remove “local” there or explicitly qualify it as DeepSeek-V2-Lite-only. This will keep the execution guidance consistent with the corrected scope here.


## Workflow

Expand All @@ -43,6 +45,10 @@ Before starting pytest, confirm:
- The selected vllm command runs.
- pytest, afd_plugin, lm_eval, datasets, and huggingface_hub are importable.
- HF_HOME points to the Hugging Face cache used for GSM8K and model weights.
- HF_ENDPOINT is reachable: `gsm8k.py` defaults the lm-eval child to
`https://hf-mirror.com`, so an unreachable mirror stalls or fails at GSM8K
dataset resolution only after the servers are already up and devices
reserved. Confirm the default mirror is reachable or override it.
- GPU: the selected devices are visible to CUDA.
- NPU: torch_npu and the Ascend runtime work.

Expand Down Expand Up @@ -80,8 +86,10 @@ export AFD_E2E_DEVICES=0,1,2,3

Device order defines roles: the 2A2F AFD scenarios use the first two devices
for Attention DP2/TP1 and the last two for FFN DP2/TP1/EP2. `baseline-graph`
uses the first four for native DP4/TP1/EP4. The local 2A1F scenarios use the
first two for Attention and the third for FFN.
uses the first four for native DP4/TP1/EP4.
The 2A1F scenarios use the first
two for Attention and the third for FFN (used local-only for DeepSeek-V2-Lite; and
gate topology for Qwen3 MoE and Qwen3.6 MoE).

### 4. Run

Expand Down Expand Up @@ -144,3 +152,4 @@ gate failure.
| AFD_NPU_E2E_MODEL | NPU | no; downloads the selected suite's model when unset |
| AFD_NPU_E2E_VLLM_BIN | NPU | no; defaults to vllm |
| HF_HOME | both | recommended; HF dataset/model cache |
| HF_ENDPOINT | both | recommended; `gsm8k.py` defaults the lm-eval child to https://hf-mirror.com, confirm reachable or override |
24 changes: 13 additions & 11 deletions tests/e2e/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,19 @@ adapter family on real CUDA hardware.
Each default gate runs four scenarios:

- `baseline-graph`
- `afd-eager-2a2f`
- `afd-graph-2a2f`
- `afd-graph-dbo-2a2f`
- `afd-eager-2a2f` (DeepSeek-V2-Lite gate only; Qwen3 MoE/Qwen3.6 use `afd-eager-2a1f`)
- `afd-graph-2a2f` (DeepSeek-V2-Lite gate only; Qwen3 MoE/Qwen3.6 use `afd-graph-2a1f`)
- `afd-graph-dbo-2a2f` (DeepSeek-V2-Lite gate only; Qwen3 MoE/Qwen3.6 use `afd-graph-dbo-2a1f`)

Each scenario evaluates the first 7 GSM8K samples. If `AFD_E2E_DEVICES` is set,
that value is used as-is; otherwise the defaults are:

- `0,1,2,3` for the gate scenarios. The 2A2F AFD cases use the first two for
Attention DP2/TP1 and the last two for FFN DP2/TP1/EP2; `baseline-graph`
uses all four for DP4/TP1/EP4.
- The 2A1F local cases use the first two for Attention and the third for FFN.
- `0,1,2,3` for the gate scenarios. The 2A2F AFD cases (DeepSeek-V2-Lite gate
only) use the first two for Attention DP2/TP1 and the last two for FFN
DP2/TP1/EP2; `baseline-graph` uses all four for DP4/TP1/EP4.
- The 2A1F cases use the first two for Attention and the third for FFN,
leaving the fourth device idle. For DeepSeek-V2-Lite these are local-only
cases; for Qwen3 MoE and Qwen3.6 MoE they are the suite's gate scenarios.

Tests run sequentially and must not skip. Every GSM8K evaluation uses 8

@jiangkuaixue123 jiangkuaixue123 Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also update the “Local 2A1F cases” section below (lines 94–98). It still says all 2 Attention + 1 FFN scenarios are local-only and that CI gates do not run them, which contradicts the Qwen gate scope clarified here. Qualify both the heading and that sentence as DeepSeek-V2-Lite-specific, e.g. “The DeepSeek-V2-Lite 2A1F scenarios are local-only; its CI gate selects the 2A2F scenarios.”

few-shot examples and a 4096-token maximum model length.
Expand Down Expand Up @@ -89,11 +91,11 @@ by AFD scenarios. The suite uses the same GSM8K-7, eight-shot, 4096-token,
`compute_gate_on_attention=true`, pipeline-parallel, asynchronous, and
multi-node execution are not covered; quantization is unverified.

### Local 2A1F cases
### DeepSeek-V2-Lite local 2A1F cases

The 2 Attention + 1 FFN scenarios are local-only cases; CI gates do not run
them. They use the first two devices for Attention DP2/TP1 and the third for
FFN DP1/TP1/EP1, and run GSM8K-7.
The DeepSeek-V2-Lite 2A1F scenarios are local-only; its CI gate selects the
2A2F scenarios. They use the first two devices for Attention DP2/TP1 and the
third for FFN DP1/TP1/EP1, and run GSM8K-7.

```bash
python -m pytest -q -s \
Expand Down
Loading