Skip to content

feat(qwen4_exp): stream the PLE n-gram table from disk #311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jason-fxz wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat(qwen4_exp): stream the PLE n-gram table from disk #311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cf89fb3
feat(qwen4_exp): stream the PLE n-gram table from disk (--ple-backend…
jason-fxz Aug 31, 2026
697dedf
fix(qwen4_exp): hash disk PLE rows with the checkpoint-loaded constants
jason-fxz Aug 31, 2026
8f0395b
fix(kernel): handle io_uring partial submission and validate ple_stor…
jason-fxz Aug 31, 2026
2b27ea2
fix(qwen4_exp): validate PLE row coverage, widen deferred-fill signal…
jason-fxz Aug 31, 2026
ccddade
fix(qwen4_exp): zero the eager PLE staging for the warmup prefill
jason-fxz Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions python/freetoken/engine/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ class EngineConfig:
moe_backend: str = "auto"
# NVFP4 routed-expert GEMM backend (--nvfp4-backend): auto|marlin|flashinfer|triton.
nvfp4_backend: str = "triton"
# PLE table backend: "disk" (default) reads rows from the checkpoint files per fill, "pinned" preloads the table into page-locked host RAM.
ple_backend: str = "disk"
# Expert-bank host load (--expert-load): auto|serial|parallel. "auto" reads scattered
# experts in parallel but falls back to serial when free RAM can't cover the banks + the
# parallel reader's extra (non-reclaimable) whole-shard buffer; "serial" forces the
Expand Down
8 changes: 3 additions & 5 deletions python/freetoken/engine/engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -917,11 +917,9 @@ def rebuild_runtime_cache(

def forward_batch(self, batch: Batch, args: BatchSamplingArgs) -> ForwardOutput:
assert torch.cuda.current_stream() == self.stream
with self.ctx.forward_batch(batch):
if self.graph_runner.can_use_cuda_graph(batch):
logits = self.graph_runner.replay(batch)
else:
logits = self.model.forward()
use_graph = self.graph_runner.can_use_cuda_graph(batch)
with self.ctx.forward_batch(batch), self.model.forward_host_ctx(batch, use_graph):
logits = self.graph_runner.replay(batch) if use_graph else self.model.forward()
if self.cpu_moe_executor is not None:
# One pinned read: surfaces a fired flag-handshake watchdog (dead coordinator
# -> stale expert outputs) as a loud error instead of silent corruption.
Expand Down
Loading