Design discussion: first-class out-of-tree model providers

[ErikBPF]

Design discussion: first-class out-of-tree model providers

Summary

While prototyping Qwen3.6 mixed-GGUF execution on ROCm, we found that the model can
benefit from model-specific runtime policy, but the PoC currently places parts of
that policy in generic MoE and cache code.

Before making the Qwen work production-shaped, I would like to align on a broader
model-support boundary: explicitly trusted, out-of-tree model providers using the
existing lazy model registry and runtime-preparation lifecycle.

The goal is not a general plugin framework. It is a narrow way for a separately
distributed model package to provide model semantics, checkpoint adapters, and
model-owned execution policy while FreeToken retains scheduling, cache admission,
backend mechanisms, and graph lifecycle.

PoC evidence

Tested on Orion with an Unsloth Qwen3.6-35B-A3B Q4_K_M GGUF, 8,192-token context,
1,024 generated tokens:

Experiment	Result	Interpretation
Adjacent single-stream control	32.085 tok/s	Reproducible comparison base
Shared-expert/copy overlap	35.004 tok/s	+9.10%, unchanged expert residency
Historical HIP graph run	64.536 tok/s	Correct, but not currently reproducible
Ten-sample current-host graph soak	32.349 tok/s	Exact and fault-free
Earlier apparent BS4	56.528 aggregate tok/s	Actually executed as two BS2 waves
True BS4 GGUF	111.915 aggregate, 28.016/request	1.980x aggregate; 12/12 exact outputs
True BS4 FTW	113.657 aggregate, 28.719/request	+1.56% over GGUF; same hashes

The ownership distinction is important:

• 9.10% came from model execution and copy choreography.
• 1.980x aggregate came from generic scheduling and real batching.
• 1.56% came from checkpoint representation.
• HIP graph replay correctness came from the backend copy mechanism.

These should not become one model-specific optimization surface or one performance
claim.

Coupling exposed by the PoC

The experimental path currently spans four layers:

• The Qwen model coordinates Q8 activation reuse, shared-expert work, expert copies,
  and its fused routed epilogue.
• Generic layers/moe.py gained Qwen-named prepared-decode methods and environment
  checks.
• Generic moe/offload_cache.py gained Qwen-named copy schedules and two-bank
  assumptions.
• Fusion policy is read from a process-global environment value at module import.

That shape would make every optimized model edit generic MoE/cache code, prevents
per-model-instance policy, hides active fallback provenance, and leaves capability
mismatches to hot-path hasattr checks.

It also conflicts with the direction of ongoing generic work:

• perf(moe): pool heterogeneous GGUF cache rows #199 explores heterogeneous GGUF cache-row geometries.
• feat(moe): --moe-collect-stats, so expert-cache behaviour is measurable #231 makes cache behavior measurable independently of model policy.
• feat: Automatic KV/MoE Laddering for decode speed vs context-length trade off ( upto 33% faster decode )  #300 treats KV/MoE residency tradeoffs as scheduler policy.
• [Feature Request] Support compressed-tensors per-channel FP8 checkpoints for Qwen3.5 MoE #252 is a format extension for an already-supported architecture.

Existing lifecycle seam

FreeToken already has most of the required lifecycle:

• ModelSpec
  names the model class, configuration parser, and weight iterator through lazy import
  strings.
• The engine calls prepare_for_runtime() after weights and MoE cache attachment but
  before KV allocation and graph capture.
• Server workers use multiprocessing spawn, so provider resolution must work safely
  in every process rather than relying on parent-only registry mutation.

Proposed boundary

checkpoint architecture
  -> explicitly allowed model provider
       -> model/config/weight-format semantics
       -> model-owned runtime policy
       -> model-specific kernels and safe fallback
            |
            v
       generic core mechanisms
       banks, views, copies, events, backend capabilities
            |
            v
       generic scheduler
       batching, KV/MoE admission, residency, telemetry

Provider-owned

• Model construction and configuration mapping.
• Checkpoint-format adapters.
• Model-specific execution choreography.
• Namespaced custom operators.
• Model-level capability checks and exact fallback.

Core-owned

• Provider discovery and explicit trust enforcement.
• Model resolution and lifecycle.
• Generic bank, view, copy, event, and graph-safe mechanisms.
• Backend capability reporting.
• KV/MoE admission, batching, cache policy, and telemetry.

Narrow provider mechanism

Use standard Python package entry points, keyed by model architecture:

• Discovery enumerates metadata without importing model or CUDA code.
• Only an explicitly allowed provider matching the selected architecture is loaded.
• Provider targets remain lazy import paths like the current ModelSpec fields.
• Resolution is safe and repeatable in spawned scheduler processes.
• Duplicate providers fail explicitly; none silently override another.
• Packages declare a compatible FreeToken range plus a small provider API level.
• Model runtime policy resolves after weights/cache attachment and before graph capture.

This uses importlib.metadata; no general hook framework or new dependency is needed
for the initial contract.

Qwen as a reference provider

The Qwen provider would own:

• Qwen3.6/Qwen3.5 MoE semantics.
• GGUF Q4_K/Q5_K/Q6_K and FTW loading behavior.
• Shared/routed expert choreography and Q8 activation reuse.
• Qwen-specific fused epilogues.
• CUDA/HIP optimized implementations and an exact unfused fallback.

Core changes exposed by that extraction should remain model-neutral: prepared copies,
events, bank views, capability reporting, and graph-safe execution primitives.

Existing built-in registrations can remain behind a compatibility bridge yielding the
same ModelSpec; migrating every built-in model should not block the first external
provider.

Trust and failure behavior

Installing a provider should not be consent to execute it. Startup should require an
explicit allowlist and reject, before GPU allocation:

• A matching provider that was not allowed.
• Duplicate ownership without explicit selection.
• An incompatible provider API level or FreeToken version.
• A provider unsupported by the selected hardware/runtime.

Startup provenance should identify the distribution, version, architecture provider,
checkpoint adapter, resolved runtime policy, backend capabilities, and fallback reason.

Related contributions

The correctness fixes remain intentionally separate from this design discussion:

• fix(rocm): avoid indirect host pointers during HIP graph capture #316 fixes HIP graph capture for expert copies without changing model policy.
• fix(ftw): preserve auxiliary expert banks #317 preserves auxiliary expert banks in FTW without changing model discovery.

Non-goals

• Scheduler plugins or arbitrary graph-capture hooks.
• Silent overrides of built-in architectures.
• Migrating every built-in model in the first change.
• Reporting aggregate batching gains as single-stream model gains.
• Treating the historical 64.536 tok/s run as reproducible performance.

Questions for maintainers

1. Is an explicitly allowlisted, out-of-tree model-provider boundary directionally
   acceptable for FreeToken?
2. Should providers be able to add only a checkpoint format to an existing model
   architecture, or must they own the complete architecture implementation?
3. What is the minimum stable provider surface beyond the current ModelSpec fields?
4. Should tokenizer, reasoning/tool parsers, and multimodal input processing be part
   of the first provider contract?
5. What compatibility promise should apply to provider-supplied CUDA, HIP, Triton,
   and PyTorch custom operators?

If this direction is useful, the next step would be to reduce the Qwen PoC into a
reference provider and identify the smallest model-neutral mechanism changes needed
in core.

[ErikBPF]

We have now published the full Qwen3.6 GGUF + ROCm/HIP PoC as public implementation evidence:

• Draft diff: Draft PoC: Qwen3.6 GGUF on ROCm ErikBPF/FreeToken#2
• Draft branch: https://github.com/ErikBPF/FreeToken/tree/draft/qwen36-rocm-full-poc
• Smaller provider-loading seam: PoC: load model providers from entry points ErikBPF/FreeToken#1

The full PoC covers mixed Q4_K/Q5_K/Q6_K expert banks, Qwen GDN correctness, HIP GGUF kernels, split-bank transfer overlap, and true multi-request admission. On Orion it moved from 55.712 aggregate tok/s in the original ROCm PoC to 111.915 aggregate decode tok/s, principally by admitting a real BS4 workload instead of two BS2 waves.

This also makes the coupling problem concrete: supporting one model currently crosses model parsing/loading, quantized kernels, expert-cache behavior, graph safety, and scheduler policy. The smaller provider seam is an initial experiment toward giving model integrations an explicit boundary; the full branch shows the broader surface that a production design still needs to account for.

The public branch is a squashed snapshot whose tree matches the validated final PoC. It is intentionally marked draft: intermediate rejected experiments are omitted, and the snapshot still needs decomposition, rebase, and current-hardware validation before proposing upstreamable slices.