Skip to content

Spec — ship colibri distributed expert workers: rebase #551 + fix fmt=6 rotation + token-exact parity gate #1

Open
0 of 8 issues completed
Open
Spec — ship colibri distributed expert workers: rebase #551 + fix fmt=6 rotation + token-exact parity gate#1
0 of 8 issues completed
Labels
specPRD/spec — not directly implementable

Description

@lineape
lineape
opened on Aug 15, 2026

Status: RESEARCHED spec — ready for ticket-cutting. The seven open questions are now answered (a second-round research pass on 2026-08-14 independently re-derived every load-bearing claim against the local clone). Corrections to the first draft are folded in inline; see "Corrections vs. first draft" at the bottom. This is a spec (PRD), not a directly implementable ticket.

Problem Statement

The operator runs frontier MoE models (GLM-5.2, DeepSeek-V4-Flash, eventually Kimi K3) via colibri, a pure-C engine that streams routed experts from disk and maps better than llama.cpp to the fleet's tiered hardware (a fast box for attention + Optane/NVMe/disk for experts). colibri's distributed expert workers feature (upstream PR JustVugg#551) would let the fleet fan routed-expert compute across machines — a coordinator holding attention/routing/KV, workers computing the routed expert FFNs — instead of one box streaming every expert from its own disk.

That feature is stalled and unshippable today, for three compounding reasons:

  1. The maintainer refuses to merge feat: add distributed expert workers JustVugg/colibri#551 because its worker re-implements the expert FFN (expert_gate_up → siluf → e8_rot_rows → matmul_qt) as a parallel copy of moe(), with no token-exact worker-vs-local parity test, so it could "silently drift from the real kernel."
  2. Research confirmed the fear is real: the worker copy is missing the fmt=6 (E8/IQ3) input pre-rotation that moe() applies before gate/up, so it produces wrong numbers for E8/IQ3-quantized experts — a bug present even against feat: add distributed expert workers JustVugg/colibri#551's own base, not introduced by drift.
  3. The PR is 762 commits behind dev (based on e4b5bf3d, GitHub mergeable: CONFLICTING), and the author has not followed up on the maintainer's request.

Solution

Rebase JustVugg#551 onto current dev (research shows this is near-trivial), fix the missing fmt=6 input rotation in cluster_worker_run, and add the two-layer token-exact parity gate (mirroring the JustVugg#550 template) that proves worker output equals local moe() output. Land it upstream (primary path); maintain a fork (explicit fallback). The deliverable is therefore "fix a latent correctness bug in a stalled PR + ship the test that would have caught it" — a stronger contribution than a bare unblock. Note the realistic upstream outcome is "removes the correctness blocker; PR kept open and positioned first-of-the-trio for when the maintainer's breadth phase opens" (see "Institutional context" below — the maintainer deferred JustVugg#550 on priority even though it ships the gate).

Institutional context — why JustVugg#551 was actually shelved (full discussion, read 2026-08-14)

The "discussion" on JustVugg#551 is thin: the PR body plus exactly one comment (the maintainer's blocker). There are no reviews and no inline review comments on JustVugg#551. The real history lives on the parent PR JustVugg#380 and on the template JustVugg#550.

Maintainer merge patterns (from a PR-postmortem pass, 2026-08-14)

A review of the maintainer's merged vs. rejected/deferred PRs shows a consistent, predictable bar. Rules relevant to this work:

User Stories

  1. As a fleet operator, I want to distribute routed-expert FFN computation across multiple machines, so that a frontier MoE model runs faster than it does streaming every expert from one box's disk.
  2. As a fleet operator, I want the coordinator to keep attention, routing, KV cache, and the dense layers local, so that the fast box (eli-itx with its RTX 3080) serves attention at full GPU speed while workers absorb the bulk expert compute.
  3. As a fleet operator, I want workers to serve routed experts from wherever the expert weights live (Optane / NVMe / RAM), so that the R740's 1.5 TB of Optane becomes a ~74 GB/s expert store instead of a ~7 GB/s NVMe one.
  4. As a fleet operator, I want the distributed path env-gated off by default, so that single-machine inference is byte-for-byte unchanged unless I explicitly enable the cluster.
  5. As a fleet operator, I want a token-exact guarantee that the worker path produces identical output to local inference, so that I can trust distributed runs not to silently corrupt results.
  6. As a contributor, I want the parity test to catch the missing fmt=6 input rotation, so that the E8/IQ3 correctness bug is fixed and demonstrated rather than merely asserted.
  7. As an upstream maintainer, I want a worker-vs-local token-exact gate in CI, so that I can merge a re-implemented core kernel on trust rather than hope.
  8. As a fleet operator, I want the transport layer to be an interface (TCP today), so that faster transports (2.5GbE, USB, etc.) can slot in later without a rewrite.
  9. As a contributor, I want to rebase feat: add distributed expert workers JustVugg/colibri#551 onto current dev with minimal conflict resolution, so that the PR tracks a fast-moving upstream rather than rotting further.
  10. As a fleet operator, I want to measure distributed vs single-box throughput by controlling which experts/layers live where, so that I can determine whether distribution is "worth it" without a full tuning effort.
  11. As a contributor, I want the E8/IQ3 (fmt=6) and int4 (fmt=4) quantized expert paths both covered, so that the gate exercises the formats the worker actually handles.
  12. As a fleet operator, I want the shared expert to stay on the coordinator, so that the parity assertion scopes only to the routed experts and does not compare mixed contributions.
  13. As a contributor, I want the COLIEX01 wire framing reused as the shared data-plane contract, so that the native worker path and any future browser/WebGPU path share one auditable protocol.
  14. As a fleet operator, I want the MVP to run eli-itx as coordinator with the BC-250s as workers, so that the feature is validated on hardware I already own before the R740 exists.
  15. As a contributor, I want a byte-level C wire-protocol test using synthetic tensors, so that the framing is proven correct without needing a model.
  16. As a fleet operator, I want a tiny deterministic oracle model as the test fixture, so that the token-exact gate runs in seconds and is reproducible in CI without 744 GB of weights.

Implementation Decisions

  • Rebase, don't re-implement. A test merge of pr551 onto upstream/dev produced exactly one conflicted file: c/coli (two hunks), and the auto-merged c/colibri.c compiles clean (gcc -fsyntax-only → exit 0). Commit distance is exactly 762 (e4b5bf3d..upstream/dev). Note on the why (corrected from first draft): the "easy" outcome is real but not because the FFN primitives are byte-identical. expert_gate_up and siluf are byte-identical; e8_rot_rows is also byte-identical but lives in c/quant.h, not c/colibri.c. matmul_qt is byte-identical only because it is a one-line shim calling matmul_qt_ex; the real primitive matmul_qt_ex has diverged between pr551 and dev (dev adds an w->fmt==8 fp8 passthrough, CUDA S-threshold retry vs pr551's permanent disable, a Metal g_moe_exact gate, and a gs argument to coli_metal_gemm/coli_cuda_matmul). There are no "benign shim renames": st_init and st_init_multi both exist in both branches (c/st.h); only the call site changed (dev passes COLI_MODEL_DIRS via st_init_multi). The merge is clean because the intervening 762 commits went textually around the expert-FFN core without overlapping the worker's added lines — not because nothing changed underneath. The worker's FFN call sequence must still be reconciled with the rebased matmul_qt_ex (see resolved Q1/Q3).
  • Fix the fmt=6 input pre-rotation by eliminating the parallel copy, not patching it. moe() rotates the gate/up input for fmt=6 experts (via the E8_XE macro) before expert_gate_up, then rotates the intermediate before the down matmul — two rotations total. cluster_worker_run does only the second, feeding raw x to expert_gate_up. The maintainer's objection is that the worker re-implements the four-step FFN (expert_gate_up → siluf → e8_rot_rows → matmul_qt) as a parallel copy of moe()'s inner compute — which is exactly why it drifted. So: extract that four-step sequence into ONE shared helper (consistent with the existing coli_cuda_expert_mlp naming on the GPU side) called by both moe()'s CPU path and the worker, and fix the worker to apply the shared E8_XE input rotation before it. This reduces duplication in upstream's own code (the sequence is already inlined ~6× inside moe()), keeps the per-layer rotation caching that fmt=5: a deployable E8-lattice grouped container — int3 that beats shipped int4 quality at 25% fewer bytes (design + plan) JustVugg/colibri#452 protects, and makes the drift structurally impossible — directly answering the maintainer's "re-implemented core kernel" concern rather than fixing one symptom.
  • Prefactor first (ticket Extract a shared routed-expert FFN helper (pure refactor, byte-identical) #9). The shared-helper extraction is a pure refactor of moe()'s six inlined FFN copies — byte-identical, one file, no new knobs — and is split into its own ticket, done first, and submitted upstream as a standalone PR. Rationale: it removes the maintainer's "re-implemented core kernel" objection at the root, shrinks the feat: add distributed expert workers JustVugg/colibri#551 diff, and is the highest-odds-of-merge slice (depth cleanup, not breadth feature). Two of the six copies apply the fmt=6 down-rotation and four don't (Vulkan/CUDA/CPU fallback paths) — the extraction settles which is canonical.
  • Coordinator/worker split (as feat: add distributed expert workers JustVugg/colibri#551 already designs): coordinator = attention, routing, KV, dense, shared expert; worker = routed expert FFN only. The worker never routes on its own — it receives pre-routed batches. Correction: the wire carries (expert_id, nr, raw input rows) only; the routing weights never cross the wire — the coordinator applies them locally to the worker's returned rows. So the split is: worker computes the unweighted per-expert FFN; the coordinator owns the weights (and thus any weighting divergence is impossible by construction).
  • Wire contract = COLIEX01 framing (COLI_CLUSTER_MAGIC "COLIEX01", pr551:c/colibri.c:1686), which originated in feat: add distributed expert workers JustVugg/colibri#551 (PR feat: add WebGPU expert workers JustVugg/colibri#552, which copied it, was closed without merging — so there is no overlap in dev). Reuse it as-is.
  • Transport = an interface. TCP data plane + cluster.py HTTP registry for discovery. Future transports are out of scope but not foreclosed — the worker registry and wire framing already abstract the transport.
  • Env-gated off (CLUSTER_WORKERS / --cluster-coordinator; CLI: coli cluster coordinator / coli cluster worker, coordinator discovery via coli serve); the single-machine path is untouched unless enabled.
  • Scope: native TCP expert workers only. Dense MLP sharding (feat: add dense MLP activation sharding JustVugg/colibri#550) and WebGPU workers (feat: add WebGPU expert workers JustVugg/colibri#552) are separate slices, not this spec.

Testing Decisions

  • Two-layer gate, mirroring feat: add dense MLP activation sharding JustVugg/colibri#550's template:
    1. C wire test (pattern: feat: add dense MLP activation sharding JustVugg/colibri#550's c/tests/test_dense_protocol.c) — #include "../colibri.c", a socketpair, synthetic floats, assert byte-exact framing. No model.
    2. Python token-exact test (pattern: feat: add dense MLP activation sharding JustVugg/colibri#550's c/tests/test_dense_sharding.py) — the glm_tiny oracle, teacher-forcing prefill mode (deterministic: TEMP=0, DRAFT=0), run twice (local vs delegated), asserting identical token signatures. Correction: feat: add dense MLP activation sharding JustVugg/colibri#550's test asserts baseline ≡ delegated equality of the mismatch lines (it does not assert zero mismatches). Our gate must be stronger to satisfy the maintainer's "token-exact worker-vs-local" bar: assert both that the two runs are identical and that there are zero [ORACLE] mismatch lines.
  • Fixture: glm_tiny (generated by c/tools/make_glm_oracle.py) already routes (8 routed experts, top-2, 1 shared), so it exercises the routed-expert path. Correction: the default oracle emits float32 expert weights (the "bf16" docstring is stale; no dtype cast in code), which expert_load reads natively via its runtime-quantization fallback — so the default oracle runs without conversion, but a fmt=6 (or fmt=4) conversion is required to actually catch the bug (a pure-f32 fixture never exercises the fmt=6 rotation). Produce fmt=6 expert weights via c/tools/iq3_pack.py (fmt=6 only) or c/tools/convert_fp8_to_int4.py's e8 path (emits fmt=4/5/6). The --fp8 oracle is not directly readable as-is (it writes <name>_scale_inv while the engine expects <name>.qs block scales).
  • Assertion scopes to routed experts only (the shared expert stays on the coordinator).
  • Determinism/pinning (new, from resolved Q4–Q6): the gate must pin worker and coordinator to identical numeric paths — same ARCH build flag, CPU-only backends, and a fixed numeric-env block (see resolved questions below). Routing itself is already lossless/deterministic (ids from uniq[], weights coordinator-side, inputs raw memcpy).
  • Prior art: feat: add dense MLP activation sharding JustVugg/colibri#550's c/tests/test_dense_protocol.c + c/tests/test_dense_sharding.py; feat: add distributed expert workers JustVugg/colibri#551's existing c/tests/test_cluster.py (HTTP registry only — this is the gap being closed).

Out of Scope

Resolved Questions (answers from the 2026-08-14 research pass)

  1. Format coverage for the gate. The input pre-rotation exists only for fmt==6 — fmt=4 (int4) and fmt=8 (fp8) have no activation transform anywhere. Therefore fmt=6 is mandatory (it is the only rotation-bearing format and the only one that exercises the bug). Add fmt=4 as a control arm: it exercises the worker's expert_gate_up fmt=4 pair-fusion path with no rotation, so a fmt=4-pass/fmt=6-fail result isolates the divergence to the rotation itself (not the socket protocol or the FFN math). Defer fmt=8 (see Q3).
  2. Routing capture mechanism. Confirmed lossless and deterministic, end-to-end. Routing/top-k runs only in the coordinator's moe() (pr551:c/colibri.c:2896), which fills idxs[]/ws[]/keff[] and a distinct-expert uniq[]. cluster_moe_batch shards uniq[] across workers by (eid+layer)%g_cluster_n; cluster_item copies raw input rows via memcpy; the wire carries (eid, nr, inputs) and the coordinator applies ws[] to returned rows locally. Top-k tie-breaking is deterministic (strict > over ascending expert id) and irrelevant to parity (the worker never re-routes). No float weights are serialized, so there is no lossy round-trip. The gate's pass/fail is attributable solely to the worker's expert-FFN arithmetic.
  3. fmt=8 (fp8) coverage. fmt=8 did not exist when feat: add distributed expert workers JustVugg/colibri#551 branched (e4b5bf3d has zero fp8 references; it landed later upstream, renumbered 100→7→8 across Feature request: fp8-e4m3-b128 passthrough weight format — zero loss at int8 cost (repack, not requantize) JustVugg/colibri#524/Vulkan: fmt=7 MXFP4 decode + Kimi K3 GPU tier (follow-up to #418) JustVugg/colibri#705). The worker has no fmt=8 path, and qt_resolve_fmt (pr551) misclassifies an fp8 tensor as int8 and then fails the scale-count check with a hard exit(1) — fail-hard, not silent corruption. Out of scope for pass 1; no guard strictly needed (nothing silently wrong). Optionally add a cheap protocol-version/format rejection so a future fp8 snapshot fails cleanly.
  4. Arch-flag parity. Real. matmul_e8's AVX2 arm uses two interleaved accumulators + tree reduction vs the scalar single-accumulator path, and the selector is a compile-time #ifdef __AVX2__ with no runtime dispatch or env override (g_no_fused_pair/g_idot/g_i4s do not control matmul_e8). Build worker and coordinator with the same ARCH; recommend ARCH=x86-64-v3 (portable AVX2) over native.
  5. CPU vs GPU parity. Real risk. The worker is always CPU (it returns from main() at pr551:c/colibri.c:6663 before CUDA/Metal/Vulkan init). The coordinator can serve routed experts on GPU (Vulkan coli_vk_expert_group_issue, Metal skip-CPU, CUDA coli_cuda_expert_mlp). Force CPU: build the pure-C binary and pin COLI_CUDA=0 COLI_METAL=0 COLI_VULKAN=0 as defense-in-depth (the init blocks are env-gated).
  6. Env parity. Confirmed, and broader than first flagged. FP-affecting env vars that must be pinned identically on both sides (minimum set): IDOT=0 I4S=1 COLI_NO_FUSED_PAIR=1 SPEC_PIN=0 XEXP=0 I4_ACC512=0 I3_AVX512=0 DRAFT=0 (plus the three GPU vars from Q5). IDOT=0 forces the exact-f32 int4/int8 kernels and sidesteps the g_i4s/spec_pinned gate; COLI_NO_FUSED_PAIR=1 makes the worker's and coordinator's gate+up paths structurally identical. The worker does honor these (all reads precede its early-return).
  7. Worker init. The hand-rolled load_cfg + st_init + ebits/dbits is sufficient for the expert-FFN path — expert_load_impl consumes only m->c, m->ebits, m->S, and the passed ESlot (worker uses demand=0, so disk-class counters are skipped). One change required: swap st_init → st_init_multi(COLI_MODEL_DIRS) so a worker matches a sharded coordinator's tensor namespace. Do not port the rest of model_init (tokenizer/KVCache/router/embed/MTP) — dead weight for an expert-only worker.

References & provenance

Corrections vs. first draft (what the re-check changed)

  1. Removed the claim that the four FFN primitives are "byte-identical between pr551 and dev" and that the only churn is "benign renames (matmul_qt→matmul_qt_ex, st_init→st_init_multi)". Reality: e8_rot_rows is in quant.h; matmul_qt is byte-identical only as a 1-line shim while matmul_qt_ex diverged (fp8/CUDA/Metal); st_init/st_init_multi both pre-exist in both branches (only the call site changed).
  2. Corrected c/coli hunk 2 from "an argparse add" to "pr551's full cluster subparser block".
  3. Corrected the feat: add dense MLP activation sharding JustVugg/colibri#550 template claim: it asserts baseline≡delegated equality of mismatch lines, not "zero [ORACLE] mismatch lines". Our gate asserts both.
  4. Corrected the fixture claim: default oracle is f32, not bf16; expert_load reads f32 (and fp8) natively, so a conversion is required only to catch the fmt=6 bug (fmt=6/fmt=4 expert weights), not to run at all.
  5. Corrected the coordinator/worker split: routing weights do not cross the wire (coordinator applies them locally).
  6. Changed mergeable: false → mergeable: CONFLICTING (GitHub's actual field value).
  7. Replaced the seven "Open questions" with resolved answers (see above).

Local artifacts (for the agent picking this up)

  • Fork: https://github.com/lineape/colibri (this repo; this issue lives on its tracker).
  • Local clone (upstream, for reading + rebasing): /home/eli/dev/_reference-docs-and-repos/colibri — branches pr550 and pr551 already fetched; upstream/dev is current (0f335e5). No persistent merge result exists — the test merge ran in an ephemeral worktree that was removed. The picking-up agent should redo the rebase in a fresh branch. The clone is codegraph-indexed (.codegraph/ at the root) — implementers should run codegraph explore (or the codegraph MCP tool) before grep/read.

Metadata

Metadata

Assignees

No one assigned

    Labels

    specPRD/spec — not directly implementable

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions