feat: distributed expert workers — rebased, fmt=6 rotation fixed, token-exact parity gate

[lineape]

Summary

This revives upstream #551 (distributed expert workers: coli cluster coordinator / coli cluster worker, the COLIEX01 wire framing, CLUSTER_WORKERS env gate) and makes it shippable. On top of the rebase it:

1. Fixes a latent correctness bug in the original feat: add distributed expert workers #551. cluster_worker_run re-implemented the expert FFN as a parallel copy of moe() and was missing the fmt=6 (E8/IQ3) gate/up input pre-rotation, so it produced wrong numbers for E8/IQ3-quantized experts. The worker now calls the shared expert_ffn helper (refactor: extract shared routed-expert FFN helper (byte-identical) #1035) and applies the same input rotation moe() does — the parallel copy is gone, so the drift that caused the bug is structurally impossible.
2. Fixes a feature-killing bug in the original feat: add distributed expert workers #551. cluster_init's connect bookkeeping was inverted, so the coordinator always exited "no expert workers reachable" even with a live worker.
3. Adds the two-layer token-exact parity gate, mirroring the feat: add dense MLP activation sharding #550 discipline.
4. Hardens the registry and restores cross-host operation. The registry subclasses openai_server's handler (inheriting the DNS-rebinding guard and connection caps), and exposes the existing --allowed-host escape hatch on coli cluster coordinator so a cross-host worker can register.

Correctness gate

Byte-for-byte worker-vs-local equality, proven token-exact. c/tests/test_cluster_sharding.py runs glm_tiny in teacher-forcing prefill twice — local CPU vs cluster-delegated — and asserts zero [ORACLE] mismatch lines in both runs and identical token signatures. It covers fmt=6 (E8/IQ3, the rotation-bearing format — the arm that fails pre-fix) and fmt=4 (int4-grouped, the no-rotation control). Determinism is pinned: same binary/ARCH for both sides, CPU-only (COLI_CUDA=COLI_METAL=COLI_VULKAN=0), and a fixed numeric-env block (IDOT=0 I4S=1 COLI_NO_FUSED_PAIR=1 SPEC_PIN=0 XEXP=0 I4_ACC512=0 I3_AVX512=0 DRAFT=0). The gate demonstrably fails on the pre-fix worker (fmt=6 delegated drops to 0/32 with 32 mismatches) and passes post-fix (32/32, zero mismatch). It skips cleanly when the gitignored glm_tiny fixture is absent, and is wired into CI (cluster-parity job regenerates the fixtures then runs the gate). A no-model C test (test_cluster_protocol.c) asserts byte-exact COLIEX01 framing in both directions.

The oracle was run twice (local + delegated, union of both format arms). The default path is unchanged unless CLUSTER_WORKERS is set — the cluster path is env-gated OFF and the single-machine build is behaviourally identical.

RAM budget / expert cache interaction (#403)

Distribution does not double-count the RAM budget. The coordinator's cap_for_ram enforces the budget against its own measured RSS; a worker is a separate process holding its own resident expert set, and its resident bytes never enter the coordinator's accounting. The worker's cache is deliberately minimal — one disk-backed slot per layer (demand=0) — so it holds no resident experts beyond what it is actively computing. The one operator-visible consequence is that fleet-wide RSS is the sum of per-box budgets (coordinator box plus each worker box), each enforced independently; there is no cross-machine accounting today, which is fine for the MVP but worth a follow-up if workers are ever co-located on the coordinator's box.

Validation

• make → 0 warnings; make check → C suite ALL PASS (0 failures), Python Ran 473 tests OK.
• Parity gate: fmt6 32/32, fmt4 32/32, zero [ORACLE] mismatch (and demonstrably fails pre-fix).
• test_cluster_protocol falsifiability-verified (mutating a framing field fails).

Compatibility

• Env-gated OFF by default (CLUSTER_WORKERS); single-machine path unchanged.
• Worker honors COLI_MODEL_DIRS via st_init_multi (matches a sharded coordinator).
• Unresolvable expert tensors surface as a clean worker-side error, not a mid-connection fatal.
• Cross-host workers register via --allowed-host (same escape hatch as coli serve); the registry defaults to loopback + bind address. The raw TCP worker binds 0.0.0.0 with no auth, matching the trusted-LAN assumption of the data plane (transport is an interface; auth is a follow-up).

Note

This PR depends on the helper refactor in #1035 (review that one first). CI may sit in action_required for a first-time contributor — would appreciate the workflow run being approved.

[lineape]

Just to add some context beyond the automated gate: I also ran this through a real multi-machine integration test, not just loopback.

The setup was one coordinator and up to three expert workers, all separate physical Linux boxes, connected over a WireGuard mesh (Tailscale). Two of the workers were on a different tailnet from the coordinator, so the cross-tailnet path was exercised too. One worker was a hosted VPS with over 200 ms of round-trip latency. Everything CPU-only, no GPU.

Results:

• Correctness: token-exact (32/32 vs the oracle) for fmt=6 and fmt=4, at 1, 2, and 3 workers, on every host — zero [ORACLE] mismatches across the board.
• Sharding: with 3 workers the routed experts split across the three TCP sessions by (eid + layer) % n and the coordinator reassembled the weighted sum correctly — the first time that path has been exercised at n > 1.
• Latency behaviour: as expected, decode is round-trip-bound (~1 RTT per sparse layer per token), and the 200 ms VPS worker landed right where the RTT predicted. Worker count doesn't speed up single-token decode (each token's experts land on one shard owner), but the differential path is what the parity gate protects, and it held everywhere.

One observation worth flagging for the maintainer, not blocking: the "expert hit %" telemetry always reports 0.0% under the cluster path — cluster_moe_batch bypasses the local pin/LRU counters, so the hit-rate stat silently doesn't cover cluster-delegated experts. The worker's own cache is fine; it's just an unreported number.

Happy to provide more detail or reproduce anything on request.

[JustVugg]

Heads-up, and an apology: this went dirty because of us, not you.

#1063 landed on dev — it makes model families registry-owned, so c/coli, c/openai_server.py, c/doctor.py and c/resource_plan.py now all read one descriptor table (c/family_registry.py) instead of each carrying its own branches. That was a deliberate structural change agreed in Discussion #1057, and it rewrote exactly the files your branch touches.

A rebase on current dev should be mechanical — the conflicts will be in the family/dispatch branches that no longer exist, and the replacement is usually "read it from the registry" rather than "add another branch". If a conflict isn't obvious, say so on the PR and I'll work through it with you rather than leave you guessing at the new contract.

Two things worth knowing while you're in there:

• Unknown model_type is now refused explicitly instead of silently falling through to the GLM engine. If your change relied on that fallback anywhere, it won't behave the same.
• resource_plan.py now refuses to plan for families without a measured adapter (Kimi, OLMoE, Inkling, V4) rather than returning a plausible-looking zero. That was the bug that hit two model PRs independently in the same day.

Sorry for the churn. Landing the registry before your branches was the right call for the project, but it does mean the cost of it fell on the people with open work.

[lineape]

No apology needed — this is exactly the churn a registry refactor is supposed to cause, and the new shape actually made the rebase easier. Landing #1063 first was the right call.

The branch is rebased onto current dev (including #1067 — the first CI run here tripped over the same OLMoE/numpy loader crash your dev CI hit, so I picked up your fix) and the conflict flag has cleared. It went about as you predicted: all conflicts were in c/coli, and every one resolved to "read it from the registry":

• need_model / need_worker_model now take their build target from resolve_model(...).descriptor.build_target instead of this branch's old per-arch dispatch table, keeping the shared require_model helper.
• The cluster subcommand dispatch sits inside the new try/except (FamilyConfigError, UnknownFamilyError) handler in main(), so cluster commands refuse unknown families explicitly like everything else.

One resolution was a judgment call rather than mechanical, so flagging it per your offer: need_worker_model now resolves the model's family and explicitly refuses non-GLM models ("cluster expert workers currently support the GLM engine only"). The old code hardcoded the GLM engine path, which for a non-GLM model is exactly the silent-fallthrough-to-GLM the registry now bans — and the worker loop only exists in colibri.c. If you'd rather express that as a capability flag on FamilyDescriptor instead of an id check, happy to rework it to whatever fits the registry's contract.

Beyond the 13 rebased commits there are two small fixes that came out of post-rebase review:

• fix: check the model directory before resolving its family — my first pass at the need_model resolution resolved the engine (which reads config.json) before the directory check, so a missing model dir said "unsupported model" instead of "model not found"; test_cli_output caught it.
• build: install cluster.py; complete test_cluster_protocol prerequisites — cluster.py was missing from the install target (cluster commands would ImportError on an installed tree, though checkouts were fine), and the wire test includes ../colibri.c but only depended on compat.h, so it wouldn't rebuild when the engine changed.

Nothing load-bearing moved: the c/colibri.c feature hunks are byte-identical to the pre-rebase branch (verified by diffing the two diffs, not by eye), the worker still routes through the shared expert_ffn with the fmt=6 input pre-rotation intact, and locally everything is green — make check passes, the parity gate is token-exact 32/32 on both fmt=6 and fmt=4, and the COLIEX01 protocol and registry tests pass.