Experiment: validate placement and backend residency assumptions

[terrizoaguimor]

Summary

This PR is the local-first Phase 0 and placement-contract work requested before a runtime CNRE admission policy is considered. It combines the offline residency simulator with small fixes that make placement accounting, backend selection, and fallback claims more truthful.

It does not add an external cache service, change routing, add prefetch, or change model outputs by policy. It also does not yet implement a runtime frequency-admission policy.

Why These Changes Are Connected

Several reports describe the same user-visible failure family: the plan says that a GPU/cache tier is available, but the actual execution falls back to CPU/disk, or the computed budget omits a memory term and fails later.

Related reports:

• [Bug]: Vulkan not working on deepseek-v4 (Intel GPU) #887: Intel Vulkan/DeepSeek V4 reports CPU/storage execution instead of GPU use.
• [Bug]: Vulkan Brain Map not reported #892: Vulkan residency is absent from Brain Map/telemetry.
• [Bug]: Metal announces "batched routed experts on GPU" and dispatches nothing — expert su GPU 0 in every configuration #813: Metal advertises routed-expert GPU execution but rejects the published fmt=4 gs64 container.
• [Feature]: METAL Attention support for fmt:4 #585: grouped-int4 Metal attention/backend support remains the broader follow-up.
• [Bug]: RAM_GB is not clamped to the #653-corrected MemAvailable on unified memory; CAP_RAISE then inflates against it and the run is OOM-killed mid-prefill #759 and GB10/unified memory: auto RAM_GB ignores the CUDA expert tier (pre-allocation snapshot) -> OOM-kill #653: RAM/VRAM accounting on unified-memory devices.
• The cap computed by --auto-tier drives the process into the OOM killer #766 and CUDA_EXPERT_GB=auto leaves no prefill scratch; a real prompt permanently disables a device #767: planner/runtime memory projection and prompt-dependent fallback.
• [Bug]: CUDA_EXPERT_GB=auto fills all VRAM on single-GPU; lazy dense uploads then fail 60x and fall back to CPU silently #687: lazy CUDA tensor uploads can disable GPU work after the expert tier consumes the card.
• [Bug]: Possible Performance Regression for GLM5.2 in new 1.5.0 release over 1.4.0 release #856: mixed expert widths caused an observed cache-capacity regression, fixed in plan: price each expert row at its own width, not the container's widest (#856) #869.
• v1.5.0 leaves ~60 GB of RAM unused and loses 13 points of expert hit rate: 18% slower decode than v1.4.0 on a 128 GB single-GPU host #885: a related v1.5 residency under-utilization report remains under investigation.
• [Bug]: Kimi-K3 Can not allocate nearly all free VRAM #848: Kimi Vulkan tier fill was gated by RAM misses, fixed in The VRAM expert tier could only be fed by disk reads #877.
• [Bug]: resource_plan.discover_gpus() is nvidia-smi only — AMD/ROCm cards can never be sized (dev + main) #662: AMD/ROCm discovery is the precedent for backend-neutral accelerator discovery.
• [Bug]: Nvidia GPU's not working with Kimi K3 #783: Kimi CUDA is not implemented and should not be implied by generic GPU flags.

Implemented

Offline simulator

• Strict GLM ROUTE_TRACE parsing and validation.
• First-seen batch unions plus per-row selection multiplicity.
• GLM 64-expert block promotion semantics.
• LRU, LFU/frequency admission, SLRU, and synthetic pin policies.
• Uniform and policy-specific dynamic capacity allocation.
• Held-out category gate: at least 10% mean gain and no category worse than 3%.
• Layer-specific felt-cost sensitivity.
• JSON output retaining specs, allocations, trace sources, gates, and sensitivity.
• 29 focused simulator/backend tests.

Placement accounting

• resource_plan.py now records per-layer expert widths and maximum width.
• Shared working-set sizing uses maximum expert width instead of median width.
• Unified-memory fixtures constrain RAM and VRAM against one physical pool.
• Explicit RAM requests are reported when clamped by a shared GPU pool.
• doctor reports a backend-neutral accelerator.gpu check instead of calling every device CUDA.
• Apple Silicon planning prefers performance-core count when available.

Backend truthfulness

• COLI_VK_DEV selects the primary Vulkan physical-device enumeration index for reproducible multi-GPU/hybrid testing.
• DeepSeek V4's Makefile warns that it has no Vulkan backend instead of implying that VK=1 enables one.
• Metal batched routed MoE accepts grouped int4 and carries the group size into the shader.
• Metal grouped MoE tests cover g64 and g32 ragged execution.
• Metal startup wording now says that device initialization is format-dependent rather than promising universal routed-expert dispatch.

Real-trace evidence

Before this PR, a DigitalOcean H200 pilot collected 13 GLM-5.2 traces: 8 train and 5 held-out across coding, chat, reasoning, multilingual, and long-context prompts.

The simulator predicted the following held-out felt-wait gains against uniform LRU:

Expert budget	Candidate	Mean gain	Worst category
120 GB	Uniform frequency	+39.91%	+34.84%
120 GB	Dynamic frequency	+40.45%	+35.41%
160 GB	Uniform frequency	+53.35%	+49.92%
160 GB	Dynamic frequency	+54.50%	+50.66%
200 GB	Uniform frequency	+65.31%	+62.64%
200 GB	Dynamic frequency	+66.29%	+63.07%

These are offline predictions, not runtime speedup claims. The pilot also showed why backend and overlap telemetry must be correct: RAM_GB=120, PIPE=0 measured 1.66 tok/s at 57.2% hit, while RAM_GB=120, PIPE=1 measured 2.00 tok/s at the same hit rate. Hit rate alone is not the objective.

Verification

• Full Python suite: 407 passed, 57 skipped.
• make metal-test: passed, including grouped-int4 batched MoE.
• Focused planner/doctor/backend tests: passed.
• python3 -m py_compile: passed.
• git diff --check: passed.

Scope and follow-up

The simulator and methodology are intentionally included so the next runtime policy can be measured against the real held-out gate rather than selected on synthetic or in-sample wins.

The next PR should be a separate, small, GLM-only opt-in admission policy with no routing change, no new prefetch, fixed memory budget, and ABBA measurements for tok/s, p50/p99 latency, felt wait, physical bytes, admissions, rejections, evictions, migration bytes, and output identity.

The Intel Vulkan report in #887 still needs hardware confirmation. This PR improves device selection and removes misleading generic CUDA wording, but it does not claim DeepSeek V4 Vulkan support that the engine does not currently implement.

[JustVugg]

Green on all 17, and the individual observations look sound. I am still going to ask you to split it, and the reason is in your own summary:

Several reports describe the same user-visible failure family: the plan says that a GPU/cache tier is available, but the actual execution falls back to CPU/disk.

That several reports share a family is an argument for splitting, not for bundling. #887, #892, #813, #585, #759, #653, #766, #767, #687 are nine issues, on four backends, reported by different people on hardware none of us has. One PR that touches colibri.c, inkling.c, backend_metal.mm, backend_vulkan.c, resource_plan.py and doctor.py and adds a simulator means:

• no reporter can confirm their own issue. @BranBushes cannot tell whether his Intel Vulkan case is fixed without also carrying the Metal changes; @ThefloorMiner cannot A/B the Metal claim without the Vulkan ones. The people who can actually verify are each blocked on code irrelevant to them.
• a revert is all-or-nothing. If one of the nine turns out wrong on hardware we do not have, undoing it takes the other eight with it.
• and 2,060 lines across four backends is more than I can review honestly. I would be approving the ones I can reason about and waving through the rest, which is exactly how [Bug]: Possible Performance Regression for GLM5.2 in new 1.5.0 release over 1.4.0 release #856 and v1.5.0 leaves ~60 GB of RAM unused and loses 13 points of expert hit rate: 18% slower decode than v1.4.0 on a 128 GB single-GPU host #885 got in — both from a single constant nobody enumerated the consumers of.

The natural cut is the one your description already makes:

1. The offline residency simulator (tools/residency_sim.py, its test, docs/experiments/) — self-contained, adds no runtime behaviour, and can go in on its own merits today.
2. Placement/accounting truthfulness (resource_plan.py, doctor.py, docs/ENVIRONMENT.md) — testable here, no hardware needed.
3. Vulkan ([Bug]: Vulkan not working on deepseek-v4 (Intel GPU) #887, [Bug]: Vulkan Brain Map not reported #892) — @BranBushes and @MasterCATZ can confirm. Note fix: report Vulkan expert residency in telemetry and brain map #891 is also touching Vulkan telemetry; worth coordinating so you two are not writing the same thing twice.
4. Metal ([Bug]: Metal announces "batched routed experts on GPU" and dispatches nothing — expert su GPU 0 in every configuration #813, [Feature]: METAL Attention support for fmt:4 #585) — @ThefloorMiner and @RDouglasSharp have the hardware, and Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587 is already green and touching backend_metal.mm.

Send 1 and 2 and I will take them without ceremony. 3 and 4 want their reporters attached.

One correction while you are here, since you cite it: #856 was fixed in #869, but the same constant was also feeding autopin_lru_reserve(), which is #885 — 60 GB of resident set declined on a 128 GB host. Same defect, second consumer, and I did not notice the second one when I fixed the first. If your placement work touches that path, that is the shape to watch for.

[terrizoaguimor]

Per your review, I split PR #893 into two focused PRs from dev:\n\n1. Offline simulator: https://github.com/JustVugg/colibri/pull/902\n2. Placement/accounting: https://github.com/JustVugg/colibri/pull/903\n\nThe simulator PR contains only residency_sim.py, its tests, and the experiment methodology. The placement PR contains only resource_plan.py, doctor.py, their tests, and the environment documentation.\n\nI intentionally left the Vulkan and Metal changes out of both PRs so #887/#892 and #813/#585 can be coordinated with the reporters and existing PRs #891/#587. The original PR remains open temporarily as the superseded bundle; I will close it after the split PRs are visible.

[terrizoaguimor]

Superseded by the focused PRs requested in review: #902 (offline simulator) and #903 (placement/accounting). Backend-specific Vulkan and Metal changes will be proposed separately with the relevant hardware reporters.