Add generated quant-kernel JIT and accelerate Gemma4, BGE-M3, and GLiNER2

[timkaye11]

Summary

This PR adds an evidence-gated GPU kernel pipeline and applies it across Gemma4, BGE-M3, GLiNER2, and core multimodal/reranking inference.

• Adds shared descriptor/IR kernel planning, Metal and CUDA lowering, checked-in artifacts, route telemetry, guarded runtime JIT modes, caching, conformance gates, and safe fallback.
• Expands Gemma4 QAT with generated routes, batching, radix prompt caching, MTP/speculative decoding, and long-context split-GQA support for E2B and E4B.
• Accelerates BGE-M3 and GLiNER2 Metal execution with resident pipelines, fused attention/FFN paths, and correct GLiNER2 relation endpoints.
• Keeps the default CLIP/CLAP Q4_K path lightweight while preserving qualified generated, profiling, and exact-JIT routes.
• Adds antfly inference chat with model pulling, streamed multi-turn generation, context trimming, interruption, slash commands, and per-turn statistics.
• Hardens model lifecycle, admission/eviction, prompt-cache budgeting, streaming cancellation, remote credentials, multimodal processing, and inference SDK/OpenAPI behavior.

Benchmarks

Local, unattested ReleaseFast measurements used a MacBook Air with an Apple M4 8-core GPU and 16 GB RAM. Metal was enabled; CUDA, ONNX, and PJRT were disabled. The same host, model bytes, inputs, and harnesses were used on both sides.

The control is the saved main/base snapshot at 8ad98e6, not current main. Branch measurements were collected on the reviewed candidate immediately before the final main merge. HTTP results generally use three warmups and ten measured requests. Negative latency deltas are faster.

Core models

Workload	Input / output	Branch	Saved main/base	Delta
CLIP text	46 tokens	246.19 ms p50	234.95 ms p50¹	+4.79% latency
CLIP image	1 JPEG	219.31 ms p50	208.75 ms p50	+5.06% latency
CLAP audio	1 WAV	240.56 ms p50	239.26 ms p50	+0.54% latency
GLiNER2 entities	430 tokens	227.91 ms / 1,886.7 input tok/s	657.83 ms / 653.7 input tok/s	-65.35% latency; 2.89x throughput
GLiNER2 entities + relations	430 tokens	529.14 ms; 24 entities, 4 relations	HTTP 500	Functionality fixed
mxbai-rerank-base-v1	512 tokens	531.29 ms / 963.7 input tok/s	504.48 ms / 1,014.9 input tok/s	+5.32% latency
Florence2 caption	51 output tokens	9,952.03 ms / 5.125 tok/s	10,468.45 ms / 4.872 tok/s	-4.93% latency
Gemma4 E2B TTFT	470 prompt tokens	1,265.5 ms	1,335.5 ms	-5.24%
Gemma4 E2B decode	64 output tokens	38.187 tok/s	34.006 tok/s	+12.29% throughput
Gemma4 E2B generation	470 + 64 tokens	2,937.5 ms	3,216.5 ms	-8.67% latency

¹ Main's ten CLIP-text samples produced five response hashes. The value shown is the median of the three samples matching the branch's deterministic, byte-exact output. CLIP/CLAP values are repeat medians; the reranker value is the median of two cooled isolated runs.

Gemma4 E2B long context

The same raw prompt was repeated to approximately 2K tokens with a forced 300-token completion. The branch tokenizer produced 2,065 prompt tokens versus 2,048 on the control, so normalized prefill throughput is included. Branch used one warmup and five measurements; control used three warmups and ten.

Metric	Branch	Saved main/base	Delta
TTFT / prefill	9,143 ms	15,526.5 ms	-41.11% latency
Prefill throughput	225.86 tok/s	131.90 tok/s	+71.23%
Decode throughput	48.00 tok/s	28.864 tok/s	+66.30%
Generation latency	15,394 ms	25,913.5 ms	-40.59%
Fresh-process total	15,987 ms	26,787.5 ms	-40.32%

Both revisions generated exactly 300 tokens and were deterministic within each revision. Token IDs differ because of the known tokenizer-path difference, so this is performance evidence rather than cross-revision token parity.

Additional qualification results

These earlier branch results were not rerun during the final M4 sweep.

Workload	Antfly	Comparator	Result
Gemma4 E4B Metal, 256-token decode	24.87 tok/s plain; 31.27 tok/s MTP k=1	llama.cpp b8990: 28.69 tok/s	MTP +25.7% vs plain; +9.0% vs llama.cpp
Gemma4 E2B CUDA, L4, 2,051+300	740.8 ms TTFT; 110.2 tok/s	llama.cpp: 329.5 ms; 114.4 tok/s	Decode within 4%; total-latency ratio 1.173
BGE-M3 Metal, real-text batch 16	12.52 embeddings/s, ~857 input tok/s	Native CPU was 27x slower at batch 1	Cosine 0.9973 vs llama.cpp

GLiNER2 shape	Metal	Native CPU	Fastino MPS
215 tokens, batch 16	1,248.8 ms	1,662.6 ms	1,240.8 ms
430 tokens, batch 16	2,375.3 ms	3,389.7 ms	3,384.3 ms

GLiNER2 Metal is approximately 6.9x faster than its pre-optimization batch-16 baseline and 33-43% faster than native CPU at these shapes.

Correctness and validation

• CLIP/CLAP return deterministic 512-dimensional embeddings; reranker ordering and its relevant score (0.998552143573761) match main exactly; Florence2 caption text is exact.
• GLiNER2 matches all 23 entity texts, labels, and spans from main with maximum confidence drift 0.0003637075; all four relation endpoints resolve correctly.
• A 2,003-token Gemma4 gate produced exact target/live/MTP agreement for 64 generated tokens and natural-EOS agreement at 128 tokens.
• Full Metal inference suite: 3,060 selected; 3,042 passed; 18 skipped; zero failures.
• Generated-source freshness passed for 67 files plus Metal/CUDA runtime regions; compiler readiness passed 103 compiler, 17 Metal-renderer, and 14 CUDA-policy tests.
• Metal route-all passed 50/50 routes; production regression passed 14/14 with zero slow fallbacks; split-GQA passed 360 policy and 28 device cases.
• GLiNER2 pipeline tests passed 16/16, and the production sweep passed CLIP/CLAP, GLiNER2 entities/relations, reranker, Florence2, and Gemma4.
• CUDA source/artifact policy passed in the final Mac review. CUDA runtime evidence comes from the earlier paired L4 qualification because CUDA execution was unavailable locally.

The final review found no unresolved P1/P2 defect in the reviewed changes.

[ajroetker]

Seems like we probably want this to be an e2e test or not in ci

[dovinmu]

1. BGE-M3 managed-index batching — confirmed product blocker. Default enrichment batching (8) repeatedly hits ResourceLimitExceeded and makes no backfill progress. ANTFLY_ENRICHMENT_EMBED_BATCH_ITEMS=1 works, but requiring that override isn’t acceptable.

2. Model-pull receipt incompatibility — confirmed CI blocker. The downloader now writes receipt version 2 (.repos/antfly-pr372-5b1c04d/zig/pkg/inference/src/registry/download.zig:1640), while the E2E model helper accepts only version 1 (.repos/antfly-pr372-5b1c04d/zig/e2e/inference/models.py:395). Downloads complete successfully, but the harness rejects them, producing 77 inference failures. This looks mechanical to fix, but it is definitely blocking the required check.

3. Managed-runtime recovery — probable product blocker. BGE-M3 backfills also emitted MissingWeight followed by Canceled, with the index reporting runtime_unavailable and repair_degraded. It eventually became healthy after restart/resume, but we did not prove same-process recovery. I would require a clean no-restart reproduction before merging. Evidence is in results/campaigns/2026-08-04-pr372-ohr-bge-m3/antfly-attempt-2.log:19.