CPU-5/6: an integer projection path that is faster and NOT quality-validated - #315
Open
chrishayuk wants to merge 5 commits into
Open
CPU-5/6: an integer projection path that is faster and NOT quality-validated#315chrishayuk wants to merge 5 commits into
chrishayuk wants to merge 5 commits into
Conversation
shipped Q8 x F32 348 ms/token 2.87 tok/s KL 0.000158
candidate Q8 x asym-Q8[16] 264 ms/token 3.79 tok/s KL 0.000283
1.32x faster, and it clears every frozen quality gate with 10.7%
headroom on the one that binds:
G1 KL mean 0.000283 <= 0.000316 ok
G2 KL p99 0.001701 <= 0.009666 ok
G3 top-1 99.5977% >= 99.00% ok (shipped is 99.5977%)
G4 flips at BF16 margin >= 0.10: 0 ok
CPU-4Y priced Q4 x Q8 at 135 ms and stopped, deliberately, before asking
whether the arithmetic preserves the model. It does not. This rung asks,
and the answer redirected the whole programme.
## Blanket Q4 is REJECTED, and no exception set rescues it
arm KL mean KL p99 top-1 flips>=.10
shipped 0.000158 0.00097 99.60% 0 PASS
bf16xq8b 0.000610 0.00324 99.02% 0 FAIL
q8xq8b 0.000742 0.00487 98.91% 0 FAIL
q4xq8b 0.053840 0.35731 90.98% 38 REJECT
Q4 breaks the margin gate decisively: 38 flips where the reference
separated its top two by >= 0.10. Subtracting the activation floor gives
a Q4 weight cost of 0.05323 against Q8's 0.00016 -- 332.7x, against a
quantisation step ratio of 18.1x whose SQUARE is 329x. Logit KL scales as
the step squared, to within 1%.
That turns the exception search into arithmetic. R3 measured it: restoring
the FFN (~70% of bytes) removed only 45% of the damage, so sensitivity is
non-uniform and concentrated in attention+head at 1.83x the per-byte rate
-- the opposite knee from PR #299's Granite FFN. Class-level rescue is
decisively implausible under the measured bracket: the cheapest split
(head only, 9% of bytes) is still ~27x the gate and saves 4.5% of traffic.
## The binding constraint was the ACTIVATION, not the weights
`bf16xq8b` holds the checkpoint's own weights and still fails by 1.9x, so
every integer arm inherits that floor. A programme that had gone looking
for a better 4-bit code first would have optimised the wrong operand.
Per-tensor int8 is not a usable activation representation: the residual
stream's peak/rms is 28-36 at depth, leaving ~2 effective bits.
per-tensor Q8 activation, exact weights rel_rms 0.476
per-block-64 0.047
block 16 + asymmetric KL 0.000287, all gates PASS
## And the kernel ladder, which cost three falsified hypotheses
K1 scalar precomputed sums SLOWER 868 vs 757
K2 batch integer reductions +20% bit-identical
K3 stay vector through floats 1.73x sym16 484 -> 279
K4 vector precomputed sums ~0% index consumed, time unmoved
K5 delete transient buffers 1.95x asym16 516 -> 264
K1 and K4 both removed ARITHMETIC and changed nothing, because the loop
was never ALU-bound. The cost was per-row folded scale buffers: ~320
vector load/store ops against ~320 SDOTs, doubling the inner loop. Delete
them and the asymmetric correction drops from a 230 ms tax to ~1 ms.
K2/K3/K4/K5 are all gated on bit-identity except K3, which reassociates
by design; the quality gates are re-established on the full bank for the
arithmetic that actually runs.
## Method
quality-bank-1 is discovery (plus a 20% screen, selected by rule).
quality-bank-2 is frozen, verified disjoint, and untouched -- one shot,
on the final candidate only. Gates and bands were frozen before any arm
ran; the selection rule before any rescue rung; K3/K4/K5's performance
lines before each was written.
fmt clean, clippy clean on aarch64 and the non-aarch64 path, 3043 tests.
Bank 2's entire value is that no candidate was selected using it, so
every way of accidentally spending it has to be refused by the tool
rather than avoided by care.
`accept.py` runs the ONE selected candidate and refuses unless:
working tree clean
candidate SOURCE (crates/) identical to the freeze SHA
bank identity == quality-bank-2, digests re-derived and unchanged
arithmetic == Q8[64] x asymmetric-Q8[16] -> I32 -> F32
and no stray policy override is set
no existing bank-2 result
The last is the one-shot rule made mechanical: a second run against the
same bank would turn it into a discovery bank retroactively, so the file
existing is itself the refusal.
All of it is stamped into the result, so the acceptance number names the
immutable state it validated instead of resembling it. Bench tooling may
move after the freeze; the candidate may not, which is why the drift
check is scoped to `crates/` rather than to HEAD.
Prompted by a near miss: `run_bank.py` read `prompts.json` from its own
directory, so pointing the output at a bank-2 directory produced a
bank-2-shaped result containing BANK-1 prompts, with no error. Fixed in
the freeze commit by making the bank explicit and self-identifying; this
makes the provenance checked rather than conventional.
…adjudicator
CPU-5 failed Bank 2 and is closed as FAIL. Its defect was that G1 carried
an ABSOLUTE number across prompt sets:
intended candidate degradation <= 2x shipped degradation
implemented candidate KL on bank N <= 2x shipped KL measured on BANK 1
It was not underpowered. Bank 1's paired statistic needed ~10 prompts at
95% and had 69; pairing cuts variance 12x. So CPU-6 repairs the gate, not
the candidate, which is UNCHANGED at df36ca9.
D_p = mean_pos KL(candidate_p||BF16) - 2 * mean_pos KL(shipped_p||BF16)
PASS iff upper95(mean_p D_p) <= 0
by stratified prompt bootstrap preserving each category's frozen count,
seed 20260825, 20,000 resamples. Every bank carries its OWN shipped
anchor, so nothing numerical travels between prompt sets except the 2x
rule. Each bank is adjudicated ALONE; pooling 400 prompts would be one
larger sample rather than two replications.
Banks 3A and 3B: 200 prompts each, frozen SIMULTANEOUSLY as an
invariant — cpu6_freeze.py refuses to freeze either unless both exist and
pass mutual disjointness, against quality-bank-1, quality-bank-2, the
spent SENSITIVITY-1B' calibration, and each other. Demonstrated
non-vacuous: planting a bank-1 text and a 3B text into 3A produces a
refusal.
The adjudicator is written and adversarially self-tested BEFORE any arm
runs, because it is the last remaining degree of freedom. Its controls:
1.5x passes, 2.5x fails, 2.0x WITH SCATTER does not validate (mean D ~ 0,
upper95 > 0), stratification is preserved in every replicate, and an
extreme category is not diluted.
Recorded as a limitation rather than metadata: all 400 prompts are
hand-authored by one agent, so the banks bound execution-side variation,
not authorship bias.
Decision: 3A PASS and 3B PASS validates; either FAIL retires the
representation. No Bank 4. Both banks run regardless, since 3B is
replication evidence even when it cannot rescue the candidate.
…stinct (CPU-6) A single `freeze_sha` conflated three different things. Bank 3 does not validate "the code at HEAD"; it validates the CPU-5 candidate whose IMPLEMENTATION was frozen at df36ca9, under a PROTOCOL frozen separately, executed from some clean HEAD that may be later than both. candidate_source_sha df36ca9 what is being validated protocol_sha where the gate was frozen execution_head what actually ran crates_identical_to_candidate_source The adjudicator checks the candidate identity and prints all three, so a report a year from now says what its verdict rests on without needing this conversation.
The six-arm run was started and cancelled ~20 minutes in to free the machine. No partial arm retained, nothing computed from one, both output directories deleted so a later run cannot mix a stale arm with a fresh one. Everything that could bias the result was already frozen — prompts, protocol, gate, bootstrap seed, adjudicator — so the verdict is exactly as trustworthy whenever it is computed. Parking costs time, not validity. Also records that the original ~15 h estimate was wrong by ~3x: measured 0.65 s/position on a quiet machine puts the six arms at ~4.6 h. The bad estimate came from extrapolating Bank-2 throughput that was contaminated by a duplicate process I had already diagnosed — a plan built from a contaminated number inherits the contamination.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CPU-5 asked whether a faster CPU integer projection path preserves the model. It does not, and the rung that would have settled the surviving candidate is frozen and unrun.
What this contains
Read the headline carefully
df36ca9f's subject says "beats production". That is a SPEED claim and the quality claim behind it did not hold:The candidate arithmetic is quality-unvalidated. It is reachable only through
LARQL_CPU_ARITHMETICand is OFF by default, so nothing in this PR changes production behaviour — but it should not be enabled anywhere on the strength of the commit subject.Bank 2's failure is a defect in the GATE as much as in the representation: G1 was intended as "candidate degradation <= 2x shipped degradation" and implemented as "candidate KL on bank N <= 2x shipped KL measured on BANK 1" — an absolute number carried across prompt sets. Whether the candidate is relatively worse on Bank 2 is unknowable and stays unknowable; Bank 2 is spent and its missing anchor must never be measured now. CPU-6 repairs the gate and re-tests the unchanged candidate; it is ~4.6 h on a quiet machine and has never been run.
Durable findings, independent of whether the candidate ever ships
Known debt
The crate's per-file coverage policy fails on this branch, including files this PR introduces (
integer.rs66%,replay.rs6%,cost.rs87%,arithmetic.rs87%). Local coverage reads lower than CI does on ubuntu, but not by that margin. Not addressed here.Full state:
docs/cpu-execution-roadmap.md. Protocol:bench/prompts/CPU6-VALIDATION.md.