docs(bench): R4.1 frozen, R4.2 closed — GPTQ is affordable on this Mac - #313
Open
chrishayuk wants to merge 6 commits into
Open
docs(bench): R4.1 frozen, R4.2 closed — GPTQ is affordable on this Mac#313chrishayuk wants to merge 6 commits into
chrishayuk wants to merge 6 commits into
Conversation
…oder Every Granite verdict in the SENSITIVITY programme measures the damage nvfp4-nearest-v1 does. None establishes that the damage is a property of the FORMAT. R4 asks whether we have been measuring the model's tolerance or one encoder's choices, which puts it upstream of the whole sensitivity question. One variable only: the encoder recipe. Codec ABI, precision map, kernels, runtime and evaluation are held identical, and the candidate pack must report exactly 2,283,690,080 payload bytes — asserted, not assumed, so no "but it used more bits" explanation is available for a result in either direction. This is the experiment the codec-ABI / encoder-recipe separation was built for. R4 is the first CALIBRATED encoder here, which opens a leakage route that did not previously exist. It reuses the frozen disjoint 12-prompt bank (token digest df0e3644, text digest f628ce47, zero overlap with Q-BANK verified by content) rather than minting a new one, and pack provenance must record which calibration chose the values. Primary arm is uniform R0 only — mixing a protection profile in would confound the encoder change with a precision-map change, which is R5. Metrics are pre-committed as the WHOLE response: KL mean/p50/p95/p99/max, dNLL, flips, high-margin flips, per-category breakdown, payload bytes. No quality threshold is invented, because any would be arbitrary or fitted. If GPTQ improves the mean and worsens the tail, that is the finding and it is reported as such. The per-category table is the leakage control: an encoder fitted to twelve prompts that only improves regimes resembling them would show it there. Second question, run ONLY if the first arm wins materially: does the late-FFN knee survive a better encoder? If R0-GPTQ lands near nearest's late5-ffn and further protection then buys little, much of what looked like late-layer sensitivity was nearest rounding making bad choices in those layers. That would reinterpret the sensitivity programme without invalidating it — every banked measurement stays true of nvfp4-nearest-v1. Records an implementation fact established before committing: GPTQ needs the full input second-moment matrix E[x x^T] per site, not the per-feature diagonal the SENSITIVITY capture banked. Those are different objects. The down_proj site is 8192-wide, so its Hessians run ~268 MB per layer and ~10.7 GB across the stack if held at once — the most expensive Hessian belongs to the operand the sensitivity programme found most interesting.
…ciency gate added Three corrections and one addition, all before any encoder or result exists. RANK CLAIM CORRECTED. An earlier analysis claimed that with n=458 and d=8192 only 5.6% of directions are "covered", so GPTQ degenerates to round-to-nearest elsewhere. That is FALSE. H_lambda = X^T X + lambda I is full rank after damping, and H^-1 = lambda^-1 I minus a DENSE low-rank correction, so the sequential update couples essentially every column pair. Verified numerically at n=20, d=100: H^-1 is 100% dense off-diagonal and every row of Cholesky(H^-1) reaches every later column. The rank fact is true; that quality interpretation is not, and it must not appear in the R4 record. HESSIAN STRATEGY SETTLED: dense H, one site at a time. Peak is 256 MB (128 MB triangular) for the 8192-wide down_proj site; the 10.7 GB figure only ever applied to holding all 40 layers at once. Woodbury is rejected on the merits rather than on memory — GPTQ needs the Cholesky of H^-1, and the Cholesky of (lambda I + low-rank) has no compact low-rank form. A block-diagonal H would change the algorithm and, if ever wanted, gets its own recipe (nvfp4-gptq-block-v1) and its own test. SEQUENTIAL CANDIDATE-PATH CALIBRATION frozen as part of the algorithm, not as an implementation detail: layer L+1 is calibrated from activations produced after layers 0..L have already been encoded by the candidate. The encoder therefore fits activations the candidate will actually produce. CALIBRATION SUFFICIENCY GATE added. The frozen 12-prompt SENSITIVITY bank was sized for a per-feature DIAGONAL, where 458 samples are ample; GPTQ estimates a full d x d correlation, a different object with no evidence of convergence at 458. The fix is not "make n >= d" — full empirical rank is not the scientific requirement. Instead: a newly written disjoint pool with deterministic nested prefixes (458 / 2048 / 8192 / 32768 / 65536), a separate held-out calibration-validation partition, and N chosen by held-out reconstruction error congruent with GPTQ's own objective. NO Q-BANK OBSERVATION before N is frozen. Running at 458 and attributing a null to sample size afterwards was rejected as unfalsifiable: a loss would read as "underpowered" and a win as "evidently sufficient", an escape hatch available only after the result. N0 = 458 stays in the ladder deliberately — it is the set already known adequate for the diagonal statistic, so the ladder directly tests whether a covariance-aware encoder needs more than a diagonal-sufficient sample. Either outcome is evidence. R4.1 recorded as STILL OPEN: NVFP4 carries 16-element group scales and a tensor scale, so the recipe must pin damping, column/act order, block size, tensor- and group-scale selection, code selection, when scales are recomputed, and whether compensated weights may change a later group scale. The last two matter most — if compensation can trigger a rescale, column order changes the GRID rather than only the rounding, and two runs of "the same" recipe would not be the same encoder.
…loses "materially" FIXED-GRID. Only the E2M1 code nibbles may differ from nvfp4-nearest-v1; every scale byte is byte-identical by construction rather than assertion. That makes R4 answer one question — "is nearest ROUNDING the problem?" — rather than the compound question about rounding vs group scales vs tensor scale, which no single arm could separate. The grid is transcribed from larql_models::quant::nvfp4 as the authority, not from generic NVFP4 descriptions: W is [out, in] row-major, groups are contiguous runs of 16 along the INPUT axis, lo nibble is the even element, tensor_scale = amax/(448*6) over the whole matrix, group scale = f32_to_e4m3(amax_group / 6 / tensor_scale) per (row, group), and the code is f32_to_e2m1(value * inv) with ties to even index. Scales are computed ONCE from the original weights and never recomputed — GPTQ's static-groups mode. This closes the ambiguity that would otherwise be fatal: compensation cannot trigger a rescale, so column order cannot mutate the grid, only the rounding decisions on it. Parameters: blocksize 128, damping 0.01*mean(diag(H)), act-order false, original K order. SATURATION IS INSTRUMENTED. nvfp4-nearest-v1 cannot saturate, because the group scale is chosen so the group's amax lands exactly at E2M1_MAX. Fixed-grid GPTQ can, because compensation moves values against a scale chosen for the originals, and f32_to_e2m1 saturates to +/-6 rather than erroring. That is a real cost of byte-identical scales, so it is counted and reported per site class — it distinguishes "compensation does not help here" from "compensation was clipped away", which are different findings. Calibration-aware SCALE selection is explicitly deferred to a separate later recipe (nvfp4-gptq-scales-v1). Folding it in would make a win unattributable. N-SELECTION RULE loses the word "materially", which would have left the decision to be made after seeing the numbers. Replaced with a one-standard- error rule: smallest N whose held-out reconstruction is within one SE of the best, for every site class. Resampling is over VALIDATION PROMPTS, not token positions, because positions inside a prompt are not independent and a position-level bootstrap would understate the error badly. R4.2 added: memory is settled but COMPUTE is not. XᵀX cost grows linearly in N at the 8192-wide site (~0.06 TFLOP at N=458, ~8.8 TFLOP at N=65,536, per layer, times 40). So the cost benchmark now precedes writing the calibration corpus — there is no point writing 65,536 positions if that rung cannot be run. The benchmark is pure cost and cannot contaminate the experiment.
R4.1 — nvfp4-gptq-v1 fully specified as FIXED-GRID GPTQ (only E2M1
codes may differ from nvfp4-nearest-v1; every scale byte identical by
construction). Three guards added: dead coordinates (exact rule on the
raw Hessian diagonal, no epsilon — falls back to ordinary nearest,
preserving the one-variable contract); a hard byte-level
zero-compensation oracle (disable compensation, the entire payload
must match nearest, not just scale bytes); numerical precision framed
as part of the recipe, decided by measurement rather than asserted.
R4.2 — cost/feasibility, now closed:
1. First measurement: the existing hand-rolled
larql_compute::cpu::ops::linalg::cholesky (f64-only, no LAPACK) did
not complete one d=8192 factorisation in 6+ minutes CPU — manually
killed. Worded carefully as an implementation-backend negative, not
an R4.2 verdict.
2. Accelerated probe (lapack-sys against Accelerate, wrappers validated
against linalg.rs's own reference on synthetic matrices first, rel.
error 1e-15..1e-16): GEMM-based H-accumulation 8.3x faster than
scalar streaming; f64 Cholesky+fused-inverse+inverse-Cholesky in
6.3s total, residual 1.6e-15; f32 Cholesky alone in 0.33s, residual
2.8e-6. Resolved the open question: dense-H GPTQ at d=8192 is
tractable in seconds, not minutes — the negative was entirely
linalg.rs, never the algorithm or matrix size.
3. Precision gate, wiring the previously-unwired f32 inverse (spotri)
and running the mechanical acceptance gate frozen in ENCODER-R4.md
BEFORE this measurement (residual <=1e-4, update-vector max-rel
<=1e-3, both derived from the standard Cholesky backward-error bound
n*u, not fitted to any observed number): f32 fails on the
update-vector probe (max-rel 60.5, though RMS is only 6.7e-6 — a
near-zero-denominator artifact, disclosed honestly, not used to
overturn the frozen verdict). Production precision = f64.
4. Peak memory measured via two isolated `/usr/bin/time -l` process
invocations (never both precisions in one process — RSS
high-water-marks don't go down): f64 and f32 report essentially
identical ~14.37 GiB peaks, because this probe's design (load the
whole 3B model to capture one layer) dominates over the d=8192
linalg's own ~1-2 GiB working set. Recorded as a property of the
probe, not evidence real per-site memory is 14 GiB.
5. GEMM ladder actually benchmarked (deterministic synthetic X, no
model load) instead of extrapolated from N0's rate — and the
earlier extrapolation in this same doc was wrong: N4=65,536 costs
~79s of GEMM across 40 layers, not the ~4.2 hours a linear
extrapolation from N0 implied. GEMM scales sublinearly on this
hardware; nothing here disqualifies any N-ladder rung on cost.
Real calibration-CAPTURE cost (running the forward pass, as opposed
to GEMM's synthetic-X cost) is flagged as separate and still open
for the implementation step, not resolved or blocked by this one.
R4.2 CLOSED: no low-rank approximation, no block-diagonal H, no
weakened GPTQ was needed. Full dense-H fixed-grid GPTQ is a practical
offline-compiler operation on Granite-scale models on this machine.
The remaining question moves from "can we afford this" to "does it
help" — calibration sufficiency (rule already frozen, not yet run),
then whether Hessian-aware code selection recovers meaningful Q-BANK
quality at identical bytes, which R4 still has a genuine chance to
falsify.
Probes: larql-probes/examples/encoder_r4/{r4_2_cholesky_cost,
r4_2_accelerated_cholesky_cost,r4_2_precision_and_memory}.rs — the
first two files are the historical record of the negative-then-
positive arc, kept rather than deleted; the third is the closing
measurement. That crate's Cargo.toml still points at worktree-encoder-r4
via a temporary local path dependency (not the usual pinned git rev) —
revert once this is pushed.
Also fixes a real sequencing bug the Sequence section's flat numbering
invited: "R4.0-CAL" is two steps on opposite sides of the encoder
implementation, not one. R4.0-CAL-A (freeze pools/digests/rules) needs
no encoder; R4.0-CAL-B (run the ladder, measure held-out reconstruction
of Q_N(W)) cannot run before nvfp4-gptq-v1 exists, since the
sufficiency statistic IS the encoder's own output. R4.2's GEMM-cost
result (all N affordable) answers a different question (economic) than
R4.0-CAL-B (statistical) — the two aren't interchangeable. Labelled
explicitly in the Sequence steps so this can't be misread as one step
again.
… GPTQ ladder Freezes the five things R4.0-CAL-A was scoped to, and nothing else: calibration pool, validation pool, the Q-BANK-1 exclusion gate, the nested N-prefix ladder, and a reference to the already-frozen one-SE selection rule (unchanged, not re-litigated here). Calibration extends the existing frozen N0=458 bank (unchanged, taken as a fixed prefix) with prompts from Walden and On the Origin of Species, interleaved so every rung gets both sources, not one drained before the other. Validation is a fully disjoint pool from Pride and Prejudice and Twenty Thousand Leagues Under the Sea — different books entirely, so cross-pool overlap is impossible by construction, and also checked: 0 exact-text overlap, 0 substring overlap, both directions, against both Q-BANK-1 and each other. Achieved nested prefixes: N = 458 12 prompts 458 positions (unchanged N0) N = 2,048 16 prompts 2,134 positions N = 8,192 27 prompts 8,893 positions N = 32,768 68 prompts 32,994 positions N = 65,536 122 prompts 65,972 positions Validation: 683 prompts, 176,879 positions — ~10x Q-BANK-1's own prompt count, sized for prompt-level bootstrap power since the one-SE rule resamples prompts, not positions. Two content-quality issues were caught and fixed before freezing: Walden's table-of-contents block (no blank lines, read as prose to a length-only filter) and On the Origin of Species's per-chapter argument lists (short, properly period-terminated fragments that a periods-per-length check alone missed) — fixed with an average-AND- median sentence-length filter. A source front-loading issue was also self-caught and fixed via round-robin interleaving. Digests: calibration a709493137de628d09eef104efbd1d09565e54c4aa9d6f7e6a554ad054cd4329 validation 7ee9442f64a486c3a934096d529c49a09f5524e57dba80e45031992460a23c97 bench/prompts/quality-bank-1/r4_cal_a_build.py build|verify mirrors freeze_calibration.py's freeze/verify convention; each Gutenberg source is sha256-pinned so verify refuses on drift rather than silently rebuilding a different pool. Source cache directory is gitignored (reproducible via sha256-checked re-fetch, not vendored). Does not touch the encoder: no nvfp4-gptq-v1 pack, no Q_N(W), no reconstruction error. R4.0-CAL-B (choosing N from held-out reconstruction) needs the encoder built first and is out of scope here.
…both frozen oracles pass
ENCODER-R4.md Sequence step 8, scoped narrowly per the frozen spec:
"Produce the first deterministic nvfp4-gptq-v1 tensor that passes the
zero-compensation and scale-identity oracles." New module beside
nvfp4_pack.rs, not a new crate:
gptq/hessian.rs dead/alive column partition from the raw H[j,j]==0
rule (checked before damping, no epsilon); the
reduced (alive-only) sub-matrix GPTQ factorises
gptq/sequential.rs Cholesky(H_lambda) -> H^-1 -> Cholesky(H^-1), then
GPTQ's sequential per-row column elimination
gptq/pack.rs orchestration: nearest-v1's own frozen scale/code
pass, merged with GPTQ's codes on alive columns
Takes a raw calibration Hessian directly (Array2<f64>) rather than
capturing its own activations, uses the existing scalar
larql_compute::cpu::ops::linalg Cholesky rather than the LAPACK backend
R4.2 benchmarked (every fixture here is small enough that the proven
scalar path is fast — no new dependency owed yet), and does not wire
into VINDEX3 REPRESENT dispatch. All three are the explicitly deferred
"one tensor -> one layer -> sequential full model" expansion.
Dead-coordinate handling is implemented as an explicit reduced-matrix
gather rather than relying on the (provably valid, but harder-to-audit)
emergent block-decoupling a full-matrix ridge-only solve would also
produce — "no error propagated from or into a dead column" is a
structural guarantee of what got fed to the linear algebra, not a
floating-point property a reader has to trust.
All four implementation oracles pass:
1. zero compensation all-zero H -> byte-identical to nvfp4-nearest-v1,
full payload, not just scales
2. normal GPTQ scale bytes always identical to nearest; only
E2M1 payload nibbles may differ
3. determinism same weights + same H, twice -> byte-identical
4. stored decode decode-then-reencode under the same frozen grid
reproduces the stored codes exactly
Two test cases are exact, hand-derived arithmetic (H=[[2,1],[1,2]],
ridge=0 has a closed-form propagation factor of exactly +0.5*err[0]),
not just eyeballed output: a code flip (0.24,0.24 -> codes 0,1) and a
compensation-induced saturation (5.0,5.6 -> codes 6,7, second element
saturating though 5.6 alone would not) — saturation is instrumented
per-element, something nearest-v1 can never produce.
23 new tests, cargo test -p larql-vindex gptq all passing; hessian.rs
and pack.rs at 100% line coverage, sequential.rs at 95%; cargo clippy
-p larql-vindex --all-targets -- -D warnings clean; cargo fmt --check
clean; full crate suite (3033 tests) unaffected.
EncoderRecipe::gptq_v1() added as a named recipe (nvfp4_pack.rs);
EncoderRecipe::current() still returns nearest_v1 — REPRESENT dispatch
wiring is explicitly out of scope for this milestone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Pre-registration + closing measurements for ENCODER-R4 — the experiment asking whether Granite's measured NVFP4 quantization damage is a property of the format, or of
nvfp4-nearest-v1's rounding choices specifically. One variable: encoder recipe (nearest vs. GPTQ), byte-identical everything else, judged once against the frozen granite-4.1-3b Q-BANK-1 control.Docs-only change (
bench/prompts/quality-bank-1/ENCODER-R4.md) — no engine code touched.R4.1 —
nvfp4-gptq-v1fully specifiedFIXED-GRID GPTQ: only E2M1 code nibbles may differ from
nvfp4-nearest-v1; every scale byte (tensor + every group) is byte-identical by construction, computed once from the original weights, never recomputed. Grid transcribed fromlarql_models::quant::nvfp4as authority. Three guards added:H[j,j]==0, before damping), no epsilon threshold — falls back to ordinary nearest rounding, preserving the one-variable contract.R4.2 — cost/feasibility, closed
linalg.rsCholesky (f64-only, no LAPACK) didn't complete oned=8192factorization in 6+ minutes CPU — killed. Worded as an implementation-backend negative, not an R4.2 verdict.lapack-sysagainst Accelerate, validated againstlinalg.rs's own reference first): GEMM-based H-accumulation 8.3x faster; f64 Cholesky+fused-inverse+inverse-Cholesky in 6.3s total, residual 1.6e-15. Resolved: dense-H GPTQ at d=8192 is tractable in seconds, not minutes — the negative was entirely the implementation, never the algorithm or matrix size./usr/bin/time -lruns: f64 and f32 both ~14.37 GiB — dominated by this probe's whole-model residency, not the d=8192 linalg itself. Recorded honestly as a probe-design artifact, not a per-site cost claim.N=65,536costs ~79s across 40 layers, not the ~4.2 hours a linear extrapolation implied. Nothing disqualifies any calibration-size rung on cost.R4.2 CLOSED: no low-rank approximation, no block-diagonal H, no weakened GPTQ was needed. Full dense-H fixed-grid GPTQ is a practical offline-compiler operation at this scale.
Sequencing fix
Split "R4.0-CAL" into two explicit phases so it can't be misread as one step: R4.0-CAL-A (freeze calibration/validation pools, disjointness, nested N prefixes, digests — needs no encoder) and R4.0-CAL-B (run the implemented encoder at each N, measure held-out reconstruction, choose N — cannot run before the encoder exists, since the sufficiency statistic is the encoder's own output).
What's still open
Calibration sufficiency (rule frozen, not yet run — needs the encoder implemented first), and ultimately whether Hessian-aware code selection recovers meaningful Q-BANK quality at identical bytes. No GPTQ codes computed, no reconstruction inspected, no Q-BANK observed anywhere in this PR — that discipline is what keeps the eventual one-shot Q-BANK run genuinely one-shot.