diff --git a/DESIGN.md b/DESIGN.md index b7b01c0..ba58874 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -6,21 +6,19 @@ ## Contents -1. [The Embedding Problem](#1-the-embedding-problem) - - 1.1 [Embeddings and the unit hypersphere](#11-embeddings-and-the-unit-hypersphere) - - 1.2 [Incommensurability](#12-incommensurability) - - 1.3 [Concentration of measure](#13-concentration-of-measure) - - 1.4 [The surface-area information bound](#14-the-surface-area-information-bound) - - 1.5 [The L1 metric and the cross-polytope](#15-the-l1-metric-and-the-cross-polytope) - - 1.6 [The quaternary coordinate](#16-the-quaternary-coordinate) - - 1.7 [Index design consequences](#17-index-design-consequences) - - 1.8 [Relational geometry and the lingua franca](#18-relational-geometry-and-the-lingua-franca) +1. [The Quantization Problem](#1-the-quantization-problem) + - 1.1 [The quantization problem](#11-the-quantization-problem) + - 1.2 [Concentration of measure](#12-concentration-of-measure) + - 1.3 [The surface-area information bound](#13-the-surface-area-information-bound) + - 1.4 [The L1 metric and the cross-polytope](#14-the-l1-metric-and-the-cross-polytope) + - 1.5 [The quaternary coordinate](#15-the-quaternary-coordinate) + - 1.6 [The quantization problem restated](#16-the-quantization-problem-restated) 2. [Quantization](#2-quantization) - - 2.1 [The thermal constraint](#21-the-thermal-constraint) + - 2.1 [The resource constraint](#21-the-resource-constraint) - 2.2 [Binary quantization](#22-binary-quantization) - 2.3 [Ternary quantization](#23-ternary-quantization) - - 2.4 [Standard Q4 (different problem)](#24-standard-q4-different-problem) - - 2.5 [Quaternary semantic quantization](#25-quaternary-semantic-quantization) + - 2.4 [Reconstruction vs. structural quantization](#24-reconstruction-vs-structural-quantization) + - 2.5 [Quaternary quantization](#25-quaternary-quantization) - 2.6 [The Lee metric](#26-the-lee-metric) - 2.7 [The Gray map](#27-the-gray-map) - 2.8 [The complement involution](#28-the-complement-involution) @@ -31,89 +29,35 @@ - 3.4 [Block file organisation](#34-block-file-organisation) - 3.5 [Window queries](#35-window-queries) - 3.6 [Bucket density](#36-bucket-density) +4. [The Combinatorial Structure](#4-the-combinatorial-structure) + - 4.1 [The Geode factorization and hierarchical quantization](#41-the-geode-factorization-and-hierarchical-quantization) + - 4.2 [Euler's polytope formula as a quantization constraint](#42-eulers-polytope-formula-as-a-quantization-constraint) + - 4.3 [Mixed-precision quantization](#43-mixed-precision-quantization) + - 4.4 [Analytical threshold geometry](#44-analytical-threshold-geometry) + - 4.5 [Reconstruction and series reversion](#45-reconstruction-and-series-reversion) +5. [Application: Semantic Embeddings](#5-application-semantic-embeddings) + - 5.1 [Embeddings and the unit hypersphere](#51-embeddings-and-the-unit-hypersphere) + - 5.2 [Incommensurability](#52-incommensurability) + - 5.3 [The thermal constraint](#53-the-thermal-constraint) + - 5.4 [Relational geometry and the lingua franca](#54-relational-geometry-and-the-lingua-franca) --- -## 1 The Embedding Problem +## 1 The Quantization Problem -**Why are embeddings incommensurable?** Two models trained on the same corpus -produce vectors that cannot be directly compared — their coordinate frames are -arbitrary rotations of one another, fixed only by random initialisation and training -order. This section builds the answer from first principles: what an embedding is, -why its coordinate frame is unfixed, and why that constraint is less limiting than it -first appears. +### 1.1 The quantization problem -### 1.1 Embeddings and the unit hypersphere +What is the natural discrete coordinate system for a continuous signal? -A transformer model maps a document to a vector in $\mathbb{R}^n$ by passing its token -sequence through the network and L2-normalising the hidden-state activation at a -selected token position. The output is a point on the unit hypersphere: - -$$e \in S^{n-1} = \left\{ x \in \mathbb{R}^n : \|x\| = 1 \right\}$$ +A point in $\mathbb{R}^n$ — or on the unit hypersphere $S^{n-1}$ — must be represented in a finite alphabet. The question is which alphabet, which metric, and which encoding preserve the structure that matters while discarding the structure that does not. -Each coordinate satisfies $\sum_i e_i^2 = 1$. Semantic similarity between documents is -measured by cosine similarity, which equals the dot product between normalised vectors: +This question arises in many domains. Semantic embeddings are one important application (§5): a transformer model produces a vector on $S^{n-1}$ and the retrieval system must compress it to a discrete code. But the same geometric problem appears in model weight quantization, signal processing, sensor data encoding, and communication systems. In each case, a continuous measurement must be mapped to a small number of discrete levels, and the fidelity of that mapping is governed by the geometry of the space and the metric under which distances are measured. -$$\text{sim}(u, v) = u \cdot v = \cos\theta_{uv}$$ - -where $\theta_{uv}$ is the angle between $u$ and $v$. - -**Why not mean-pool?** Mean-pooling the token activations — computing -$\bar{h} = \frac{1}{T}\sum_{t=1}^{T} h_t$ where $h_t \in \mathbb{R}^n$ is the -hidden-state activation at position $t$ — produces, for unit-normalised token vectors -$h_t$, a vector inside the unit ball ($\|\bar{h}\| \leq 1$). When the document's tokens -activate semantically diverse directions, as any non-trivial document does, the -centroid is short because distinct directions cancel. The subsequent L2 renormalisation -stretches $\bar{h}$ back to $S^{n-1}$ by the factor $1/\|\bar{h}\|$. The components that -survive cancellation are those common to all token positions — model-specific bias and -positional structure — not the document's semantically distinctive content. -Mean-pooling as a dimensional-reduction technique therefore increases the noise and -lowers the signal. +The framework developed in this document is general. The quantization machinery — the four-cell decomposition, the Lee metric, the Gray map, the complement involution, the transition key — depends only on the L1 geometry of high-dimensional space, not on the source of the continuous signal. Application-specific considerations (the embedding coordinate frame, the thermal constraint of edge inference) are treated in §5. --- -### 1.2 Incommensurability - -$S^{n-1}$ has no preferred orientation. Every rotation $Q \in O(n)$ is an isometry: - -$$u \cdot v = (Qu) \cdot (Qv)$$ - -A model trained on a corpus places semantically similar documents near each other on -the sphere, but the absolute coordinate frame is determined by random initialisation -and training order. A second model trained on the same corpus produces an embedding -space related to the first by some unknown $Q \in O(n)$: - -$$e_B(x) \approx Q \cdot e_A(x)$$ - -$Q$ is not available from the model weights and cannot be recovered without a large set -of paired examples and a Procrustes alignment solve. - -For any two documents $x, y$ embedded by different models: - -$$e_A(x) \cdot e_B(y)$$ - -depends entirely on $Q$. The dot product measures alignment between two arbitrary -coordinate frames, not semantic similarity between documents. - -An untrained transformer also produces vectors on $S^{n-1}$, but with no training -signal to cluster similar documents, the geometry carries no semantic content. An -embedding is meaningful as a retrieval key only after training. - -**Coordinate-frame incommensurability does not imply relational incommensurability.** -The rotation $Q$ scrambles absolute positions. It does not scramble differences. -For any documents $a, b, c, d$ satisfying $e_A(a) - e_A(b) + e_A(c) \approx e_A(d)$ -in Model A's frame, the same relation holds in Model B's frame, because $Q$ distributes -linearly over vector arithmetic: - -$$Q(e_A(a) - e_A(b) + e_A(c)) = e_B(a) - e_B(b) + e_B(c) \approx e_B(d)$$ - -Semantic analogies, concept offsets, and the parallelogram structure of the embedding -space are invariant under rotation. Incommensurability is a property of absolute -coordinates, not of semantic geometry. The implications are developed in §1.8. - ---- - -### 1.3 Concentration of measure +### 1.2 Concentration of measure The volume of the $n$-dimensional unit ball: @@ -148,7 +92,7 @@ $$\mathbb{E}[u \cdot v] = 0 \qquad \mathrm{Var}(u \cdot v) = \frac{1}{n}$$ The standard deviation of cosine similarity between two random unit vectors is $1/\sqrt{n}$. At $n = 256$ this is $\approx 0.063$: the full range $[-1, +1]$ is compressed into -noise of order $\pm 0.06$. A trained model must push similar documents closer than +noise of order $\pm 0.06$. A trained model must push similar items closer than this noise floor. Each coordinate of a random unit vector satisfies: @@ -160,9 +104,9 @@ i.i.d. $\mathcal{N}(0, 1/n)$. --- -### 1.4 The surface-area information bound +### 1.3 The surface-area information bound -The shell thickness $\varepsilon^{*}(n) \sim c/n$ shrinks as $n$ grows. If the semantic +The shell thickness $\varepsilon^{*}(n) \sim c/n$ shrinks as $n$ grows. If the space has a characteristic scale $L$, the absolute shell thickness is: $$\delta(n) = \frac{c \cdot L}{n}$$ @@ -187,13 +131,14 @@ The surface area of $S^{n-1}$ is $\mathcal{A}_{n-1}(1) = 2\pi^{n/2}/\Gamma(n/2)$ The minimum distinguishable angular separation grows as $\sim 1/\sqrt{n}$, so the marginal information per additional dimension decreases as the shell compresses. -The semantic content of a document is encoded in which region of $S^{n-1}$ its -embedding occupies. The quantization problem is the problem of discretising that +The information content of a signal is encoded in which region of $S^{n-1}$ its representation occupies. The quantization problem is the problem of discretising that surface efficiently. +The Euler polytope formula $V - E + F = \chi$ constrains the combinatorial capacity of any quantization lattice: the number of cells ($F$), boundaries ($E$), and vertices ($V$) are not independent. This is the combinatorial analog of the surface-area bound: just as the continuous information capacity is determined by the area of $S^{n-1}$, the discrete information capacity is determined by the topology of the quantization lattice (§4.2). + --- -### 1.5 The L1 metric and the cross-polytope +### 1.4 The L1 metric and the cross-polytope The L2 distance couples all $n$ coordinates under a square root: @@ -221,7 +166,7 @@ faces lie on the coordinate halfspaces, so the grid and the ball are aligned. --- -### 1.6 The quaternary coordinate +### 1.5 The quaternary coordinate Under the L1 metric, $d_1(u, v)$ decomposes into $n$ independent scalar problems: for each dimension $i$, how far apart are $u_i$ and $v_i$? The full distance is their @@ -250,82 +195,34 @@ $$d_L(u, v) = \sum_{i=1}^{n} \min(|u_i - v_i|,\ 4 - |u_i - v_i|)$$ The Gray map (§2.7) makes this computable by `popcnt(XOR)` on 64-byte vectors without symbol decoding. -**The float32 activation** is not the ground truth that the quaternary symbol +**A continuous measurement** is not the ground truth that the quaternary symbol approximates. It is an overcomplete representation of an ordinal position. The bits encoding intra-cell displacement contribute to the L2 norm and to shell concentration -(§1.3); they are not recoverable signal for L1 retrieval. The quantization discards +(§1.2); they are not recoverable signal for L1 retrieval. The quantization discards them. --- -### 1.7 Index design consequences +### 1.6 The quantization problem restated -Three consequences follow from §1.1–1.6. - -**Coordinate-frame incommensurability.** Quantization thresholds are calibrated from -one model's activation distribution. A different model produces different thresholds. -Quaternary codes in absolute terms are not comparable across models; an index must be -rebuilt when the model changes. The relational structure of the embedding space — -differences, analogies, topological trajectories — is preserved across models (§1.8), -but recovering it requires a Procrustes solve over paired examples, which is -thermally infeasible on constrained hardware. +The question is not: how many bits per dimension approximate angular distance on $S^{n-1}$? The question is: what is the natural discrete coordinate system of the L1 unit ball? The answer — four cells per dimension — is derived in §2.5. -**The thermal constraint and the geometric constraint coincide.** Cross-model -alignment requires a Procrustes solve over paired examples — geometrically necessary -but thermally impossible on constrained hardware. Using the LLM's own activations -satisfies both constraints simultaneously. - -**The quantization problem restated.** The question is not: how many bits per -dimension approximate angular distance on $S^{n-1}$? The question is: what is the -natural discrete coordinate system of the L1 unit ball? The answer — four cells per -dimension — is derived in §2.5. +This question applies to any domain where continuous signals must be discretized under resource constraints: semantic embeddings, model weight compression, signal quantization, and sensor encoding all face the same geometric problem. --- +## 2 Quantization -### 1.8 Relational geometry and the lingua franca - -The rotation $Q$ is unknown and unrecoverable without paired supervision. Direct -cross-model coordinate comparison is therefore meaningless. But $Q$ acts uniformly -on the entire embedding space — it cannot selectively rotate some concepts while -leaving others fixed — so the *shape* of the space is preserved. - -The canonical illustration: $\text{king} - \text{man} + \text{woman} \approx \text{queen}$. This -vector arithmetic works not because "king" lives at a special absolute address, but -because the training corpus encodes a consistent geometric offset between gendered -role-pair concepts. Both Model A and Model B, having trained on the same language, -construct the same parallelogram in their respective (rotated) frames. The analogical -relationship is not a coincidence of one model's random initialisation; it is a -property of the data. - -The transition key (§3) exploits this directly. By recording the *sequence of -semantic directions visited* during inference — the topological trajectory through -the embedding space — rather than any absolute coordinate, it indexes shape rather -than position. Two models reading the same passage may produce entirely different -float32 vectors at each layer, yet execute the same sequence of semantic turns: the -same "hairpin" at the subordinate clause, the same "commitment" at the topic word. -The run-reduced key captures that shared structure without requiring knowledge of $Q$. - -Embeddings across models are not fully incommensurable. Their coordinate frames are; -their semantic geometry is not. The gap between those two facts is where Q2 operates. - -$$\underbrace{\text{king} - \text{man} + \text{woman}}_{\text{vector arithmetic on any model}} \approx \underbrace{\text{queen}}_{\text{same answer, rotated frame}}$$ - -The coordinate frames differ by $Q$. The parallelogram does not. Viewed from the -right angle, the incommensurability dissolves. 🤓 +### 2.1 The resource constraint ---- +Quantization operates under a resource budget. The question is: given fixed compute, memory, or energy, what is the highest-fidelity discrete representation? -## 2 Quantization - -### 2.1 The thermal constraint +The resource constraint applies equally to edge inference (thermal), communication channels (bandwidth), and storage systems (capacity). In each case, the number of distinguishable levels per dimension is bounded by the available budget. -Running a large language model on a consumer device operates near the thermal ceiling +**The thermal constraint as a concrete instance.** Running a large language model on a consumer device operates near the thermal ceiling of that hardware. A second dedicated embedding model, run in parallel or in sequence, -would exceed the thermal budget, cause throttling, and drain the battery. - -The constraint is therefore: given that an LLM is already running and already producing -activations, what is the highest-quality semantic index constructible from those -activations alone, at a fixed and predictable compute cost? +would exceed the thermal budget, cause throttling, and drain the battery. The constraint is therefore: given that an LLM is already running and already producing +activations, what is the highest-quality index constructible from those +activations alone, at a fixed and predictable compute cost? This thermal instance is developed further in the embedding application (§5.3). --- @@ -363,13 +260,13 @@ $$I(v_i;\ q_{\text{tern}}(v_i)) = \log_2 3 \approx 1.585 \text{ bits}$$ **What remains lost.** The ternary alphabet $\{-, 0, +\}$ has group structure $\mathbb{Z}_3$, which admits no fixed-point-free involution — there is no map $\theta: \mathbb{Z}_3 \to \mathbb{Z}_3$ satisfying $\theta^2 = \text{id}$ and -$\theta(x) \neq x$ for all $x$. The complement relationship between a concept and its -semantic opposite — present in the activation space as the relationship between +$\theta(x) \neq x$ for all $x$. The complement relationship between a value and its +structural opposite — present in the signal space as the relationship between strong-positive and strong-negative directions — is not representable. --- -### 2.4 Standard Q4 (different problem) +### 2.4 Reconstruction vs. structural quantization Standard 4-bit quantization for LLM weight compression (GPTQ, AWQ, and related methods) assigns 4 bits per parameter using a learned or analytical codebook that @@ -379,17 +276,15 @@ $$\min_{\hat{W}} \| W - \hat{W} \|_F^2$$ subject to $\hat{W}$ having 4-bit entries ($2^4 = 16$ levels per dimension). -This is weight quantization, not activation quantization for retrieval. The objective, -metric, and distribution are all different. The two methods are not in the same design -space. +This is **reconstruction quantization**: the objective is to minimize $\|W - \hat{W}\|_F^2$, approximating the original signal as closely as possible. **Structural quantization** (the subject of this document) has a different objective: preserve relational and topological structure — distances, trajectories, and complement relationships — rather than pointwise values. The two objectives share the quaternary alphabet but differ in metric, distribution, and purpose. --- -### 2.5 Quaternary semantic quantization +### 2.5 Quaternary quantization The preceding analysis identifies what a correct scheme requires: -1. **Sign** — which side of the semantic hyperplane. +1. **Sign** — which side of the decision hyperplane. 2. **Magnitude class** — near the boundary or strongly committed. 3. **Complement structure** — a fixed-point-free involution $\theta$ satisfying $\theta^2 = \text{id}$ and $\theta(x) \neq x$ for all $x$. @@ -426,10 +321,12 @@ graph LR ``` **Empirical calibration.** In practice $\tau^{*}$ is estimated from a reservoir sample -of 1 024 document activations per compaction cycle, using the empirical 25th and 75th +of 1 024 sample activations per compaction cycle, using the empirical 25th and 75th percentiles of $v_i$ to keep the symbol distribution close to equiprobable without assuming a specific activation shape. +**Analytical threshold computation.** For source distributions expressible as polynomial or mixture models, the equiprobable threshold $\tau^*$ can be computed analytically via the hyper-Catalan series (Wildberger & Rubine 2025; the formal development is in §4.4). The threshold equation $F(\tau) = k/4$ for CDF $F$ becomes a polynomial in the distribution parameters, and the threshold solution $\alpha = \sum_\mathbf{m} C_\mathbf{m} \cdot t_2^{m_2} t_3^{m_3} \cdots$ converges without iteration. Truncation order trades precision for compute cost — a natural fit for the resource-constrained setting of §2.1. This does not replace empirical calibration; it provides a second path when a parametric model of the source distribution is available. + Under the equiprobable target, each dimension carries: $$I(v_i;\ q(v_i)) = \log_2 4 = 2 \text{ bits}$$ @@ -452,10 +349,9 @@ $\mathbb{Z}_4$, the Lee distance is: $$d_L(u, v) = \min(|u - v|,\ 4 - |u - v|)$$ -**Why this metric matches semantic distance.** A weak-negative and a weak-positive -activation ($B$–$C$) are adjacent states: the concept was weakly activated in both -documents, with opposite sign. Lee distance 1 reflects this. The complement pairs -$A$–$C$ and $B$–$D$ represent semantic opposition: one document activates a dimension +**Why this metric matches structural distance.** A weak-negative and a weak-positive +activation ($B$–$C$) are adjacent states: the coordinate was weakly committed in both cases, with opposite sign. Lee distance 1 reflects this. The complement pairs +$A$–$C$ and $B$–$D$ represent structural opposition: one signal activates a dimension strongly in one direction, the other in the complementary direction. Lee distance 2 reflects this. Strong-negative and strong-positive ($A$–$D$) share strong commitment to their respective directions; the cyclic metric assigns them distance 1, not 2. @@ -623,8 +519,8 @@ transition sequence but different run lengths map to the same $R$. *Proof.* Run-reduction discards run-length information. $\square$ -Run-length invariance means a document that visits a semantic state once and a -document that dwells in it for many consecutive dimensions share the same key. The key +Run-length invariance means a signal that visits a quantization state once and a +signal that dwells in it for many consecutive dimensions share the same key. The key records which states were visited, not how long each visit lasted. **Example of run-reduction:** @@ -682,7 +578,7 @@ distinction is resolved by the Lee-distance re-ranking step. The key is left-aligned: $r_0$ occupies the most significant two bits. -**Definition.** The *semantic depth* of a transition at position $i$ in $R$ is $i$. +**Definition.** The *resolution depth* of a transition at position $i$ in $R$ is $i$. Depth 0 is the first transition; depth 31 is the finest resolvable discrimination. **Proposition 3.6** *(Prefix clustering).* Two keys $K_1$ and $K_2$ share a common @@ -764,12 +660,12 @@ Any practically chosen window satisfies $\delta < 2^{61}$ ($\approx 2.3\times10^ a query spanning more than $10^{18}$ consecutive keys is a corpus scan, not a window query. -**Semantic meaning of $\delta$.** Setting $\delta = 4^{32-j} - 1$ retrieves exactly +**Interpretation of $\delta$.** Setting $\delta = 4^{32-j} - 1$ retrieves exactly all documents whose transition sequences agree with the query in the first $j$ transitions (Corollary 3.7): -| Prefix length $j$ | Half-width $\delta$ | Semantic meaning | -|:-----------------:|:-------------------:|:-----------------| +| Prefix length $j$ | Half-width $\delta$ | Interpretation | +|:-----------------:|:-------------------:|:---------------| | 32 | 0 | Exact key match | | 31 | 3 | First 31 transitions match | | 30 | 15 | First 30 transitions match | @@ -782,8 +678,8 @@ transitions (Corollary 3.7): ### 3.6 Bucket density Documents with identical transition sequences share the same key and occupy the same -bucket. This is the intended behaviour: co-located documents traversed the same -semantic path. +bucket. This is the intended behaviour: co-located signals traversed the same +quantization path. Let $D$ be the number of distinct transition sequences in a corpus of $C$ documents. Mean bucket size is $C/D$. @@ -800,14 +696,265 @@ observable universe overstates it by 16 orders of magnitude — the observable u contains $\approx 10^{80}$ atoms and $2^{64} \approx 10^{19}$ — but the sparsity conclusion is correct regardless. +**Admissible sequence count.** The transition trie has a root of arity $q = 4$ and all subsequent nodes of arity $q - 1 = 3$ (each successor must differ from its predecessor). The number of distinct transition sequences of length $k$ is: + +$$D(k) = q \cdot (q-1)^{k-1} = 4 \cdot 3^{k-1}$$ + +For $k = 32$ (the key capacity), $D(32) = 4 \cdot 3^{31} \approx 2.47 \times 10^{15}$, occupying $\approx 1.34 \times 10^{-4}$ of the $2^{64}$ address space. The no-repeat constraint eliminates $\approx 99.99\%$ of possible 64-bit keys, concentrating all valid transition sequences into a sparse subset. + +The generating function for all transition sequences is $S(x) = (1 + x)/(1 - 3x)$, with the Geode factorization $S - 1 = S_1 \cdot G$ where $S_1 = 4x$ and $G = 1/(1 - 3x)$ (§4.1). This decomposition makes the information gain per additional symbol explicit: the first symbol contributes $\log_2 4 = 2$ bits; each subsequent symbol contributes $\log_2 3 \approx 1.585$ bits. A 32-symbol key carries $2 + 31 \times \log_2 3 \approx 51.1$ effective bits of information within its 64-bit container. + +**Hyper-Catalan bucket density.** The expected bucket density is not uniform across the trie. The hyper-Catalan framework (§4.3) counts the number of structurally distinct subtrees at each branching node; when the source distribution is non-uniform (as it is for any trained model), certain subtrees attract disproportionate occupancy. The ratio of observed to expected bucket density at each trie node is a diagnostic of the source distribution's departure from uniformity, connecting the combinatorial structure of §4 to the empirical bucket statistics. + **Corollary 3.9** *(Non-trivial windows).* For any practical corpus, a window of half-width $\delta = 4^{32-j} - 1$ with $j \leq 28$ returns a non-empty result only when the query has neighbours that shared its first $j$ transitions. An empty result is -informative: no indexed document followed the same high-level semantic path as the +informative: no indexed document followed the same high-level path as the query. --- +## 4 The Combinatorial Structure + +### 4.1 The Geode factorization and hierarchical quantization + +The central structural result of Wildberger & Rubine (2025) is the factorization: + +$$S - 1 = S_1 \cdot G$$ + +where $S$ is the generating function for all structured codewords, $S_1$ is the generating function for the first quantization step (the coarsest level), and $G$ is the *Geode* — the generating function for everything after the first level has been decided. + +In the language of quantization, this factorization describes **hierarchical quantization**: first decide the coarse cell, then refine within it. The Geode $G$ counts the refinement possibilities at each subsequent level. + +For Q2's transition key, the factorization is concrete. The generating function for all transition sequences of length $\geq 1$ is: + +$$S(x) - 1 = \frac{4x}{1 - 3x} = \underbrace{4x}_{S_1} \cdot \underbrace{\frac{1}{1-3x}}_{G}$$ + +The first factor $S_1 = 4x$ records the first symbol $r_0$ (4 choices, selecting the block file). The Geode $G = 1/(1-3x) = 1 + 3x + 9x^2 + \cdots$ counts all possible continuations — the tail of the key after the first symbol is fixed. + +| Level | Paper | Q² transition key | General quantization | +|:-----:|:------|:------------------|:--------------------| +| Full structure | $S$ | All transition sequences | All codewords | +| First level | $S_1$ | $r_0$ (first symbol → block file) | Coarse quantization cell | +| Refinement | $G$ | $K_{\text{tail}}$ (remaining key) | Sub-cell refinement | +| Recursion | $S = 1 + S_1 G$ | Key = prefix + tail | Hierarchical VQ | + +The factorization is *recursive*: $G$ itself contains $S$, so the refinement at level 2 factors the same way. This is the algebraic version of a multi-resolution quantization scheme — coarse-to-fine, with the Geode counting the degrees of freedom at each level. + +Window queries (§3.5) are already implicitly performing this hierarchical decomposition. A query with prefix length $j$ fixes the first $j$ symbols (the coarse structure) and retrieves all documents that refine those $j$ symbols in any way. The Geode factorization provides the theory behind why this works: the number of valid refinements at depth $j$ is exactly $3^{32-j}$, which is the coefficient structure of $G$. + +--- + +### 4.2 Euler's polytope formula as a quantization constraint + +The hyper-Catalan coefficient from Wildberger & Rubine (2025): + +$$C_\mathbf{m} = \frac{(E-1)!}{(V-1)! \cdot \mathbf{m}!}$$ + +is governed by Euler's polytope formula $V - E + F = \chi$, where: + +- $F$ = number of cells (the codewords of the quantization lattice), +- $E$ = number of cell boundaries (where the quantization function is discontinuous), +- $V$ = number of vertices (where three or more cells meet), +- $\chi$ = Euler characteristic of the underlying space. + +Euler's formula constrains these quantities: you cannot have $F$ cells, $E$ boundaries, and $V$ vertices in arbitrary combination. The topology of the quantization lattice determines admissible $(V, E, F)$ triples. + +For Q2 specifically, the $\mathbb{Z}_4$ cycle has $V = 4$ vertices, $E = 4$ edges, and $F = 1$ face (the single outer region): + +$$4 - 4 + 1 = 1 = \chi \quad \checkmark$$ + +For the product lattice $\mathbb{Z}_4^n$, the face structure scales with $n$ and its Euler characteristic is computable from the hyper-Catalan framework. + +When extending to higher-order alphabets ($q = 8$, $q = 16$, etc.) or to non-cyclic topologies, Euler's formula provides an *a priori* constraint on what quantization lattice geometries are possible. Rather than searching over lattice designs, one enumerates the admissible $(V, E, F)$ triples and the hyper-Catalan coefficient tells how many distinct quantizations each topology supports. + +--- + +### 4.3 Mixed-precision quantization + +The Bi-Tri (and higher) hyper-Catalan arrays (Wildberger & Rubine 2025, Table 1) count structures with $m_2$ binary splits, $m_3$ ternary splits, and $m_4$ quaternary splits. In quantization terms: + +- A **binary split** is a 1-bit quantization step (above/below threshold). +- A **ternary split** is a $\log_2 3 \approx 1.585$-bit step (below/near/above). +- A **quaternary split** is a 2-bit step (Q2's $\{A, B, C, D\}$). + +A general quantization framework may mix these: use 2-bit precision on high-variance dimensions and 1-bit on low-variance dimensions. The number of distinct mixed-precision codebooks with $m_2$ binary dimensions, $m_3$ ternary dimensions, and $m_4$ quaternary dimensions is: + +$$C_{(m_2, m_3, m_4)} = \frac{(E-1)!}{(V-1)! \cdot m_2! \cdot m_3! \cdot m_4!}$$ + +This is a row of the hyper-Catalan array, directly from the paper. + +**Concrete example.** Consider a 256-dimensional signal where variance analysis identifies 128 high-variance dimensions and 128 low-variance dimensions. Allocating 2 bits to the high-variance dimensions (quaternary, preserving sign and magnitude class) and 1 bit to the low-variance dimensions (binary, preserving sign only) yields a mixed-precision code of $128 \times 2 + 128 \times 1 = 384$ bits — a 25% reduction from the uniform quaternary code — with the hyper-Catalan framework bounding the number of structurally distinct such allocations. + +--- + +### 4.4 Analytical threshold geometry + +For non-Gaussian distributions — mixtures, heavy-tailed activations, quantized signal processing — the equiprobable threshold equation: + +$$F(\tau) = \frac{k}{q}, \qquad k = 1, \ldots, q-1$$ + +for CDF $F$ becomes a polynomial in the distribution parameters. The hyper-Catalan series solves this combinatorially: + +$$\alpha = \sum_\mathbf{m} C_\mathbf{m} \cdot t_2^{m_2} t_3^{m_3} \cdots$$ + +where each $C_\mathbf{m}$ is a hyper-Catalan number and $t_j$ are functions of the distribution parameters. + +The series converges by direct evaluation — no Newton iteration, no gradient descent. On constrained hardware, a closed-form series that can be truncated to the precision affordable within the resource budget (§2.1) is preferable to an iterative solver that may not converge within budget. The truncation order itself is a resource-allocation decision: each additional term in the hyper-Catalan series refines the threshold, trading compute cost for quantization precision. + +For the standard case ($q = 4$, Gaussian source), the series reduces to the known quartile $\tau^* = \Phi^{-1}(3/4)$. For mixture-of-Gaussians sources — the natural model for multi-modal activation distributions — the threshold is expressible as a hyper-Catalan series in the mixture weights. + +--- + +### 4.5 Reconstruction and series reversion + +If Q2 requires a decode path — for lossy compression applications rather than retrieval — the optimal reconstruction point for symbol $s$ is: + +$$\hat{x}(s) = \mathbb{E}[x \mid q(x) = s]$$ + +For non-uniform distributions, this is *not* the cell centroid; it is the conditional expectation within each quantization cell. Computing it for parametric distributions requires inverting the CDF within each cell, which is a series-reversion problem. + +The hyper-Catalan series provides this inversion combinatorially, without numerical root-finding. Wildberger & Rubine (2025, §10) show that Lagrange inversion and the hyper-Catalan series are two faces of the same coin: the series coefficients that solve the forward threshold problem also yield the inverse. + +This is noted as a future extension. The current Q2 pipeline is retrieval-only (quantize, index, search); no reconstruction step is needed. Should a decode path become necessary — for example, in signal compression or approximate model distillation — the reconstruction formula is already provided by the series-reversion machinery. + +--- + +## 5 Application: Semantic Embeddings + +The framework of §1–§4 is general: it applies to any continuous signal that must be discretized under resource constraints. This section develops the primary application — semantic embeddings produced by transformer models — where the continuous signal is a point on the unit hypersphere and the resource constraint is the thermal budget of edge inference. + +### 5.1 Embeddings and the unit hypersphere + +A transformer model maps a document to a vector in $\mathbb{R}^n$ by passing its token +sequence through the network and L2-normalising the hidden-state activation at a +selected token position. The output is a point on the unit hypersphere: + +$$e \in S^{n-1} = \left\{ x \in \mathbb{R}^n : \|x\| = 1 \right\}$$ + +Each coordinate satisfies $\sum_i e_i^2 = 1$. Semantic similarity between documents is +measured by cosine similarity, which equals the dot product between normalised vectors: + +$$\text{sim}(u, v) = u \cdot v = \cos\theta_{uv}$$ + +where $\theta_{uv}$ is the angle between $u$ and $v$. + +**Why not mean-pool?** Mean-pooling the token activations — computing +$\bar{h} = \frac{1}{T}\sum_{t=1}^{T} h_t$ where $h_t \in \mathbb{R}^n$ is the +hidden-state activation at position $t$ — produces, for unit-normalised token vectors +$h_t$, a vector inside the unit ball ($\|\bar{h}\| \leq 1$). When the document's tokens +activate semantically diverse directions, as any non-trivial document does, the +centroid is short because distinct directions cancel. The subsequent L2 renormalisation +stretches $\bar{h}$ back to $S^{n-1}$ by the factor $1/\|\bar{h}\|$. The components that +survive cancellation are those common to all token positions — model-specific bias and +positional structure — not the document's semantically distinctive content. +Mean-pooling as a dimensional-reduction technique therefore increases the noise and +lowers the signal. + +--- + +### 5.2 Incommensurability + +$S^{n-1}$ has no preferred orientation. Every rotation $Q \in O(n)$ is an isometry: + +$$u \cdot v = (Qu) \cdot (Qv)$$ + +A model trained on a corpus places semantically similar documents near each other on +the sphere, but the absolute coordinate frame is determined by random initialisation +and training order. A second model trained on the same corpus produces an embedding +space related to the first by some unknown $Q \in O(n)$: + +$$e_B(x) \approx Q \cdot e_A(x)$$ + +$Q$ is not available from the model weights and cannot be recovered without a large set +of paired examples and a Procrustes alignment solve. + +For any two documents $x, y$ embedded by different models: + +$$e_A(x) \cdot e_B(y)$$ + +depends entirely on $Q$. The dot product measures alignment between two arbitrary +coordinate frames, not semantic similarity between documents. + +An untrained transformer also produces vectors on $S^{n-1}$, but with no training +signal to cluster similar documents, the geometry carries no semantic content. An +embedding is meaningful as a retrieval key only after training. + +**Coordinate-frame incommensurability does not imply relational incommensurability.** +The rotation $Q$ scrambles absolute positions. It does not scramble differences. +For any documents $a, b, c, d$ satisfying $e_A(a) - e_A(b) + e_A(c) \approx e_A(d)$ +in Model A's frame, the same relation holds in Model B's frame, because $Q$ distributes +linearly over vector arithmetic: + +$$Q(e_A(a) - e_A(b) + e_A(c)) = e_B(a) - e_B(b) + e_B(c) \approx e_B(d)$$ + +Semantic analogies, concept offsets, and the parallelogram structure of the embedding +space are invariant under rotation. Incommensurability is a property of absolute +coordinates, not of semantic geometry. The implications are developed in §5.4. + +--- + +### 5.3 The thermal constraint + +The thermal constraint is the concrete instance of the general resource constraint (§2.1) specific to edge inference. Running a large language model on a consumer device operates near the thermal ceiling of that hardware. A second dedicated embedding model, run in parallel or in sequence, would exceed the thermal budget, cause throttling, and drain the battery. The constraint is therefore: given that an LLM is already running and already producing activations, what is the highest-quality semantic index constructible from those activations alone, at a fixed and predictable compute cost? + +**Coordinate-frame incommensurability.** Quantization thresholds are calibrated from +one model's activation distribution. A different model produces different thresholds. +Quaternary codes in absolute terms are not comparable across models; an index must be +rebuilt when the model changes. The relational structure of the embedding space — +differences, analogies, topological trajectories — is preserved across models (§5.4), +but recovering it requires a Procrustes solve over paired examples, which is +thermally infeasible on constrained hardware. + +**The thermal constraint and the geometric constraint coincide.** Cross-model +alignment requires a Procrustes solve over paired examples — geometrically necessary +but thermally impossible on constrained hardware. Using the LLM's own activations +satisfies both constraints simultaneously. + +--- + +### 5.4 Relational geometry and the lingua franca + +The rotation $Q$ is unknown and unrecoverable without paired supervision. Direct +cross-model coordinate comparison is therefore meaningless. But $Q$ acts uniformly +on the entire embedding space — it cannot selectively rotate some concepts while +leaving others fixed — so the *shape* of the space is preserved. + +The canonical illustration: $\text{king} - \text{man} + \text{woman} \approx \text{queen}$. This +vector arithmetic works not because "king" lives at a special absolute address, but +because the training corpus encodes a consistent geometric offset between gendered +role-pair concepts. Both Model A and Model B, having trained on the same language, +construct the same parallelogram in their respective (rotated) frames. The analogical +relationship is not a coincidence of one model's random initialisation; it is a +property of the data. + +The transition key (§3) exploits this directly. By recording the *sequence of +semantic directions visited* during inference — the topological trajectory through +the embedding space — rather than any absolute coordinate, it indexes shape rather +than position. Two models reading the same passage may produce entirely different +float32 vectors at each layer, yet execute the same sequence of semantic turns: the +same "hairpin" at the subordinate clause, the same "commitment" at the topic word. +The run-reduced key captures that shared structure without requiring knowledge of $Q$. + +The generalized framework makes the lingua franca case stronger. The transition key captures relational structure that is invariant under rotation. This invariance is not an accident of the semantic embedding application — it is a consequence of the general framework: structural quantization (§2.4) preserves relational geometry by design. The embedding application is a special case where the rotation $Q$ corresponds to the arbitrary coordinate frame of a trained model. + +Embeddings across models are not fully incommensurable. Their coordinate frames are; +their semantic geometry is not. The gap between those two facts is where Q2 operates. + +$$\underbrace{\text{king} - \text{man} + \text{woman}}_{\text{vector arithmetic on any model}} \approx \underbrace{\text{queen}}_{\text{same answer, rotated frame}}$$ + +The coordinate frames differ by $Q$. The parallelogram does not. Viewed from the +right angle, the incommensurability dissolves. 🤓 + +--- + +## References + +- Hammons, A. R., Kumar, P. V., Calderbank, A. R., Sloane, N. J. A., & Solé, P. (1994). The $\mathbb{Z}_4$-linearity of Kerdock, Preparata, Goethals, and related codes. *IEEE Trans. Inform. Theory* 40:2, 301–319. +- Wildberger, N. J. & Rubine, D. (2025). A Hyper-Catalan Series Solution to Polynomial Equations, and the Geode. *Amer. Math. Monthly* 132:5, 383–402. DOI: 10.1080/00029890.2025.2460966 + +--- + [^1]: The $1/n_s$ variance normalisation ensures the dot product of two random unit vectors has bounded variance regardless of dimension. This is the same normalisation used in transformer attention ($1/\sqrt{d_k}$). diff --git a/PREDICTIONS.md b/PREDICTIONS.md index c95cdf5..762e6b3 100644 --- a/PREDICTIONS.md +++ b/PREDICTIONS.md @@ -23,6 +23,9 @@ Section references of the form §T-x refer to [TESTING.md](TESTING.md). - [P12 — Regime diagnostic: semantic distance vs. conceptual distance](#p12--regime-diagnostic-semantic-distance-vs-conceptual-distance) - [P13 — The regime dial and the sɪ test](#p13--the-regime-dial-and-the-s-test) - [P14 — Phylomemetic fingerprinting](#p14--phylomemetic-fingerprinting) +- [P15 — Admissible sequence count and trie sparsity](#p15--admissible-sequence-count-and-trie-sparsity) +- [P16 — Geode progressive quantization](#p16--geode-progressive-quantization) +- [P17 — Mixed-precision adaptive quantization](#p17--mixed-precision-adaptive-quantization) - [Summary table](#summary-table) --- @@ -484,14 +487,14 @@ Specifically: or map bitwise differences through a per-symbol LUT/SIMD transform before summing. - The information gain per symbol from $\mathbb{Z}_4$ to $\mathbb{Z}_8$ is $\log_2 8 - \log_2 4 = 1$ bit, but the additional bit encodes intra-cell position - within the four magnitude classes, which §D-1.6 shows is not recoverable signal + within the four magnitude classes, which §D-1.5 shows is not recoverable signal for L1 retrieval. **Falsification condition.** If a $\mathbb{Z}_8$-encoded system with otherwise identical architecture achieves a statistically significant improvement in retrieval on a held-out benchmark, the $\mathbb{Z}_4$ optimality prediction is falsified. This would imply that the fifth-through-eighth quantization cells carry retrievable semantic -signal, contradicting §D-1.6. +signal, contradicting §D-1.5. Test protocol: §T-3 (embedding models). @@ -822,6 +825,131 @@ Test protocol: §T5 (Gutenberg + AI corpus, new phase). --- +## P15 — Admissible sequence count and trie sparsity + +The no-repeat constraint in the transition sequence (§D-3.1) eliminates the vast +majority of possible 64-bit keys. The number of admissible transition sequences of +length $k$ over a $q$-ary alphabet is: + +$$D(k) = q \cdot (q-1)^{k-1}$$ + +For $q = 4$ and $k = 32$ (the Q² key capacity): + +$$D(32) = 4 \cdot 3^{31} \approx 2.47 \times 10^{15}$$ + +This occupies $\approx 1.34 \times 10^{-4}$ of the $2^{64}$ address space — the +no-repeat constraint concentrates valid keys into a sparse subset comprising less than +0.014% of all possible 64-bit values. + +**Prediction.** The empirical key distribution of a real corpus occupies a strict +subset of the admissible trie. Specifically: + +1. **Trie sparsity**: for any corpus of size $C \leq 10^{10}$, the fraction of the + admissible trie occupied is $C / D(32) \leq 4.05 \times 10^{-6}$ — the corpus is + sparse even within the already-sparse admissible set. +2. **Non-uniform trie occupancy**: certain subtrees of the transition trie are + disproportionately occupied because the source distribution favours certain symbol + transitions over others (cf. complement-bigram suppression, P3). The occupied + subtree structure is a fingerprint of the source distribution. +3. **Information efficiency**: a 32-symbol key carries $2 + 31 \times \log_2 3 \approx + 51.1$ effective bits of information within its 64-bit container, giving an + efficiency of $\approx 79.9\%$. The remaining $\approx 12.9$ bits are "wasted" by + the no-repeat constraint — but this waste is the cost of the run-reduction that + makes the key a topological invariant (§D-3.1, Lemma 3.3). + +**Falsification condition.** Empirical key distributions do not respect the trie +structure — i.e., keys appear that violate the no-repeat constraint, indicating a bug +in the run-reduction step. + +Test protocol: §T-0 (unit tests), §T-1 (null baseline), §T-6 (general quantization). + +--- + +## P16 — Geode progressive quantization + +The Geode factorization (§D-4.1): + +$$S - 1 = S_1 \cdot G$$ + +predicts that the transition key has a natural multi-resolution structure: the first +$j$ symbols form a coarse address, and the remaining symbols refine within that coarse +cell. The Geode $G = 1/(1 - 3x)$ counts the refinement possibilities: $3^{k}$ +distinct continuations of length $k$ after the first symbol is fixed. + +**Prediction.** Multi-resolution retrieval using the transition key exhibits +diminishing information gain per additional symbol: + +1. **Logarithmic decay of information per symbol**: the first symbol contributes + $\log_2 4 = 2$ bits; each subsequent symbol contributes $\log_2 3 \approx 1.585$ + bits. The cumulative information at depth $j$ is: + + $$I(j) = 2 + (j-1) \cdot \log_2 3 \approx 2 + 1.585(j-1) \text{ bits}$$ + +2. **Retrieval recall is a step function of prefix depth**: window queries at + successively deeper prefix lengths $j$ should show diminishing returns in + precision improvement. The transition from "broad" to "narrow" retrieval occurs at + a characteristic depth $j^*$ that depends on corpus density, not on the key + structure itself. + +3. **The Geode coefficient predicts candidate-set size**: a window query at prefix + depth $j$ retrieves at most $3^{32-j}$ candidates from the admissible trie. In + practice the candidate set is smaller (most of the admissible trie is unoccupied), + but the Geode coefficient provides the tight upper bound. + +**Concrete test.** For a corpus of $C$ documents, measure retrieval precision and +candidate-set size as a function of prefix depth $j \in \{4, 8, 12, 16, 20, 24, 28, 32\}$. +The candidate-set size should decrease geometrically as $O(3^{-j})$ while precision +increases monotonically. The crossover point where precision plateaus is $j^*$. + +**Falsification condition.** Information gain per symbol is non-monotonic or +candidate-set sizes do not follow the predicted $3^{32-j}$ bound within the admissible +trie. + +Test protocol: §T-3 (embedding models), §T-6 (general quantization). + +--- + +## P17 — Mixed-precision adaptive quantization + +The hyper-Catalan framework (§D-4.3) predicts that the number of structurally +distinct mixed-precision codebooks — allocating different bit-widths to different +dimensions — is finite and computable. For a signal with $m_2$ binary-quantized +dimensions, $m_3$ ternary-quantized dimensions, and $m_4$ quaternary-quantized +dimensions, the count of distinct mixed-precision trees is: + +$$C_{(m_2, m_3, m_4)} = \frac{1}{N} \binom{N}{n_0,\ m_2,\ m_3,\ m_4}$$ + +where $N = 1 + 2m_2 + 3m_3 + 4m_4$ is the total node count and $n_0 = 1 + m_2 + 2m_3 + 3m_4$ is the leaf count. + +**Prediction.** Adaptive bit allocation guided by per-dimension variance outperforms +uniform quaternary quantization on retrieval benchmarks when the source distribution +has heterogeneous variance across dimensions: + +1. **Variance-guided allocation**: dimensions with variance below a threshold $\sigma^*$ + should be quantized at 1 bit (binary: sign only); dimensions above $\sigma^*$ should + retain 2-bit (quaternary) precision. The optimal $\sigma^*$ minimises a loss + function trading off code length against retrieval accuracy. + +2. **Compression improvement**: for a typical transformer embedding with heterogeneous + per-dimension variance, mixed-precision encoding achieves the same retrieval accuracy + as uniform quaternary at a 15–25% shorter code length, or higher accuracy at the + same code length. + +3. **Diminishing returns for ternary inclusion**: adding ternary ($\mathbb{Z}_3$) cells + for near-zero-variance dimensions offers $\leq 0.085$ bits per dimension improvement + over binary (since $\log_2 3 - 1 \approx 0.585$ bits, attenuated by the low variance + of those dimensions). The ternary option should be dominated by binary for dimensions + with variance below $\sigma^*/2$. + +**Falsification condition.** Variance-guided mixed-precision encoding does not improve +over uniform quaternary encoding on any of the retrieval benchmarks, even for models +with demonstrably heterogeneous per-dimension variance (as measured by the coefficient +of variation across dimensions). + +Test protocol: §T-3 (embedding models), §T-6 (general quantization). + +--- + ## Summary table | ID | Prediction | From | Tested in | Effort | @@ -843,9 +971,15 @@ Test protocol: §T5 (Gutenberg + AI corpus, new phase). | P14b | AI model stylometric entropy < median human-author entropy; compression scales with RLHF intensity | RLHF variance compression | §T-5 | Low | | P14c | Specific Gutenberg authors (Carroll, Shelley, Lovelace, Doyle) have influence coefficients $\alpha_i$ disproportionate to corpus share in AI output | Cross-lineage influence detection | §T-5 | High | | P14d | Recovered influence coefficients respect temporal ordering of authors by publication date | Memetic causality constraint | §T-5 | Medium | +| P15 | Admissible trie has $\approx 2.47 \times 10^{15}$ nodes; real corpora occupy $< 10^{-5}$ of it; key efficiency $\approx 79.9\%$ | Transition trie counting (§D-3.6, §D-4.1) | §T-0, §T-1, §T-6 | Low | +| P16 | Multi-resolution retrieval shows logarithmic information decay per symbol; candidate sets shrink as $O(3^{-j})$ per Geode coefficient | Geode factorization (§D-4.1) | §T-3, §T-6 | Medium | +| P17 | Variance-guided mixed-precision (binary + quaternary) matches uniform quaternary retrieval at 15–25% shorter code | Hyper-Catalan mixed-arity counting (§D-4.3) | §T-3, §T-6 | Medium | P3 requires only a frequency count on quantizer output and can be run immediately once the quantizer from [PR #9](https://github.com/devlux76/q2/pull/9) is merged. P2 and P5 require a retrieval benchmark but no new infrastructure. P4, P6, and P9 require benchmark-calibrated metric variants. P7 requires a Nussinov dynamic programming -implementation over transition sequences. +implementation over transition sequences. P15 is verifiable from the existing unit test +suite and null baselines. P16 and P17 require multi-resolution retrieval benchmarks +and variance analysis, respectively, and are tested alongside the existing embedding +benchmarks in §T-3 and the new general quantization phase §T-6. diff --git a/TESTING.md b/TESTING.md index c95dcd4..689d8ce 100644 --- a/TESTING.md +++ b/TESTING.md @@ -16,6 +16,7 @@ Section references of the form §P-x refer to [PREDICTIONS.md](PREDICTIONS.md). - [T3 — Matryoshka and dedicated embedding models](#t3--matryoshka-and-dedicated-embedding-models) - [T4 — Standard local LLMs](#t4--standard-local-llms) - [T5 — Phylomemetic fingerprinting](#t5--phylomemetic-fingerprinting) +- [T6 — General quantization benchmarks](#t6--general-quantization-benchmarks) - [Corpus layout](#corpus-layout) - [Cross-phase prediction matrix](#cross-phase-prediction-matrix) - [Model and corpus matrix](#model-and-corpus-matrix) @@ -24,11 +25,13 @@ Section references of the form §P-x refer to [PREDICTIONS.md](PREDICTIONS.md). ## Overview -Testing is organized into an initial pre-phase **T0** and five main phases **T1–T5**. +Testing is organized into an initial pre-phase **T0** and six main phases **T1–T6**. The phases progress from the least constrained (random text, no semantic structure) to the most constrained (code, where ground truth is machine-verifiable), then broaden to measure what the encoding adds over current best-practice embedding -models, and finally test stylometric fingerprinting and RLHF entropy compression. +models, test stylometric fingerprinting and RLHF entropy compression, and finally +validate the general quantization framework predictions (P15–P17) that arise from the +Wildberger-Rubine combinatorial theory. The expected performance ordering across phases is: @@ -49,6 +52,7 @@ flowchart TD T3["T3 — Matryoshka &\ndedicated embedding models\n(BEIR benchmark)"] T4["T4 — Standard local LLMs\n(Qwen3.5-0.8B-ONNX)"] T5["T5 — Phylomemetic fingerprinting\n(P14a–d: author attribution\n& RLHF entropy compression)"] + T6["T6 — General quantization\nbenchmarks\n(P15–P17: trie sparsity,\nGeode progressive, mixed-precision)"] T0 -->|"null baselines\nestablished"| T1 T1 -->|"validate against\nnull"| T2 @@ -58,6 +62,8 @@ flowchart TD T3 -->|"P2, P3, P5, P7"| T4 T4 -->|"P14a–d"| T5 T3 -->|"stylometric baselines\nP8 null distributions"| T5 + T3 -->|"P15, P16, P17\ntrie & resolution\nbenchmarks"| T6 + T0 -->|"P15 trie\ninvariants"| T6 style T0 fill:#ddf,stroke:#99c style T1 fill:#ffd,stroke:#cc9 @@ -65,6 +71,7 @@ flowchart TD style T3 fill:#fdd,stroke:#c99 style T4 fill:#eee,stroke:#999 style T5 fill:#ede,stroke:#9a9 + style T6 fill:#fed,stroke:#ca6 ``` ## T0 — Unit tests and invariants @@ -379,6 +386,66 @@ signal is not present in Q² statistics. --- +## T6 — General quantization benchmarks + +**Purpose.** Validate the predictions (P15–P17) that arise from the generalized +quaternary quantization framework and the Wildberger-Rubine combinatorial theory +(DESIGN.md §4). These tests are domain-agnostic: they measure properties of the +quantization machinery itself, independent of whether the input is a semantic +embedding, a model weight tensor, or a synthetic signal. + +**Models.** Any model producing hidden-state activations suitable for Q² encoding. +The primary models from T3 and T4 are reused: + +- `nomic-ai/nomic-embed-text-v1.5` (embedding model, T3) +- `onnx-community/Qwen3.5-0.8B-ONNX` (general LLM, T4) +- Synthetic signals (Gaussian, mixture-of-Gaussians, uniform, heavy-tailed) for + distribution-independent validation. + +**Corpus.** Reuses C0 (synthetic), C1 (general NL), and C3 (BEIR) from earlier phases. +Additionally generates synthetic continuous signals (corpus C9) for +distribution-controlled experiments. + +### C9 — Synthetic continuous signals (~0 MB) + +Generated at test time from PRNG seeds; nothing stored on disk. Each sample is a +vector in $\mathbb{R}^n$ drawn from a specified distribution: + +- **Gaussian**: $\mathcal{N}(0, I_n/n)$ — the standard calibration distribution. +- **Mixture-of-Gaussians**: $\sum_k \pi_k \mathcal{N}(\mu_k, \sigma_k^2 I_n/n)$ — tests + analytical threshold computation (§D-4.4). +- **Uniform**: $\text{Uniform}[-1/\sqrt{n}, 1/\sqrt{n}]^n$ — L2-normalised. +- **Heavy-tailed**: Student-$t$ with $\nu \in \{3, 5, 10\}$ degrees of freedom — + tests robustness of the quantization framework to non-Gaussian tails. + +**Prediction sub-tests.** + +| Prediction | Measurement | Expected result | +|:----------:|:------------|:----------------| +| §P-15 (trie sparsity) | Count distinct transition sequences in a corpus of $C$ documents; verify all lie within the admissible trie of $D(32) \approx 2.47 \times 10^{15}$ nodes | 100% of keys are admissible; occupancy $\leq C / D(32)$ | +| §P-15 (non-uniform occupancy) | Compute subtree occupancy by first symbol and first bigram; compare to uniform expectation | Subtree occupancy is non-uniform; mirrors complement-bigram suppression (P3) | +| §P-15 (information efficiency) | Compute empirical entropy of the 64-bit key distribution and compare to the theoretical $\approx 51.1$ effective bits | Empirical entropy $\leq 51.1$ bits; gap measures source-distribution redundancy | +| §P-16 (progressive retrieval) | Measure retrieval precision and candidate-set size as a function of prefix depth $j \in \{4, 8, 12, 16, 20, 24, 28, 32\}$ | Candidate sets shrink geometrically as $O(3^{-j})$; precision increases monotonically | +| §P-16 (information per symbol) | Compute mutual information between prefix of length $j$ and full key, for $j = 1, \ldots, 32$ | First symbol contributes $\approx 2$ bits; each subsequent $\approx 1.585$ bits | +| §P-17 (mixed-precision) | Implement variance-guided bit allocation; compare retrieval accuracy to uniform quaternary at matched code length | Mixed-precision matches or exceeds uniform at 15–25% shorter code | +| §P-17 (threshold validation) | For mixture-of-Gaussian signals, compare empirical thresholds to hyper-Catalan series truncated at order 1, 2, 3 | Series converges to empirical threshold; order-2 truncation suffices for 4-component mixtures | + +**T6 benchmarks are primarily computational, not corpus-dependent.** The trie +sparsity and information-efficiency measurements (P15) require only the Q² encoder +and a set of input vectors. The progressive retrieval measurements (P16) require a +retrieval index over an existing corpus. The mixed-precision measurements (P17) +require a modified encoder that supports per-dimension bit allocation. The threshold +validation uses synthetic signals only. + +**Falsification conditions.** If any generated key violates the no-repeat constraint, +the run-reduction implementation has a bug (P15). If candidate-set sizes do not +decrease geometrically with prefix depth (P16), the transition key does not behave as +a multi-resolution address. If mixed-precision encoding cannot match uniform encoding +at shorter code length for any model (P17), per-dimension variance heterogeneity is +insufficient to exploit. + +--- + ## Corpus layout The complete corpus stays comfortably under 1 GB of raw text, leaving headroom for @@ -604,6 +671,7 @@ a cross-temporal comparison against C6 (cross-lingual) and C7 (spoken). | C6 | Cross-lingual matched-content corpus | ~75–100 MB | | C7 | Spoken vs. written language corpus | ~50 MB | | C8 | Gutenberg authors + AI attributed datasets | ~150 MB | +| C9 | Synthetic continuous signals | ~0 MB | | **Total** | | **~885–960 MB** | This leaves comfortable headroom for indices and probe corpora within the 1 GB budget. @@ -612,24 +680,27 @@ This leaves comfortable headroom for indices and probe corpora within the 1 GB b ## Cross-phase prediction matrix -| Prediction | T1 | T2 | T3 | T4 | T5 | -|:----------:|:--:|:--:|:--:|:--:|:--:| -| §P-2 Hairpin density ordering | Null baseline | Hairpin in call-and-return code | Full probe corpus test | Noisy probe test | — | -| §P-3 CpG suppression | Null baseline | Confirm or falsify | Confirm at scale | Partial test | — | -| §P-4 Weighted Lee | — | — | Full benchmark | — | — | -| §P-5 Reverse complement = antonym | — | — | Full antonym test | Partial test | — | -| §P-6 Two-stage search | — | — | Latency-matched benchmark | — | — | -| §P-7 Secondary structure | — | — | Correlation with annotation | Correlation (noisier) | — | -| §P-8 Codon usage bias | Null distribution | Domain-specific biases | Multi-domain comparison | — | — | -| §P-9 Z₈ optimality | — | — | Z₄ vs. Z₈ benchmark | — | — | -| §P-10 Key collision rate | — | Collision-rate benchmark | Collision-rate benchmark | — | — | -| §P-14a Author fingerprint stability | — | — | — | — | Within vs. cross-author CV | -| §P-14b RLHF entropy compression | — | — | — | — | RLHF CV < human CV | -| §P-14c Cross-lineage influence | — | — | — | — | Author clustering | -| §P-14d Temporal ordering | — | — | — | — | Influenced → earlier source | +| Prediction | T1 | T2 | T3 | T4 | T5 | T6 | +|:----------:|:--:|:--:|:--:|:--:|:--:|:--:| +| §P-2 Hairpin density ordering | Null baseline | Hairpin in call-and-return code | Full probe corpus test | Noisy probe test | — | — | +| §P-3 CpG suppression | Null baseline | Confirm or falsify | Confirm at scale | Partial test | — | — | +| §P-4 Weighted Lee | — | — | Full benchmark | — | — | — | +| §P-5 Reverse complement = antonym | — | — | Full antonym test | Partial test | — | — | +| §P-6 Two-stage search | — | — | Latency-matched benchmark | — | — | — | +| §P-7 Secondary structure | — | — | Correlation with annotation | Correlation (noisier) | — | — | +| §P-8 Codon usage bias | Null distribution | Domain-specific biases | Multi-domain comparison | — | — | — | +| §P-9 Z₈ optimality | — | — | Z₄ vs. Z₈ benchmark | — | — | — | +| §P-10 Key collision rate | — | Collision-rate benchmark | Collision-rate benchmark | — | — | — | +| §P-14a Author fingerprint stability | — | — | — | — | Within vs. cross-author CV | — | +| §P-14b RLHF entropy compression | — | — | — | — | RLHF CV < human CV | — | +| §P-14c Cross-lineage influence | — | — | — | — | Author clustering | — | +| §P-14d Temporal ordering | — | — | — | — | Influenced → earlier source | — | +| §P-15 Trie sparsity | Null trie occupancy | — | Empirical trie occupancy | — | — | Full trie analysis + info efficiency | +| §P-16 Geode progressive | — | — | Progressive retrieval benchmark | — | — | Candidate-set scaling + info/symbol | +| §P-17 Mixed-precision | — | — | Variance-guided benchmark | — | — | Threshold validation on synthetics | Tests in the T1 column establish baseline distributions. A prediction is -**confirmed** when the T3 or T4 result is statistically significant relative to the +**confirmed** when the T3, T4, or T6 result is statistically significant relative to the T1 null. A prediction is **falsified** when the result is not distinguishable from the null at the 95% confidence level across at least two model/corpus combinations. @@ -653,8 +724,11 @@ the null at the 95% confidence level across at least two model/corpus combinatio | P14b (RLHF entropy compression) | C8 AI + human subcorpora | Triplet entropy on raw text; embedding models for transition sequences | | P14c (cross-lineage influence) | C8 AI + human subcorpora | Raw text stylometric features; mixture model fit | | P14d (temporal ordering) | C8 human subcorpus (with author dates) | Same as P14c | +| P15 (trie sparsity) | C0 (synthetic), C9 (synthetic signals), C1/C3 (NL) | All models; synthetic signals for distribution-controlled tests | +| P16 (Geode progressive) | C3 (BEIR), C9 (synthetic) | Nomic, EmbeddingGemma with multi-resolution retrieval | +| P17 (mixed-precision) | C3 (BEIR), C9 (synthetic) | Nomic, EmbeddingGemma with variance-guided bit allocation | A concrete model matrix with rows corresponding to `{MiniLM, Nomic, EmbeddingGemma, mxbai, UniXcoder, Qwen2, Qwen2.5-Coder}` and columns -corresponding to `{T1–T5, P2–P14}` clarifies exactly which model–phase combinations +corresponding to `{T1–T6, P2–P17}` clarifies exactly which model–phase combinations need to be run vs. which can be pruned for an MVP.