clean up evaluation and applications

content/08.evaluation.md

@@ -1,25 +1,83 @@
 ## Evaluation and Design Principles for Context-Adaptive Inference
 
-The two preceding sections described how adaptation is built, explicitly through a structured $f(c)$ and implicitly through emergent computation in large models. Both paradigms raise the same three practical questions once a model is deployed: how cheaply it adapts, how stably it routes a context to the right behavior, and how well that behavior holds up when the context shifts. This section collects the design and evaluation principles that answer these questions. 
+The two preceding sections described how adaptation is built, explicitly through a structured $f(c)$ and implicitly through emergent computation in large models. Both paradigms raise the same three practical questions once a model is deployed: how cheaply it adapts, how stably it routes a context to the right behavior, and how well that behavior holds up when the context shifts. This section collects the design and evaluation principles that answer these questions. For each section we pair the design principle with a way to measure it, and formalize it as a metric.
+
+### Measuring Context-Conditional Performance
+
+Every metric below builds on a single quantity: performance conditioned on context. Let $\mathcal{C}$ denote the context space and $\mathcal{D}_{\mathrm{test}}$ a test distribution over $(x, y, c)$. For a predictor $\hat{f}$, define the context-conditional risk as
+
+$$
+\mathcal{R}(\hat{f}\mid c)
+= \mathbb{E}\!\left[\, \ell\big(\hat{f}(x,c), y\big) \,\middle|\, c \,\right],
+\qquad
+\mathcal{R}(\hat{f})
+= \mathbb{E}_{c\sim \mathcal{D}_{\mathrm{test}}}\!\left[\, \mathcal{R}(\hat{f}\mid c) \,\right].
+$$
+
+A single aggregate risk hides exactly the heterogeneity these models exist to capture, so a context-stratified evaluation reports $\mathcal{R}(\hat{f}\mid c)$ across predefined bins or via a smoothed estimate $\int \mathcal{R}(\hat{f}\mid c)\,\mathrm{d}\Pi(c)$ for a reference measure $\Pi$ that weights regions of the context space. Evaluations for adaptation efficiency, routing stability, and context-specific robustness build on this conditional risk.
 
 ### Adaptation Efficiency
 
 The efficiency of a context-adaptive method hinges on design choices that trade computational tractability against statistical accuracy, and the same choices govern whether a method scales to large data while remaining interpretable.
 
 One central principle is the use of sparsity assumptions to limit the number of context-dependent parameters. This can be achieved through group sparsity, which encourages entire groups of parameters to be zero simultaneously [@doi:10.1111/j.1467-9868.2005.00532.x], hierarchical regularization that applies different strengths of shrinkage to varying levels of context specificity [@doi:10.1017/CBO9780511790942], and adaptive thresholding that adjusts sparsity levels in accordance with context complexity.
 
-Efficiency can also be enhanced through computational strategies that allocate resources adaptively. Early stopping terminates optimization for contexts where convergence occurs rapidly [@doi:10.48550/arXiv.1606.04838], while context-dependent sampling employs different sampling schemes across contexts [@doi:10.48550/arXiv.1809.09582]. Caching and warm-starting further accelerate optimization by reusing solutions from similar contexts, which is particularly effective when contexts vary smoothly [@doi:10.1561/2200000016].
+Efficiency can also be enhanced through computational strategies that allocate resources adaptively. Early stopping terminates optimization for contexts where convergence occurs rapidly [@doi:10.48550/arXiv.1606.04838], while context-dependent sampling employs different sampling schemes across contexts [@doi:10.48550/arXiv.1809.09582]. Caching and warm-starting further accelerate optimization by reusing solutions from similar contexts, which is particularly effective when contexts vary smoothly [@doi:10.1561/2200000016]. Related context-dependent computation appears in adaptive batching, per-context learning rates, and multi-fidelity pipelines that allocate compute and precision by context complexity.
+
+These efficiency choices also interact with interpretability: sparse, linear context maps stay auditable, while amortized encoders trade transparency for scale, a trade-off we return to in the interpretability section.
+
+To make these gains measurable, let $S_k(c)=\{(x_j, y_j, c)\}_{j=1}^k$ denote $k$ examples available within context $c$, and define the adaptation efficiency as
+
+$$
+\mathrm{AE}_k(c)
+= \mathcal{R}(\hat{f}_0 \mid c)
+- \mathcal{R}(\hat{f}_{S_k} \mid c),
+\qquad
+\mathrm{AE}_k
+= \mathbb{E}_{c}\!\left[\, \mathrm{AE}_k(c) \,\right],
+$$
+
+where $\hat{f}_0$ is the non-adapted baseline and $\hat{f}_{S_k}$ the adapted predictor; the curve $k \mapsto \mathrm{AE}_k$ summarizes few-shot adaptation gains across context sizes. A closely related quantity measures how much of this efficiency carries between contexts. For a shared representation $\phi$ transferred from source to target contexts $\mathcal{C}_{\mathrm{src}} \to \mathcal{C}_{\mathrm{tgt}}$, the transfer gain is
 
-A further consideration is the balance between efficiency and interpretability. Linear context functions are highly interpretable but may require many parameters, while explicit context encodings improve transparency at the potential cost of higher computational overhead, and local context modeling provides fine-grained interpretability but can be less scalable. These trade-offs should be weighed against application-specific requirements. Advanced adaptive optimizers such as Adam can efficiently train complex, nonlinear models, but the resulting systems may be less interpretable than simpler alternatives [@doi:10.48550/arXiv.1412.6980]. In practice, context-dependent computation of this kind appears in adaptive batching, per-context learning rates, and multi-fidelity optimization pipelines that adjust compute and precision according to context complexity.
+$$
+\mathrm{TG}(\phi)
+= \mathcal{R}_{\mathcal{C}_{\mathrm{tgt}}}\!\big(\hat{f}_{\mathrm{scratch}}\big)
+- \mathcal{R}_{\mathcal{C}_{\mathrm{tgt}}}\!\big(\hat{f}_{\phi}\big),
+$$
+
+the risk reduction from transferring $\phi$ rather than training from scratch, so that positive values indicate useful transfer, matching the sign convention of $\mathrm{AE}_k$. This is the same efficiency argument that motivates the amortized context encoders of the previous sections.
 
 ### Routing Stability
 
-Both paradigms decide, for a given context, which parameters or which experts govern the prediction. Explicit models make this decision overtly, through the partition splits and hierarchical routing of the discrete rung, where a boundary in context space sends a sample to one local model rather than another. Implicit models make the same decision internally, through the gating of a mixture of experts or the attention weights of a transformer. In either case the decision is a function of context, and its stability is a property worth measuring: small perturbations of the context should not cause erratic changes in which behavior is selected, or predictions become unpredictable near routing boundaries.
+Both paradigms decide, for a given context, which parameters or which experts govern the prediction. Explicit models make this decision overtly, through the partitions or context encoders that map context to model parameters. Implicit models make the same decision internally, through the gating of a mixture of experts or the attention weights of a transformer. In either case the decision is a function of context, and its stability is a property worth measuring: small perturbations of the context should not cause erratic changes in which behavior is selected, or predictions become unpredictable near routing boundaries.
+
+Instability is easiest to see at the boundaries. A partition model can flip a sample between adjacent leaves under a negligible change in a splitting variable, and an in-context learner can change its answer under a reordering or reformatting of the same prompt examples, a sensitivity documented for in-context learning where the distribution and arrangement of examples can matter more than their content [@doi:10.48550/arXiv.2202.12837]. Reporting routing stability therefore means probing the model with perturbations that should be behavior-preserving, such as reordering exchangeable examples, jittering a continuous context near a learned split, or relabeling an equivalent measurement policy, and measuring how much the selected behavior and the resulting prediction move. Smoothly routed models degrade gracefully across these perturbations; sharply partitioned or brittle ones do not, and the difference is a design choice as much as a diagnostic.
+
+This admits a single measure across both discrete and continuous perturbations. Let $\rho(c)$ denote the routing decision the model exposes, whether gate weights, a leaf assignment, or the adapted parameters $\theta(c)$, and let $\Pi_c$ be a family of behavior-preserving perturbations of context $c$. The routing stability at $c$ is the expected movement of that decision under perturbation,
+
+$$
+\mathrm{RSt}(c)
+= \mathbb{E}_{\pi \sim \Pi_c}\,
+d\big(\rho(c),\, \rho(\pi(c))\big),
+$$
 
-Instability is easiest to see at the boundaries. A partition model can flip a sample between adjacent leaves under a negligible change in a splitting variable, and an in-context learner can change its answer under a reordering or reformatting of the same prompt examples, a sensitivity documented for in-context learning where the distribution and arrangement of examples can matter more than their content [@doi:10.48550/arXiv.2202.12837]. Reporting routing stability therefore means probing the model with controlled perturbations of the context that should be behavior-preserving, such as reordering exchangeable examples, jittering a continuous context near a learned split, or relabeling an equivalent measurement policy, and measuring how much the selected behavior and the resulting prediction move. Smoothly routed models degrade gracefully across these perturbations; sharply partitioned or brittle ones do not, and the difference is a design choice as much as a diagnostic.
+with a supremum over $\Pi_c$ giving the worst-case variant. Discrete invariances such as reordering exchangeable examples enter through $\Pi_c$ directly; for continuous perturbations, normalizing by the context displacement $D(c, \pi(c))$ recovers a local Lipschitz constant of the routing map, which is the smoothness reading of stability for continuous contexts.
 
 ### Context-Specific Robustness
 
-The final axis asks whether adaptation still helps when the context distribution at test time differs from training. For explicit models, the sharpest version of this question arises when context is the pattern of observed measurements, because the deployment environment routinely presents measurement policies that were rare or absent during training. Evaluation of missingness-as-context models should report mask-stratified metrics, including worst-group performance, following group-robust evaluation practice [@doi:10.48550/arXiv.1911.08731; @doi:10.48550/arXiv.2012.07421]. Robustness should be probed with mask-shift stress tests, training under one measurement policy and testing under another, to quantify degradation and the benefit of contextualization, as formalized in the Domain Adaptation under Missingness Shift (DAMS) setting [@doi:10.48550/arXiv.2211.02093; @doi:10.48550/arXiv.2012.07421]. When imputation is used, authors should assess imputation realism by holding out observed entries under realistic mask distributions and reporting MAE/RMSE and calibration for $p(x_{\text{missing}}\mid x_{\text{observed}})$ [@doi:10.48550/arXiv.1806.02382; @doi:10.48550/arXiv.1806.02920]. For causal or estimation applications, conduct ignorability sensitivity analyses, contrasting MAR-based results with pattern-mixture or selection-model analyses under plausible MNAR mechanisms [@doi:10.2307/2337120; @doi:10.48550/arXiv.2301.05043]. Finally, include ablations that remove mask or indicator inputs, and, for trees, disable default-direction routing, to confirm that gains derive from modeling the mask signal rather than from artifacts [@doi:10.48550/arXiv.1603.02754; @doi:10.48550/arXiv.2211.09259]. Practical implementations of these ideas are widely available: GRU-D [@doi:10.48550/arXiv.1606.01865] and BRITS [@doi:10.48550/arXiv.1805.10572] provide mask- and time-aware sequence models, while GAIN [@doi:10.48550/arXiv.1806.02920] and VAEAC [@doi:10.48550/arXiv.1806.02382] offer open-source code for imputation under arbitrary masks. For tree ensembles, XGBoost supports sparsity-aware default-direction splits, making it straightforward to treat missing values as context without preprocessing [@doi:10.1145/2939672.2939785].
+The final major evaluation type asks whether adaptation still helps when the context distribution at test time differs from training. Context shift takes many forms, including new subpopulations, drifting covariates, and unseen measurement policies. For explicit models it is easiest to make concrete when context is the pattern of observed measurements, since the deployment environment routinely presents measurement policies that were rare or absent during training; we use missingness-as-context as the running example. Evaluation of missingness-as-context models should report mask-stratified metrics, including worst-group performance, following group-robust evaluation practice [@doi:10.48550/arXiv.1911.08731; @doi:10.48550/arXiv.2012.07421]. Robustness should be probed with mask-shift stress tests, training under one measurement policy and testing under another, to quantify degradation and the benefit of contextualization, as formalized in the Domain Adaptation under Missingness Shift (DAMS) setting [@doi:10.48550/arXiv.2211.02093; @doi:10.48550/arXiv.2012.07421]. When imputation is used, authors should assess imputation realism by holding out observed entries under realistic mask distributions and reporting MAE/RMSE and calibration for $p(x_{\text{missing}}\mid x_{\text{observed}})$ [@doi:10.48550/arXiv.1806.02382; @doi:10.48550/arXiv.1806.02920]. For causal or estimation applications, conduct ignorability sensitivity analyses, contrasting MAR-based results with pattern-mixture or selection-model analyses under plausible MNAR mechanisms [@doi:10.2307/2337120; @doi:10.48550/arXiv.2301.05043]. Finally, include ablations that remove mask or indicator inputs, and, for trees, disable default-direction routing, to confirm that gains derive from modeling the mask signal rather than from artifacts [@doi:10.48550/arXiv.1603.02754; @doi:10.48550/arXiv.2211.09259]. Practical implementations of these ideas are widely available: GRU-D [@doi:10.48550/arXiv.1606.01865] and BRITS [@doi:10.48550/arXiv.1805.10572] provide mask- and time-aware sequence models, while GAIN [@doi:10.48550/arXiv.1806.02920] and VAEAC [@doi:10.48550/arXiv.1806.02382] offer open-source code for imputation under arbitrary masks. For tree ensembles, XGBoost supports sparsity-aware default-direction splits, making it straightforward to treat missing values as context without preprocessing [@doi:10.1145/2939672.2939785].
+
+Implicit models face the same issue in a different form. In-context learning adapts within a single forward pass, but that adaptation is opaque and can be brittle: performance is sensitive to small changes in the prompt, and reliability under distribution shift is hard to guarantee or to audit. Recent surveys frame the open questions as developing a more complete theoretical account of when in-context learning fails, improving its reliability, and establishing methods for controlling its behavior in high-stakes applications [@doi:10.48550/arXiv.2301.00234]. The evaluation logic carries over from the explicit case: stratify performance by the kind of context, stress-test under context shift, and ablate the adaptive signal to confirm that the model is using context as intended rather than exploiting an artifact. From this perspective, the mask-shift stress test for an explicit model and the prompt-perturbation test for an in-context learner are the same experiment applied to two realizations of $f(c)$.
+
+The degradation these tests reveal can be summarized in a single score. Let $Q$ denote a family of admissible context shifts, for example $f$-divergence or Wasserstein balls over context marginals; the robustness score
+
+$$
+\mathrm{RS}(\hat{f}; Q)
+= \sup_{\widetilde{\mathcal{D}}\in Q}
+\left[
+\mathcal{R}_{\widetilde{\mathcal{D}}}(\hat{f})
+- \mathcal{R}_{\mathcal{D}_{\mathrm{test}}}(\hat{f})
+\right]
+$$
 
-Implicit models face the same issue in a different form. In-context learning adapts within a single forward pass, but that adaptation is opaque and can be brittle: performance is sensitive to small changes in the prompt, and reliability under distribution shift is hard to guarantee or to audit. Recent surveys frame the open questions as developing a more complete theoretical account of when in-context learning fails, improving its reliability, and establishing methods for controlling its behavior in high-stakes applications [@doi:10.48550/arXiv.2301.00234]. The evaluation logic carries over from the explicit case: stratify performance by the kind of context, stress-test under context shift, and ablate the adaptive signal to confirm that the model is using context as intended rather than exploiting an artifact. Read this way, the mask-shift stress test for an explicit model and the prompt-perturbation test for an in-context learner are the same experiment applied to two realizations of $f(c)$.
+reports the worst-case excess risk over that family, with higher values indicating greater sensitivity to contextual change.