diff --git a/README.md b/README.md index 6c6c1be..6bd15b1 100644 --- a/README.md +++ b/README.md @@ -47,12 +47,13 @@ and opportunities in context-adaptive inference. 5. [Principles of Context-Adaptive Inference](content/05.principles.md) 6. [Explicit Adaptivity: Structured Estimation of $f(c)$](content/06.explicit.md) 7. [Implicit Adaptivity: Emergent Contextualization within Complex Models](content/07.implicit.md) -8. [Making Implicit Adaptivity Explicit: Local Models, Surrogates and Post Hoc Approximations](content/08.interpretations.md) -9. [Context-Invariant Training: A View from the Converse](content/09.invariant.md) -10. [Applications, Case Studies, and Software Tools](content/10.applications_tools.md) -11. [Future Trends and Opportunities with Foundation Models](content/11.future_trends.md) -12. [Open Problems](content/12.open_problems.md) -13. [Conclusions](content/13.conclusion.md) +8. [Evaluation and Design Principles for Context-Adaptive Inference](content/08.evaluation.md) +9. [Making Implicit Adaptivity Explicit: Local Models, Surrogates and Post Hoc Approximations](content/09.interpretations.md) +10. [Context-Invariant Training: A View from the Converse](content/10.invariant.md) +11. [Applications, Case Studies, and Software Tools](content/11.applications_tools.md) +12. [Future Trends and Opportunities with Foundation Models](content/12.future_trends.md) +13. [Open Problems](content/13.open_problems.md) +14. [Conclusions](content/14.conclusion.md) ## How can you contribute? We welcome contributions from the community. Please see our [contribution guidelines](CONTRIBUTING.md) for more information. diff --git a/content/03.overview.md b/content/03.overview.md index 90f06e3..4152302 100644 --- a/content/03.overview.md +++ b/content/03.overview.md @@ -67,13 +67,15 @@ Before diving into flexible estimators of $f(c)$, we review early modeling strat #### Conditional and Clustered Models -One approach is to group observations into C contexts, either by manually defining conditions (e.g. male vs. female) or using unsupervised clustering. The partition can also be learned in a supervised, model-based way: model-based recursive partitioning fits a parametric model and splits the covariate space wherever its coefficients show the strongest instability, recursing to produce interpretable subgroups, with toolkits such as partykit providing standard implementations of these conditional-inference and model-based trees [@doi:10.1198/106186008X319331; @hothorn2015partykit]. Each group is then assigned a distinct parameter vector: +One approach is to group observations into C contexts, either by manually defining conditions (e.g. male vs. female) or using unsupervised clustering. Each group is then assigned a distinct parameter vector: $$ \{\widehat{\theta}_0, \ldots, \widehat{\theta}_C\} = \arg\max_{\theta_0, \ldots, \theta_C} \sum_{c \in \mathcal{C}} \ell(X_c; \theta_c), $$ where $\ell(X; \theta)$ is the log-likelihood of $\theta$ on $X$ and $c$ specifies the covariate group that samples are assigned to. This reduces variance but limits granularity. It assumes that all members of a group share the same distribution and fails to capture variation within a group. +Partitions can also be learned in a supervised, model-based way: model-based recursive partitioning fits a parametric model and splits the covariate space wherever its coefficients show the strongest instability, recursing to produce interpretable subgroups, with toolkits such as partykit providing standard implementations [@doi:10.1198/106186008X319331; @hothorn2015partykit]. + These early methods relax global homogeneity yet still rely on discrete partitions, motivating smoother and more flexible formulations explored in the next sections. #### Distance-Regularized Estimation @@ -99,11 +101,11 @@ Semi-parametric VCMs relax the linearity assumption by requiring only that param ### Contextualized Models -Contextualized models take a fully non-parametric approach, introduced in [@doi:10.48550/arXiv.1705.10301]. They assume that parameters are functions of context, $f(c)$, but do not restrict the form of $f$. Instead, $f$ is estimated directly, often with deep neural networks as function approximators: +Contextualized models [@doi:10.48550/arXiv.2310.11340] take a fully non-parametric approach. They assume that parameters are functions of context, $f(c)$, but do not restrict the form of $f$. Instead, $f$ is estimated directly, often with deep neural networks as function approximators: $$ \widehat{f} = \arg \max_{f \in \mathcal{F}} \sum_i \ell(x_i; f(c_i)). $$ -This framework has been widely applied, from machine learning toolboxes [@doi:10.48550/arXiv.2310.11340; @doi:10.21105/joss.06469] to personalized genomics [@doi:10.1101/2023.12.01.569658; @doi:10.48550/arXiv.2111.01104], biomedical informatics [@doi:10.48550/arXiv.2310.07918; @doi:10.1016/j.jbi.2022.104086; @doi:10.1101/2020.06.25.20140053], and contextual feature selection [@doi:10.48550/arXiv.2312.14254]. These examples highlight how contextual signals can drive adaptation without assuming a fixed functional form. +This flexible approximation has enabled contextualized models to be applied to a wide range of across model types and application areas, including personalized disease analysis [@doi:10.1073/pnas.2411930122; @doi:10.48550/arXiv.2111.01104; @doi:10.1101/2020.06.25.20140053], heterogeneous treatment effect estimation [@doi:10.1016/j.jbi.2022.104086; @doi:10.48550/arXiv.2310.07918; @doi:10.48550/arXiv.2411.10645; @doi:10.48550/arXiv.1705.10301], drug development [@doi:10.64898/2026.05.11.724149], and contextual feature selection [@doi:10.48550/arXiv.2312.14254] and explainability [@doi:10.48550/arXiv.1705.10301]. These examples highlight how contextual signals can drive adaptation without assuming a fixed functional form. The contextualized.ml Python package provides standard implementations [@doi:10.21105/joss.06469]. ### Partition and Latent-Structure Models @@ -111,7 +113,8 @@ Partition models extend the contextualized framework by assuming that parameters $$ \{\widehat{\theta}_0, \dots, \widehat{\theta}_N\} = \arg\max_{\theta_0, \dots, \theta_N} \left( \sum_i \ell(x_i; \theta_i) + \lambda \sum_{i = 2}^N \|\theta_i - \theta_{i-1}\| \right). $$ -By encouraging piecewise-constant structures, partition models get closer to personalized modeling, balancing fit and parsimony, moving closer to personalized inference, trading off flexibility for interpretability. When the parameters being partitioned define a network, the support itself can jump across blocks, yielding varying-coefficient varying-structure models whose graph topology rewires with context, whether over time or along a branching biological lineage [@doi:10.1214/12-EJS739; @doi:10.1093/bioinformatics/btr239]. +By encouraging piecewise-constant structures, partition models get closer to personalized modeling, balancing fit and parsimony, moving closer to personalized inference, trading off flexibility for interpretability. +When the parameters being partitioned define a network, the support itself can jump across blocks, yielding varying-coefficient varying-structure models whose graph topology rewires with context, whether over time or along a branching biological lineage [@doi:10.1214/12-EJS739; @doi:10.1093/bioinformatics/btr239]. ### Fine-tuned Models and Transfer Learning diff --git a/content/06.explicit.md b/content/06.explicit.md index dd58b85..915a06e 100644 --- a/content/06.explicit.md +++ b/content/06.explicit.md @@ -2,141 +2,100 @@ In classical statistical modeling, all observations are typically assumed to share a common set of parameters. However, modern datasets often display significant heterogeneity across individuals, locations, or experimental conditions, making this assumption unrealistic in many real-world applications. To better capture such heterogeneity, recent approaches model parameters as explicit functions of observed context, formalized as $\theta_i = f(c_i)$, where $f$ maps each context to a sample-specific parameter [@doi:10.1111/j.2517-6161.1993.tb01939.x]. -A familiar example of explicit adaptivity is multi-task learning, where context is defined by task identity. -Traditional multi-task learning (left) assigns each task its own head on top of shared representations, -while context-flagged models (right) pass task identity directly as an input, enabling richer parameter sharing. -This illustrates how explicit conditioning on context variables can unify tasks within a single model and -provides an intuitive entry point to more general forms of explicit adaptivity (Figure {@fig:mtl-context}). +A familiar example of explicit adaptivity is multi-task learning, where context is defined by task identity. Traditional multi-task learning (left) assigns each task its own head on top of shared representations, while context-flagged models (right) pass task identity directly as an input, enabling richer parameter sharing. This illustrates how explicit conditioning on context variables can unify tasks within a single model and provides an intuitive entry point to more general forms of explicit adaptivity (Figure {@fig:mtl-context}). ![Multi-task learning as explicit adaptivity. In traditional MTL (left), each task has its own head on top of shared layers. In context-flagged models (right), the task identity is provided as an input, enabling a shared model to adapt across tasks.](images/mtl_context.png){#fig:mtl-context width="75%"} - -This section systematically reviews explicit adaptivity methods, with a focus on structured estimation of $f(c)$. We begin by revisiting classical varying-coefficient models, which provide a conceptual and methodological foundation for modeling context-dependent effects. We then categorize recent advances in explicit adaptivity according to three principal strategies for estimating $f(c)$: (1) smooth nonparametric models that generalize classical techniques, (2) structurally constrained models that incorporate domain-specific knowledge such as spatial or network structure, and (3) learned function approximators that leverage machine learning methods for high-dimensional or complex contexts. Finally, we summarize key theoretical developments and highlight promising directions for future research in this rapidly evolving field. - -### Classical Varying-Coefficient Models: A Foundation - -Varying-coefficient models (VCMs) are a foundational tool for modeling heterogeneity, as they allow model parameters to vary smoothly with observed context variables [@doi:10.1111/j.2517-6161.1993.tb01939.x; @doi:10.3390/publications13020019]. In their original formulation, the regression coefficients are treated as nonparametric functions of low-dimensional covariates, such as time or age. The standard VCM takes the form +The workhorse formalism for explicit adaptivity is the varying-coefficient model (VCM), which writes each regression coefficient as a function of context [@doi:10.1111/j.2517-6161.1993.tb01939.x; @doi:10.3390/publications13020019]: $$ -y_i = \sum_{j=1}^{p} \beta_j(c_i) x_{ij} + \varepsilon_i +y_i = \sum_{j=1}^{p} \beta_j(c_i)\, x_{ij} + \varepsilon_i . $$ -where each $\beta_j(c)$ is an unknown smooth function, typically estimated using kernel smoothing, local polynomials, or penalized splines. - -This approach provides greater flexibility than fixed-coefficient models and is widely used for longitudinal and functional data analysis. The assumption of smoothness makes estimation and theoretical analysis more tractable, but also imposes limitations. Classical VCMs work best when the context is low-dimensional and continuous. They may struggle with abrupt changes, discontinuities, or high-dimensional and structured covariates. In such cases, interpretability and accuracy can be compromised, motivating the development of a variety of modern extensions, which will be discussed in the following sections. - -### Advances in Modeling $f(c)$ +Every method in this section is, at its core, a different way to estimate the maps $\beta_j(\cdot)$, or more generally $f(\cdot)$. We organize them along an axis of increasingly sophisticated statistical and machine-learning concepts. Each step forward increases the power of context-specific, personalized inference by borrowing strength from related samples and groups. Read the other way, each step lowers the amount of data that must be collected for any single group, because information flows in from neighboring contexts rather than being estimated in isolation. The progression runs from subgroups that share nothing, through mechanisms that share progressively more, to fully learned functions of context: kernel localization that dissolves boundaries entirely, structured parametric maps that learn which context features matter, and finally contextualized models that learn arbitrary context dependencies. Running alongside this is a parallel case in which the models being adapted become increasingly sophisticated, going from linear models to graphical models whose structure, and not only its values, follows context. -Recent years have seen substantial progress in the modeling of $f(c)$, the function mapping context to model parameters. These advances can be grouped into three major strategies: (1) smooth non-parametric models that extend classical flexibility; (2) structurally constrained approaches that encode domain knowledge such as spatial or network topology; and (3) high-capacity machine learning methods for high-dimensional, unstructured contexts. Each strategy addresses specific challenges in modeling heterogeneity, and together they provide a comprehensive toolkit for explicit adaptivity. +### Independent Subgroups -#### Smooth Non-parametric Models +The most basic response to heterogeneity is to split the data into subgroups and fit each one on its own. Conditional and clustered models define groups by hand, for example by sex or site, or by unsupervised clustering, and estimate a separate parameter vector within each group with no sharing between them. This removes the bias of a single global fit, but it is the most data-hungry option available: each group is estimated only from its own samples, so small groups are estimated poorly and unseen groups not at all. Every later step can be read as a way to relax this isolation, buying more reliable per-group estimates from less per-group data. -This family of models generalizes the classical VCM by expressing $f(c)$ as a flexible, smooth function estimated with basis expansions and regularization. Common approaches include spline-based methods, local polynomial regression, and RKHS-based frameworks. For instance, developed a semi-nonparametric VCM using RKHS techniques for imaging genetics, enabling the model to capture complex nonlinear effects. Such methods are central to generalized additive models, supporting both flexibility and interpretability. Theoretical work has shown that penalized splines and kernel methods offer strong statistical guarantees in moderate dimensions, although computational cost and overfitting can become issues as the dimension of $c$ increases. These estimators occupy the lower-capacity but more interpretable end of the explicit adaptivity spectrum, forming a conceptual baseline for more complex architectures discussed below. +### Sharing Across Groups -#### Structured Regularization for Graphical and Network Models +The first improvement keeps discrete groups but couples their estimates. Distance-regularized estimation asks that observations with similar contexts have similar parameters, penalizing differences in $\theta_i$ in proportion to context distance, while fused-lasso and total-variation penalties shrink differences between adjacent groups so that estimates borrow strength from their neighbors [@doi:10.1214/09-AOAS308]. The effect is to lower the effective data requirement per group: a sparsely observed context inherits information from well-observed relatives rather than standing alone. -The origins of structurally constrained models can be traced to early work on covariance selection. Dempster (1972) demonstrated that zeros in the inverse covariance matrix correspond directly to conditional independencies, introducing the principle that sparsity reflects structure [@doi:10.2307/2528966]. This principle was formalized in Lauritzen’s (1996) influential monograph, which systematized probabilistic graphical models and showed how independence assumptions can be embedded into estimation procedures [@doi:10.1093/oso/9780198522195.001.0001]. Together, these works established the conceptual foundation that explicit structure can guide inference in high-dimensional settings. +The same idea carries over from regressors to network estimators. The graphical-model lineage established that a parameter can be an estimated network: zeros in the inverse covariance matrix encode conditional independencies [@doi:10.2307/2528966; @doi:10.1093/oso/9780198522195.001.0001], and neighborhood selection and the graphical lasso recover sparse structure directly from data [@doi:10.1214/009053606000000281; @doi:10.1093/biostatistics/kxm045]. Sharing then couples networks across a discrete set of conditions. Guo et al. jointly estimate several graphical models, encouraging sparsity within each while borrowing strength across related groups [@doi:10.1093/biomet/asq060], and the Joint Graphical Lasso balances shared structure against group-specific edges across populations [@doi:10.1111/rssb.12033]. Bayesian formulations achieve the same pooling through priors rather than penalties: lattice and Markov-random-field spike-and-slab priors learn when edges should be shared across neighboring sites or sample groups, and quantify how similar the resulting networks are [@doi:10.1198/jasa.2011.tm10465; @doi:10.1080/01621459.2014.896806]. -As high-dimensional data became common, scalable estimation procedures emerged to make these ideas practical. Meinshausen and Bühlmann (2006) proposed neighborhood selection, recasting graph recovery as a series of sparse regression problems that infer conditional dependencies node by node [@doi:10.1214/009053606000000281]. Shortly thereafter, Friedman, Hastie, and Tibshirani (2008) developed the graphical lasso, a convex penalized likelihood method that directly estimates sparse precision matrices [@doi:10.1093/biostatistics/kxm045]. These contributions showed that sparsity-inducing penalties could recover large network structures reliably, thereby providing concrete tools for estimating $f(c)$ when context corresponds to a structured dependency pattern such as a graph. +### Learning the Group Boundaries -Building on these advances, later research recognized that networks themselves may vary across contexts. Guo, Levina, Michailidis, and Zhu (2011) introduced penalties that jointly estimate multiple graphical models, encouraging sparsity within each network while borrowing strength across related groups [@doi:10.1093/biomet/asq060]. Danaher, Wang, and Witten (2014) extended this framework with the Joint Graphical Lasso, which balances shared structure and context-specific edges across multiple populations [@doi:10.1111/rssb.12033]. These developments illustrate how structured regularization transforms explicit adaptivity into a principled strategy: instead of estimating networks independently, one can pool information selectively across contexts (where context $c$ is the group or task identity), making the estimation of the parameter function $f(c)$ both interpretable and statistically efficient. +Thus far group-based modeling has required knowing which groups to model ahead of estimation. The next step lets the data decide where the boundaries fall. Model-based recursive partitioning fits a parametric model, tests its coefficients for instability across candidate context variables, and splits on the variable with the strongest instability before recursing, so the partition is chosen by the data rather than pre-specified [@doi:10.1198/106186008X319331]; toolkits such as partykit fit a parametric model within each leaf [@hothorn2015partykit]. In the network case the same move recovers where structure changes. Varying-coefficient varying-structure (VCVS) graphical models treats both the edge set and the edge weights as functions of context, and when structure changes abruptly a temporally smoothed penalty recovers the changepoints together with the precision matrix on each block, the first such estimators shown to be sparsistent with established convergence rates [@doi:10.1214/12-EJS739; @kolar2009sparsistent]. TREEGL extends this to a tree of networks that switch along a branching biological lineage, borrowing strength between a parent cell type and its descendants while exposing the edges that change at each division [@doi:10.1093/bioinformatics/btr239]. -**Varying-Coefficient Varying-Structure Models.** -The methods above pool information across a fixed set of discrete networks, but in many systems the network itself rewires as an underlying context such as time or developmental stage advances. Kolar and Xing formalized this regime as the varying-coefficient varying-structure (VCVS) graphical model, in which both the edge set and the edge weights of a Gaussian graphical model are treated as functions of context rather than fixed quantities [@doi:10.1214/12-EJS739]. When the context is time and the structure changes abruptly, a temporally smoothed $\ell_1$ penalty recovers both the changepoints and the precision matrix on each piecewise-constant block, and these were the first estimators of this kind shown to be sparsistent with established convergence rates [@doi:10.1214/12-EJS739; @kolar2009sparsistent]. When structure instead drifts smoothly, kernel reweighting and total-variation penalties estimate a separate network at each point along the context axis, as in the TESLA method and related time-varying network estimators applied to rewiring gene-regulatory and political networks [@doi:10.1073/pnas.0901910106; @doi:10.1214/09-AOAS308]. The context need not be temporal: TREEGL estimates a tree of networks evolving along a branching biological lineage, borrowing strength between a parent cell type and its descendants while exposing the edges that switch at each division [@doi:10.1093/bioinformatics/btr239]. Pushing the idea further, personalized regression and Bayesian edge-regression models let structure vary at the level of the individual sample, recovering subject-specific networks indexed by clinical covariates or latent similarity rather than by a shared group label [@doi:10.1093/bioinformatics/bty250; @doi:10.1080/01621459.2021.2000866]. The unifying move across these methods is to make the discrete graph structure, and not only the continuous coefficients, an explicit function of context $f(c)$, which is what separates VCVS models from smooth varying-coefficient models that hold the support fixed. +Viewing a partition as explicit routing clarifies what these splits accomplish. Each split of the context space sends samples to a distinct parameter vector, so the boundaries encode exactly where parameters are shared and where they are separated. Hierarchical partitions capture heterogeneity at two levels, sample-level variation within a context and task-level switching across contexts, which connects partition-based models to multi-task learning (Figure {@fig:context-splits}). This explicit routing is the structured counterpart of the implicit routing performed by mixture-of-experts and attention layers in the next section, where the same shared-versus-separated decision is made inside the network rather than by an overt split. -**Piecewise-Constant and Partition-Based Models.** -Here, model parameters are allowed to remain constant within specific regions or clusters of the context space, rather than vary smoothly. Approaches include classical grouped estimators and modern partition models, which may learn changepoints using regularization tools like total variation penalties or the fused lasso. A complementary, tree-structured route learns the partition by recursive splitting: model-based recursive partitioning fits a parametric model, tests its coefficients for instability across candidate context variables, and splits on the variable with the strongest instability before recursing, so that the boundaries are chosen by the data rather than fixed in advance [@doi:10.1198/106186008X319331]. Toolkits such as partykit make these conditional-inference and model-based trees readily usable, fitting a parametric model within each leaf [@hothorn2015partykit]. This framework is particularly effective for data with abrupt transitions or heterogeneous subgroups. +![Hierarchical splits of context enable multi-level adaptivity. Explicit adaptivity can partition the context space into piecewise models, with parameters indexed both by context $c$ and task identity $(i,j)$. Such splits allow sample-level heterogeneity to be captured within contexts, while high-level partitions mimic task boundaries and enable task switching.](images/context_splits.png){#fig:context-splits width="75%"} -A key design principle is that explicit splits of the context space can emulate distinct tasks, clarifying where parameters should be shared or separated. By introducing hierarchical partitions, we can capture heterogeneity at multiple levels: sample-level variation within each context, and task-level switching across contexts. This perspective connects classical partition-based models with multi-task learning, highlighting how explicit splits of context define where parameters should be shared versus differentiated (Figure {@fig:context-splits}). +### Dissolving the Boundaries with Kernels -![Hierarchical splits of context enable multi-level adaptivity. Explicit adaptivity can partition -the context space into piecewise models, with parameters indexed both by context $c$ and task -identity $(i,j)$. Such splits allow sample-level heterogeneity to be captured within contexts, -while high-level partitions mimic task boundaries and enable task switching.](images/context_splits.png){#fig:context-splits width="75%"} - - -A subtle but important point is that the boundary between “parametric” and “nonparametric” adaptivity is porous. -If we fit **simple parametric models within each context** -- for observed contexts $c$ or latent subcontexts $Z$ -- and then **aggregate across contexts**, the resulting conditional +Partitions, however they are drawn, still impose hard boundaries. Kernel and locally weighted methods remove them, estimating a separate model at each query context from a similarity-weighted neighborhood of samples. This is the classical varying-coefficient setting, where coefficients are smooth functions of a low-dimensional context estimated with kernel smoothing, local polynomials, or penalized splines. For a query context $c^\ast$, a semiparametric VCM solves $$ -P(Y\mid X,C) \;=\; \int P(Y\mid X,C,Z)\, dP(Z\mid C) +\widehat{\theta}(c^\ast) = \arg\max_{\theta} \sum_{i=1}^n K_\lambda(c_i, c^\ast)\,\ell(x_i; \theta), $$ -can display rich, multimodal behavior that looks nonparametric. In other words, **global flexibility can emerge from compositional, context-specific parametrics**. -When component families are identifiable (or suitably regularized) and the context-to-mixture map is constrained (e.g., smoothness/TV/sparsity over $c$), the aggregate model remains estimable and interpretable while avoiding overflexible, ill-posed mixtures. - -![Compositional inference: nonparametric flexibility from parametric context-specific models. -(A) Overall conditional $P(Y \mid X, C)$. -(B) Context-specific components $P(Y \mid X, C, Z=z_i)$ for latent subgroups $Z$. -(C) Recombination via marginalization $\int_Z P(Y \mid X, C, Z)$. -(D) Aggregated distribution showing how structured parametric pieces yield multimodal, nonparametric-like behavior.](images/compositional-inference.png){#fig:compositional-inference width="85%"} - -This perspective motivates flexible function approximators: trees and neural networks can be read as learning either the **context-to-mixture weights** or **local parametric maps**, providing similar global flexibility with different inductive biases. +where $K_\lambda$ measures similarity between contexts and $\ell$ is the per-sample log-likelihood, so prediction at $c^\ast$ is a similarity-weighted combination of nearby observations. Because every sample contributes in proportion to its context similarity, no single group needs to be large; the neighborhood supplies the data. Reproducing-kernel methods extend this to higher dimensions while retaining guarantees, for example penalized RKHS estimators for partially varying-coefficient models that separate constant from smoothly varying effects with minimax prediction rates and consistent structure selection [@doi:10.1016/j.csda.2020.107039], and the same machinery underlies generalized additive models. This locally weighted view is also the thread a later section follows to show that transformers performing in-context learning realize the same estimator, with a learned attention kernel taking the place of $K_\lambda$. -**Structured Regularization for Spatial, Graph, and Network Data.** -When context has known spatial or network structure, regularization terms can promote similarity among neighboring coefficients or nodes. For example, spatially varying-coefficient models have been applied to problems in geographical analysis and econometrics, where local effects are expected to vary across adjacent regions [@doi:10.48550/arXiv.2410.07229; @doi:10.48550/arXiv.2502.14651]. On networked data, the network VCM of [@doi:10.1080/01621459.2025.2470481] generalizes these ideas by learning both the latent positions and the parameter functions on graphs, allowing the model to accommodate complex relational heterogeneity. Such structural constraints allow models to leverage domain knowledge, improving efficiency and interpretability where smooth models may struggle. These regularization principles can also be extended to temporal, hierarchical, or multilevel contexts, where smooth transitions or cross-level coupling may be encoded through Laplacian penalties or nested-group regularizers tailored to the structure of $c$. +Similarity can be defined over a topology rather than a continuous covariate. Spatially varying-coefficient models let local effects change gradually across adjacent regions [@doi:10.48550/arXiv.2410.07229; @doi:10.48550/arXiv.2502.14651], the network varying-coefficient model learns latent node positions and coefficient functions on a graph [@doi:10.1080/01621459.2025.2470481], and Laplacian or nested-group penalties encode smoothness over temporal, hierarchical, or multilevel structure. The network-valued case appears here too: kernel reweighting and total-variation penalties estimate a separate network at each point along a context axis, as in TESLA and related time-varying network estimators for rewiring gene-regulatory and political networks [@doi:10.1073/pnas.0901910106; @doi:10.1214/09-AOAS308], and covariate-dependent Bayesian graph learning lets network structure vary smoothly with observed covariates through a dual spike-and-slab prior that selects at node, covariate, and local levels [@doi:10.1093/biomtc/ujaf053]. -Beyond spatial and single-network constraints, Bayesian approaches allow explicit modeling of multiple related graphical models across contexts. Rather than estimating each network independently or pooling across all data, these methods place structured priors that encourage information sharing when appropriate. For example, [@doi:10.1198/jasa.2011.tm10465] introduced Bayesian inference for GGMs with lattice structure, demonstrating how spatial priors can capture context-dependence across neighboring sites. Building on this idea, [@doi:10.1080/01621459.2014.896806] proposed a Bayesian framework with a Markov random field prior and spike-and-slab formulation to learn when edges should be shared across sample groups, improving estimation and quantifying inter-context similarity. More recently, [@doi:10.1093/biomtc/ujaf053] extended these principles to covariate-dependent graph learning, where network structure varies smoothly with observed covariates. Their dual group spike-and-slab prior enables multi-level selection at node, covariate, and local levels, providing a flexible and interpretable framework for heterogeneous biological networks. Together, these advances illustrate how Bayesian structural priors make adaptivity explicit in graphical models, supporting both efficient estimation and scientific interpretability. +### Structured Parametric Maps -#### Learned Function Approximators +Kernel methods treat the context dimensions symmetrically through the similarity metric. The next step learns which dimensions matter. Structured parametric VCMs impose form on $f$, from the linear map $\theta_i = A c_i$ to sparsity and group penalties on the coefficient functions, so that estimation both adapts the parameters and identifies the context features driving that adaptation. Tree-based ensembles are the high-capacity realization of this idea for tabular and mixed-type data: Tree Boosted Varying-Coefficient Models estimate context-dependent coefficients with gradient-boosted trees, balancing flexibility, accuracy, and interpretability while remaining easier to tune than deep networks [@doi:10.48550/arXiv.1904.01058]; cyclic gradient boosting adds dimension-wise early stopping and feature-importance measures [@doi:10.48550/arXiv.2401.05982]; and VCBART embeds Bayesian Additive Regression Trees into the varying-coefficient framework, estimating complex effect modifiers with coherent uncertainty quantification and good scaling in high dimensions [@doi:10.1214/24-BA1470]. What these share is a readout of context relevance, through split statistics, feature importances, or posterior inclusion, that the kernel view does not provide. -As context dimensionality and data complexity grow, explicit smoothness assumptions become insufficient, motivating high-capacity learners that approximate $f(c)$ directly from data. A third class of methods is rooted in modern machine learning, leveraging high-capacity models to approximate $f(c)$ directly from data. These approaches are especially valuable when the context is high-dimensional or unstructured, where classical assumptions may no longer be sufficient. +### From Composition to Learned Functions -**Tree-Based Ensembles.** -Gradient boosting decision trees (GBDTs) and related ensemble methods are well suited to tabular and mixed-type data. A representative example is Tree Boosted Varying-Coefficient Models, introduced by Zhou and Hooker (2019), where GBDTs are applied to estimate context-dependent coefficient functions within a VCM framework [@doi:10.48550/arXiv.1904.01058]. This approach offers a useful balance among flexibility, predictive accuracy, and interpretability, while typically being easier to train and tune than deep neural networks. More recently, Zakrisson and Lindholm (2024) proposed a tree-based varying coefficient model that incorporates cyclic gradient boosting machines (CGBM). Their method enables dimension-wise early stopping and provides feature importance measures, thereby enhancing interpretability and offering additional regularization [@doi:10.48550/arXiv.2401.05982]. +The boundary between parametric and nonparametric adaptivity is porous. If we fit simple parametric models within each context, for observed contexts $c$ or latent subcontexts $Z$, and then aggregate across contexts, the resulting conditional -Overall, tree-based VCMs achieve strong predictive performance and retain a model structure that lends itself to interpretation, particularly when combined with tools such as SHAP for explaining model outputs. A recent extension of this line of research is the Bayesian tree-based varying-coefficient model VCBART [@doi:10.1214/24-BA1470]. VCBART integrates the flexibility of Bayesian Additive Regression Trees (BART) into the varying-coefficient framework, allowing the estimation of complex effect modifiers without imposing restrictive functional assumptions or requiring intensive hyperparameter tuning. Compared with classical kernel or spline-based estimators, VCBART provides coherent uncertainty quantification and improved scalability for high-dimensional or multivariate covariate effects. Empirical studies in social and spatial applications show that VCBART effectively captures nonlinear context–response interactions, marking a promising step toward unifying Bayesian inference and ensemble learning within varying-coefficient modeling. - -**Deep Neural Networks.** -For contexts defined by complex, high-dimensional features such as images, text, or sequential data, deep neural networks offer unique advantages for modeling $f(c)$. These architectures can learn adaptive, data-driven representations that capture intricate relationships beyond the scope of classical models. Applications include personalized medicine, natural language processing, and behavioral science, where outcomes may depend on subtle or latent features of the context. - -The decision between these machine learning approaches depends on the specific characteristics of the data, the priority placed on interpretability, and computational considerations. Collectively, these advances have significantly broadened the scope of explicit adaptivity, making it feasible to model heterogeneity in ever more complex settings. - -### Key Theoretical Advances - -The expanding landscape of varying-coefficient models (VCMs) has been supported by substantial theoretical progress, which secures the validity of flexible modeling strategies and guides their practical use. The nature of these theoretical results often reflects the core structural assumptions of each model class. +$$ +P(Y\mid X,C) \;=\; \int P(Y\mid X,C,Z)\, dP(Z\mid C) +$$ -**Theory for Smooth Non-parametric Models.** -For classical VCMs based on kernel smoothing, local polynomial estimation, or penalized splines, extensive theoretical work has characterized their convergence rates and statistical efficiency. Under standard regularity conditions, these estimators are known to achieve minimax optimality for function estimation in moderate dimensions [@doi:10.1111/j.2517-6161.1993.tb01939.x]. More specifically, Lu, Zhang, and Zhu (2008) established both consistency and asymptotic normality for penalized spline estimators when using a sufficient number of knots and appropriate penalty terms [@doi:10.1080/03610920801931887], enabling valid inference through confidence intervals and hypothesis testing. These results provide a solid theoretical foundation even in relatively complex modeling contexts. +can display rich, multimodal behavior that looks nonparametric. Global flexibility can emerge from compositional, context-specific parametrics. When component families are identifiable or suitably regularized and the context-to-mixture map is constrained by smoothness, total variation, or sparsity over $c$, the aggregate model remains estimable and interpretable while avoiding overflexible, ill-posed mixtures (Figure {@fig:compositional-inference}). -**Theory for Structurally Constrained Models.** -When discrete or network structure is incorporated into VCMs, theoretical analysis focuses on identifiability, regularization properties, and conditions for consistent estimation. For example, [@doi:10.1080/01621459.2025.2470481] provide non-asymptotic error bounds for estimators in network VCMs, demonstrating that consistency can be attained when the underlying graph topology satisfies certain connectivity properties. In piecewise-constant and partition-based models, results from change-point analysis and total variation regularization guarantee that abrupt parameter changes can be recovered accurately under suitable sparsity and signal strength conditions. +![Compositional inference: nonparametric flexibility from parametric context-specific models. (A) Overall conditional $P(Y \mid X, C)$. (B) Context-specific components $P(Y \mid X, C, Z=z_i)$ for latent subgroups $Z$. (C) Recombination via marginalization $\int_Z P(Y \mid X, C, Z)$. (D) Aggregated distribution showing how structured parametric pieces yield multimodal, nonparametric-like behavior.](images/compositional-inference.png){#fig:compositional-inference width="85%"} -**Theory for High-Capacity and Learned Models.** -The incorporation of machine learning models into VCMs introduces new theoretical challenges. For high-dimensional and sparse settings, oracle inequalities and penalized likelihood theory establish conditions for consistent variable selection and accurate estimation, as seen in methods based on boosting and other regularization techniques. In the context of neural network-based VCMs, the theory is still developing, with current research focused on understanding generalization properties and identifiability in non-convex optimization. This remains an active and important frontier for both statistical and machine learning communities. +This perspective motivates flexible function approximators: trees and neural networks can be read as learning either the context-to-mixture weights or the local parametric maps. The final step lets that function be arbitrary. -These theoretical advances provide a rigorous foundation for explicit adaptivity, a wide range of complex and structured modeling scenarios. +### Contextualized Models: Arbitrary Functions of Context -### Sparsity and Incomplete Measurements as Context +For contexts defined by high-dimensional or unstructured features such as images, text, or sequences, deep neural networks approximate $f(c)$ without committing to a fixed basis, partition, or similarity metric. A network can consume context as input and let its nonlinear layers induce context-dependent behavior, the pattern that reappears for implicit models in the next section; it can generate the parameters of another model directly, as in hypernetworks, which give an explicit deep-net realization of $\theta = f(c)$ [@doi:10.48550/arXiv.1609.09106]; or it can modulate a shared backbone through feature-wise affine transformations supplied by context, as in FiLM [@doi:10.48550/arXiv.1709.07871]. Taking this to its limit yields contextualized models, which treat $f$ as an unrestricted map learned end-to-end and estimate it directly, capturing complex functions of context including the feature interactions that the structured maps above cannot express [@doi:10.48550/arXiv.2310.11340]. A single deep encoder reads a sample's context and emits the parameters of its downstream model, so estimation is amortized across contexts: the cost is paid once during training, and inference for a new context is a forward pass that needs no per-group data at all. The formulation spans model types and domains, including personalized disease models [@doi:10.1073/pnas.2411930122; @doi:10.48550/arXiv.2111.01104], heterogeneous treatment effects [@doi:10.1016/j.jbi.2022.104086; @doi:10.48550/arXiv.2310.07918], drug development [@doi:10.64898/2026.05.11.724149], and contextual feature selection [@doi:10.48550/arXiv.2312.14254], with standard implementations in the contextualized.ml package [@doi:10.21105/joss.06469]. The network-valued thread reaches the same endpoint: personalized regression and Bayesian edge-regression models learn a map from a sample's covariates or latent similarity to its own network, recovering subject-specific structure rather than a shared group label [@doi:10.1093/bioinformatics/bty250; @doi:10.1080/01621459.2021.2000866]. Because the encoder infers parameters from context alone, it is the explicit object the next section reaches back to when it interprets an amortized context encoder as the bridge to in-context learning. -A central practical challenge in combining real-world datasets is inconsistent measurement: different cohorts or institutions often collect different subsets of features. One dataset may contain detailed laboratory values, another may focus on imaging or physiological measurements, and a third may emphasize clinical outcomes. If such cohorts are naively pooled, the resulting feature matrix is sparse and unbalanced. If incomplete samples are discarded, data efficiency collapses. +### Beyond Covariates: What Serves as Context -Context-adaptive models provide a natural resolution by treating **measurement sparsity itself as context.** Rather than ignoring missingness, the model learns to adjust its parameterization according to which features are observed. In effect, each measurement policy (labs-only, vitals-only, multimodal) defines a context, and explicit adaptivity allows estimation that respects these differences while still sharing information. This perspective reframes missingness from a nuisance into structured signal: it encodes which sources of evidence are available and how they should be combined. This perspective reframes missingness from a nuisance into structured signal: it encodes which sources of evidence are available and how they should be combined, reflecting ideas explored in recent multimodal learning frameworks that handle missing modalities [@doi:10.48550/arXiv.2409.07825]. +Context need not be a covariate, a task identifier, or a position in a sequence. Any signal the model can condition on can serve as context, and a useful example is the pattern of missing measurements itself. Combining real-world datasets is complicated by inconsistent measurement: different cohorts or institutions collect different subsets of features, so naive pooling yields a sparse, unbalanced feature matrix, while discarding incomplete samples wastes data. Context-adaptive models resolve this by treating measurement sparsity as context. Rather than ignoring missingness, the model adjusts its parameterization to which features are observed, so that each measurement policy, whether labs-only, vitals-only, or multimodal, defines a context and information is shared across policies while their differences are respected. This reframes missingness from a nuisance into structured signal that encodes which sources of evidence are available and how they should be combined, an idea also pursued in multimodal learning frameworks that handle missing modalities [@doi:10.48550/arXiv.2409.07825]. By conditioning on measurement availability, a model learns from fewer individuals with more heterogeneous features (Figure {@fig:sparsity-context}). The metrics and stress tests specific to missingness-as-context are collected with the shared evaluation principles later in the review. ![Patterns of missingness as context. Each dataset (e.g., cohort with labs, cohort with vitals, cohort with imaging) provides a different subset of measurements. Context-adaptive models allow integration by conditioning on measurement availability, enabling learning from fewer samples with more heterogeneous features.](images/measurement-sparsity-context.png){#fig:sparsity-context width="70%"} -Figure @fig:sparsity-context illustrates this idea: each cohort contributes a different subset of measurements (lungs, labs, vitals), and explicit adaptivity enables integration across cohorts. By conditioning on measurement availability, we can achieve greater sample efficiency, learning from fewer individuals but with richer heterogeneous features. +### Key Theoretical Advances + +The guarantees available for these methods reflect the assumptions each one makes, and they track the progression above. -Evaluation of missingness-as-context models should report *mask-stratified metrics*, including worst-group performance, following group-robust evaluation practice [@doi:10.48550/arXiv.1911.08731; @doi:10.48550/arXiv.2012.07421]. Robustness should be probed with *mask-shift stress tests*, training under one measurement policy and testing under another, to quantify degradation and the benefit of contextualization, as formalized in the Domain Adaptation under Missingness Shift (DAMS) setting [@doi:10.48550/arXiv.2211.02093; @doi:10.48550/arXiv.2012.07421]. When imputation is used, authors should assess *imputation realism* by holding out observed entries under realistic mask distributions and reporting MAE/RMSE and calibration for $p(x_{\text{missing}}\mid x_{\text{observed}})$ [@doi:10.48550/arXiv.1806.02382; @doi:10.48550/arXiv.1806.02920]. For causal or estimation applications, conduct *ignorability sensitivity analyses*, contrasting MAR-based results with pattern-mixture or selection-model analyses under plausible MNAR mechanisms [@doi:10.2307/2337120; @doi:10.48550/arXiv.2301.05043]. Finally, include *ablations* that remove mask/indicator inputs—and, for trees, disable default-direction routing—to confirm that gains derive from modeling the mask signal rather than artifacts [@doi:10.48550/arXiv.1603.02754; @doi:10.48550/arXiv.2211.09259]. Practical implementations of these ideas are widely available: **GRU-D** [@doi:10.48550/arXiv.1606.01865] and **BRITS** [@doi:10.48550/arXiv.1805.10572] provide mask- and time-aware sequence models, while **GAIN** [@doi:10.48550/arXiv.1806.02920] and **VAEAC** [@doi:10.48550/arXiv.1806.02382] offer open-source code for imputation under arbitrary masks. For tree ensembles, **XGBoost** supports sparsity-aware default-direction splits, making it straightforward to treat “NA” values as context without preprocessing [@doi:10.1145/2939672.2939785]. +#### Partitions and learned boundaries -### Context-Aware Efficiency Principles and Design +For grouped and partition-based estimators, change-point analysis and total-variation regularization establish when abrupt parameter changes can be recovered. Under suitable sparsity and signal-strength conditions, fused-lasso and total-variation penalties recover both the location of the changepoints and the parameters on each piecewise-constant block [@doi:10.1214/09-AOAS308]. -The efficiency of context-adaptive methods hinges on several key design principles that balance computational tractability with statistical accuracy. These principles guide the development of methods that can scale to large datasets while maintaining interpretability and robustness. +#### Kernel and smooth estimation -One central principle is the use of sparsity assumptions to limit the number of context-dependent parameters. This can be achieved through group sparsity, which encourages entire groups of parameters to be zero simultaneously [@doi:10.1111/j.1467-9868.2005.00532.x], hierarchical regularization that applies different strengths of shrinkage to varying levels of context specificity [@doi:10.1017/CBO9780511790942], and adaptive thresholding that dynamically adjusts sparsity levels in accordance with context complexity. +For kernel smoothing, local polynomial estimation, and penalized splines, convergence rates and efficiency are well characterized. Under standard regularity conditions these estimators achieve minimax-optimal rates for function estimation in moderate dimensions [@doi:10.1111/j.2517-6161.1993.tb01939.x], and Lu, Zhang, and Zhu established consistency and asymptotic normality for penalized spline estimators given a sufficient number of knots and appropriate penalties, enabling valid inference through confidence intervals and hypothesis tests [@doi:10.1080/03610920801931887]. -Efficiency can also be enhanced through computational strategies that allocate resources adaptively. Early stopping terminates optimization for contexts where convergence occurs rapidly [@doi:10.48550/arXiv.1606.04838], while context-dependent sampling employs different sampling schemes across contexts [@doi:10.48550/arXiv.1809.09582]. Caching and warm-starting further accelerate optimization by leveraging solutions from similar contexts, particularly effective when contexts exhibit smooth variation [@doi:10.1561/2200000016]. +#### Structure-varying estimation -A further consideration is the balance between efficiency and interpretability. Linear context functions are highly interpretable but may require many parameters, while explicit context encodings improve transparency at the potential cost of higher computational overhead. Local context modeling provides fine-grained interpretability but may be less scalable to large applications. These trade-offs should be evaluated in light of application-specific requirements. For example, advanced adaptive optimizers like Adam can efficiently train complex, nonlinear models, but the resulting systems may be less interpretable than simpler alternatives [@doi:10.48550/arXiv.1412.6980]. In practice, such context-dependent computation appears in adaptive batching, per-context learning rates, and multi-fidelity optimization pipelines that dynamically adjust compute and precision depending on context complexity. +When the support itself varies with context, analysis centers on identifiability, sparsistency, and consistent recovery of the changing structure. The VCVS estimators of Kolar and Xing were the first shown to be sparsistent with established convergence rates when the network changes abruptly over time [@kolar2009sparsistent; @doi:10.1214/12-EJS739], and for networked coefficients non-asymptotic error bounds show that consistency is attainable when the underlying graph topology is sufficiently connected [@doi:10.1080/01621459.2025.2470481]. -### Synthesis and Future Directions +#### Learned and high-capacity models -Selecting an appropriate modeling strategy for $f(c)$ involves weighing flexibility, interpretability, computational cost, and the extent of available domain knowledge. Learned function approximators, such as deep neural networks, offer unmatched capacity for modeling complex, high-dimensional relationships. However, classical smooth models and structurally constrained approaches often provide greater interpretability, transparency, and statistical efficiency. The choice of prior assumptions and the scalability of the estimation procedure are also central considerations in applied contexts. +High-capacity approximators raise harder questions. In high-dimensional and sparse settings, oracle inequalities and penalized-likelihood theory give conditions for consistent variable selection and accurate estimation, including for boosting-based estimators. For neural-network realizations of $f(c)$, generalization and identifiability under non-convex optimization remain only partly understood and are an active frontier for both the statistics and machine learning communities. -Looking forward, several trends are shaping the field. One important direction is the integration of varying-coefficient models with foundation models from natural language processing and computer vision. By using pre-trained embeddings as context variables $c_i$, it becomes possible to incorporate large amounts of prior knowledge and extend VCMs to multi-modal and unstructured data sources. Another active area concerns the principled combination of cross-modal contexts, bringing together information from text, images, and structured covariates within a unified VCM framework. +### Future Directions -Advances in interpretability and visualization for high-dimensional or black-box coefficient functions are equally important. Developing tools that allow users to understand and trust model outputs is critical for the adoption of VCMs in sensitive areas such as healthcare and policy analysis. +One direction integrates varying-coefficient models with foundation models from language and vision. Using pretrained embeddings as the context $c_i$ lets these models absorb large amounts of prior knowledge and extend to multimodal and unstructured sources, and the principled combination of cross-modal contexts, bringing text, images, and structured covariates into a single framework, remains an open problem. -Finally, closing the gap between methodological innovation and practical deployment remains a priority. Although the literature has produced many powerful variants of VCMs, practical adoption is often limited by the availability of software and the clarity of methodological guidance [@doi:10.3390/publications13020019]. Continued investment in user-friendly implementations, open-source libraries, and empirical benchmarks will facilitate broader adoption and greater impact. +Interpretability and visualization for high-dimensional or black-box coefficient functions are equally important. Tools that let users understand and trust estimated coefficient surfaces are a prerequisite for adoption in sensitive areas such as healthcare and policy. -In summary, explicit adaptivity through structured estimation of $f(c)$ now forms a core paradigm at the interface of statistical modeling and machine learning. Future progress will focus not only on expanding the expressive power of these models, but also on making them more accessible, interpretable, and practically useful in real-world applications. +Finally, the gap between methodological innovation and practical deployment remains. Many capable VCM variants exist, but adoption is often limited by the availability of software and the clarity of methodological guidance [@doi:10.3390/publications13020019]. Continued investment in usable implementations, open-source libraries, and empirical benchmarks will broaden adoption and impact. Explicit adaptivity now spans this full progression, from independent subgroups to amortized deep encoders; the principles for evaluating and deploying any point on it, shared with the implicit methods of the next section, are the subject of a later chapter. diff --git a/content/07.implicit.md b/content/07.implicit.md index 7c0369d..66c6d17 100644 --- a/content/07.implicit.md +++ b/content/07.implicit.md @@ -16,7 +16,7 @@ The capacity for implicit adaptation does not originate from a single mechanism, In contrast to explicit parameter mapping, the simplest route to implicit adaptation is to feed context directly as part of the input. The simplest form of implicit adaptation appears in neural network models that directly incorporate context as part of their input. In models written as $y_i = g([x_i, c_i]; \Phi)$, context features $c_i$ are concatenated with the primary features $x_i$, and the mapping $g$ is determined by a single set of fixed global weights $\Phi$. Even though these parameters do not change during inference, the network’s nonlinear structure allows it to capture complex interactions. As a result, the relationship between $x_i$ and $y_i$ can vary depending on the specific value of $c_i$. -This basic yet powerful principle is central to many conditional prediction tasks. For example, personalized recommendation systems often combine a user embedding (as context) with item features to predict ratings. Similarly, in multi-task learning frameworks, shared networks learn representations conditioned on task or environment identifiers, which allows a single model to solve multiple related problems [@doi:10.48550/arXiv.1706.05098]. +This basic yet powerful principle is central to many conditional prediction tasks. For example, personalized recommendation systems often combine a user embedding (as context) with item features to predict ratings. Similarly, in multi-task learning frameworks using task-flags (Figure {@fig:mtl-context}), shared networks learn representations conditioned on task or environment identifiers, which allows a single model to solve multiple related problems [@doi:10.48550/arXiv.1706.05098]. #### Interaction Effects and Attention Mechanisms @@ -40,7 +40,7 @@ Gradient-based meta-learning frameworks such as Model-Agnostic Meta-Learning (MA #### Amortized Estimation: Context Encoders -In the previous chapter we looked at modeling frameworks with explicit context-adaptive components. The most common implementation is a context encoder, which is learned to map from a task's context to a set of task-specific parameters. The result is an amortized estimator, which predicts the parameters of the downstream model that would have been produced by a classical estimator if sufficient data was collected from that context. +The contextualized models at the end of the previous section already learned a context encoder, a map from a task's context to a set of task-specific parameters. The result is an amortized estimator, which predicts the parameters of the downstream model that would have been produced by a classical estimator if sufficient data were collected under that context. Here we take the same object and ask what happens as the encoder absorbs progressively more of the estimation pipeline. This is desirable because data collection and model estimation are often far more costly than model inference. Plotting the progression of estimation methods, classical methods require a sufficient amount of data for every task. Transfer learning and meta-learning are formulated to achieve similar performance with fewer samples, but still require explicit gradient-based estimators. Pushing this to its limit produces amortized estimators, such as context encoders, which only use a task's context to infer the task-specific data distribution and produce in-context inferences. diff --git a/content/08.evaluation.md b/content/08.evaluation.md new file mode 100644 index 0000000..e9fd3a5 --- /dev/null +++ b/content/08.evaluation.md @@ -0,0 +1,25 @@ +## Evaluation and Design Principles for Context-Adaptive Inference + +The two preceding sections described how adaptation is built, explicitly through a structured $f(c)$ and implicitly through emergent computation in large models. Both paradigms raise the same three practical questions once a model is deployed: how cheaply it adapts, how stably it routes a context to the right behavior, and how well that behavior holds up when the context shifts. This section collects the design and evaluation principles that answer these questions. + +### Adaptation Efficiency + +The efficiency of a context-adaptive method hinges on design choices that trade computational tractability against statistical accuracy, and the same choices govern whether a method scales to large data while remaining interpretable. + +One central principle is the use of sparsity assumptions to limit the number of context-dependent parameters. This can be achieved through group sparsity, which encourages entire groups of parameters to be zero simultaneously [@doi:10.1111/j.1467-9868.2005.00532.x], hierarchical regularization that applies different strengths of shrinkage to varying levels of context specificity [@doi:10.1017/CBO9780511790942], and adaptive thresholding that adjusts sparsity levels in accordance with context complexity. + +Efficiency can also be enhanced through computational strategies that allocate resources adaptively. Early stopping terminates optimization for contexts where convergence occurs rapidly [@doi:10.48550/arXiv.1606.04838], while context-dependent sampling employs different sampling schemes across contexts [@doi:10.48550/arXiv.1809.09582]. Caching and warm-starting further accelerate optimization by reusing solutions from similar contexts, which is particularly effective when contexts vary smoothly [@doi:10.1561/2200000016]. + +A further consideration is the balance between efficiency and interpretability. Linear context functions are highly interpretable but may require many parameters, while explicit context encodings improve transparency at the potential cost of higher computational overhead, and local context modeling provides fine-grained interpretability but can be less scalable. These trade-offs should be weighed against application-specific requirements. Advanced adaptive optimizers such as Adam can efficiently train complex, nonlinear models, but the resulting systems may be less interpretable than simpler alternatives [@doi:10.48550/arXiv.1412.6980]. In practice, context-dependent computation of this kind appears in adaptive batching, per-context learning rates, and multi-fidelity optimization pipelines that adjust compute and precision according to context complexity. + +### Routing Stability + +Both paradigms decide, for a given context, which parameters or which experts govern the prediction. Explicit models make this decision overtly, through the partition splits and hierarchical routing of the discrete rung, where a boundary in context space sends a sample to one local model rather than another. Implicit models make the same decision internally, through the gating of a mixture of experts or the attention weights of a transformer. In either case the decision is a function of context, and its stability is a property worth measuring: small perturbations of the context should not cause erratic changes in which behavior is selected, or predictions become unpredictable near routing boundaries. + +Instability is easiest to see at the boundaries. A partition model can flip a sample between adjacent leaves under a negligible change in a splitting variable, and an in-context learner can change its answer under a reordering or reformatting of the same prompt examples, a sensitivity documented for in-context learning where the distribution and arrangement of examples can matter more than their content [@doi:10.48550/arXiv.2202.12837]. Reporting routing stability therefore means probing the model with controlled perturbations of the context that should be behavior-preserving, such as reordering exchangeable examples, jittering a continuous context near a learned split, or relabeling an equivalent measurement policy, and measuring how much the selected behavior and the resulting prediction move. Smoothly routed models degrade gracefully across these perturbations; sharply partitioned or brittle ones do not, and the difference is a design choice as much as a diagnostic. + +### Context-Specific Robustness + +The final axis asks whether adaptation still helps when the context distribution at test time differs from training. For explicit models, the sharpest version of this question arises when context is the pattern of observed measurements, because the deployment environment routinely presents measurement policies that were rare or absent during training. Evaluation of missingness-as-context models should report mask-stratified metrics, including worst-group performance, following group-robust evaluation practice [@doi:10.48550/arXiv.1911.08731; @doi:10.48550/arXiv.2012.07421]. Robustness should be probed with mask-shift stress tests, training under one measurement policy and testing under another, to quantify degradation and the benefit of contextualization, as formalized in the Domain Adaptation under Missingness Shift (DAMS) setting [@doi:10.48550/arXiv.2211.02093; @doi:10.48550/arXiv.2012.07421]. When imputation is used, authors should assess imputation realism by holding out observed entries under realistic mask distributions and reporting MAE/RMSE and calibration for $p(x_{\text{missing}}\mid x_{\text{observed}})$ [@doi:10.48550/arXiv.1806.02382; @doi:10.48550/arXiv.1806.02920]. For causal or estimation applications, conduct ignorability sensitivity analyses, contrasting MAR-based results with pattern-mixture or selection-model analyses under plausible MNAR mechanisms [@doi:10.2307/2337120; @doi:10.48550/arXiv.2301.05043]. Finally, include ablations that remove mask or indicator inputs, and, for trees, disable default-direction routing, to confirm that gains derive from modeling the mask signal rather than from artifacts [@doi:10.48550/arXiv.1603.02754; @doi:10.48550/arXiv.2211.09259]. Practical implementations of these ideas are widely available: GRU-D [@doi:10.48550/arXiv.1606.01865] and BRITS [@doi:10.48550/arXiv.1805.10572] provide mask- and time-aware sequence models, while GAIN [@doi:10.48550/arXiv.1806.02920] and VAEAC [@doi:10.48550/arXiv.1806.02382] offer open-source code for imputation under arbitrary masks. For tree ensembles, XGBoost supports sparsity-aware default-direction splits, making it straightforward to treat missing values as context without preprocessing [@doi:10.1145/2939672.2939785]. + +Implicit models face the same issue in a different form. In-context learning adapts within a single forward pass, but that adaptation is opaque and can be brittle: performance is sensitive to small changes in the prompt, and reliability under distribution shift is hard to guarantee or to audit. Recent surveys frame the open questions as developing a more complete theoretical account of when in-context learning fails, improving its reliability, and establishing methods for controlling its behavior in high-stakes applications [@doi:10.48550/arXiv.2301.00234]. The evaluation logic carries over from the explicit case: stratify performance by the kind of context, stress-test under context shift, and ablate the adaptive signal to confirm that the model is using context as intended rather than exploiting an artifact. Read this way, the mask-shift stress test for an explicit model and the prompt-perturbation test for an in-context learner are the same experiment applied to two realizations of $f(c)$. diff --git a/content/08.interpretations.md b/content/09.interpretations.md similarity index 100% rename from content/08.interpretations.md rename to content/09.interpretations.md diff --git a/content/09.invariant.md b/content/10.invariant.md similarity index 100% rename from content/09.invariant.md rename to content/10.invariant.md diff --git a/content/10.applications_tools.md b/content/11.applications_tools.md similarity index 100% rename from content/10.applications_tools.md rename to content/11.applications_tools.md diff --git a/content/11.future_trends.md b/content/12.future_trends.md similarity index 100% rename from content/11.future_trends.md rename to content/12.future_trends.md diff --git a/content/12.open_problems.md b/content/13.open_problems.md similarity index 100% rename from content/12.open_problems.md rename to content/13.open_problems.md diff --git a/content/13.conclusion.md b/content/14.conclusion.md similarity index 100% rename from content/13.conclusion.md rename to content/14.conclusion.md