Golden pipelines: second convenience API `aaanalysis.pipe` (aap) + sklearn transformer

[breimanntools]

Problem

AAanalysis exposes interpretable primitives (SequenceFeature, CPP, TreeModel,
dPULearn, ShapModel, AAclust, the metrics), but a user who just wants a result
must hand-wire the whole load → get_df_parts → scales → CPP.run → rank → plot chain
(and the feature_matrix → TreeModel.fit → eval chain for prediction) every time. That
is a high floor for first use, makes copy-paste errors easy, and gives coding agents no
single, stable, one-call entry point. There is also no sklearn-native way to put CPP
feature selection inside a Pipeline / cross_val_score without leaking the test fold.

Goal

Add a second, high-level convenience API — import aaanalysis.pipe as aap — a
stateless, thin facade of "golden pipelines" over the existing primitives (zero own
algorithm, defaults byte-identical to the explicit path), plus a sklearn-compliant
SequenceFeatureTransformer, with no new required dependency (sklearn is already
core; torch stays the [embed] extra).

Requirements

aaanalysis.pipe (aap) — core 3 golden pipelines (identify → predict → explain)

• aap.cpp_feature_map(df_seq, labels=None, subcategories=None, dpulearn=False, optimization="balanced", top_n=None, plot=True, random_state=None, n_jobs=None) -> (df_feat, ax) — wraps SequenceFeature.get_df_parts + scale selection + CPP.run + rank + CPPPlot.feature_map. subcategories filters df_scales/df_cat (via load_scales(name="scales_cat")); optimization is a ~3-level preset ("quick"/"balanced"/"thorough") collapsing the CPP knobs; dpulearn=True runs dPULearn first to derive reliable negatives → clean labels → CPP (the canonical PU→CPP path, the one sanctioned structural swap).
• aap.predict(df_feat, df_seq|df_parts, labels, model_class=None, n_cv=5, random_state=None, n_jobs=None) -> (model, df_eval) — rebuilds X = feature_matrix(df_feat.feature, df_parts, df_scales) then TreeModel.fit + cross-validated eval.
• aap.explain(df_feat, df_seq|df_parts, labels, model=None, ...) -> (df_feat_shap, ax) — pro; ShapModel SHAP values + SHAP-coloured feature map. Gated behind the pro extra via the existing missing_feature_stub mechanism.
• Option rule: parameter pass-through YES; structural component-swaps are separate composable aap.* pipelines (e.g. a stretch aap.reliable_negatives(df_seq, ...) -> labels standalone dPULearn), except the dpulearn flag on cpp_feature_map.

SequenceFeatureTransformer (in feature_engineering/, exposed in aa)

• BaseEstimator + TransformerMixin; fit(df_parts|df_seq, y) runs CPP feature selection on the train fold only, transform(...) -> X applies the SAME features → leak-free inside sklearn.pipeline.Pipeline / cross_val_score.
• Passes the relevant sklearn.utils.estimator_checks.

Principles

• Stateless (no pyplot-style global state); plain numpy/pandas out (torch via torch.from_numpy); random_state/seed threaded; numpydoc + one example notebook per pipeline; defaults asserted byte-identical to the manual path by test.

KPIs / Acceptance criteria

• aap.cpp_feature_map(...) / aap.predict(...) outputs are byte-identical to the equivalent explicit-primitive call at each optimization grade (regression-tested).
• SequenceFeatureTransformer runs in cross_val_score (n_splits ≥ 5, one score/fold, no leakage) and passes the relevant sklearn.utils.estimator_checks.
• load_dataset → aap full identify→predict CV score in ≤10 lines; one example notebook per pipeline runs under the nbmake gate.
• Repeated random_state → byte-identical predictions.

Scope / non-goals

• No new algorithm, no new required dependency; no torch in core (stays [embed]).
• ProtXplain boundary: this is user-/sklearn-idiomatic convenience (ours). The
  machine-readable tool contract / MCP / verb-orchestration layer stays in ProtXplain
  (relates ADR-0035, refined by ADR-0038).

Dependencies

• Child of the API-ergonomics epic API ergonomics: less boilerplate + a consistent mental model #126. Supersedes the closed Add simple ML pipeline wrapper for AAanalysis features #24 (sklearn
  pipeline wrapper). Relates Protocols: task-oriented, pipeline-ordered usage guide (epic) #35 / epic: ecosystem integration — consume upstream + descriptors, expose downstream (sklearn / XAI / design) #210 / Define benchmark dataset for evaluation #25. Relates ADR-0035 / ADR-0038.

Standards checklist

• CONFIRM-FIRST: new aaanalysis/pipe/ namespace + aaanalysis/__init__.py /
  __all__ re-export (new public surface) — call out explicitly at PR time.
• Frontend/backend split honored; validation block; backend trusts frontend.
• numpydoc (named Returns, per-method Examples include); reproducibility
  (random_state/seed); pro gating for aap.explain.
• tests (unit + byte-identical parity + sklearn estimator checks); no print();
  bare ValueError/RuntimeError.

[breimanntools]

Status + plan update (2026-06-24)

The four core golden pipelines have shipped (renamed/expanded from this issue's original sketch via the grill that produced ADR-0040 / ADR-0041; return contract is now the uniform (results, figs, evals) triple, ADR-0041 D1):

original sketch	shipped as	issue / PR
cpp_feature_map	find_features (now a CPPGrid + model-CV AutoML pipeline)	#252 ✅ (PR #258)
predict	predict_samples	#254 ✅ (PR #260)
explain	explain_features (pro)	#255 ✅
(new)	obtain_samples	#253 ✅ (PR #257)
(new) eval-grid plot	aap.plot_eval	#251 🔄 in flight (PR #266)

Plan: one pipeline ↔ one protocol (ADR-0041 D5)

Each golden pipeline is the executable spine of exactly one narrative protocol under protocols/protocol<N>_*.ipynb, and each such protocol is anchored by one pipeline — keeping the convenience layer and the documented workflows in lockstep. Proposed mapping onto the existing protocols (maintainer may adjust):

Pipeline	Protocol
obtain_samples	protocol3_sampling
find_features	protocol1_cpp_signature
predict_samples	protocol8_prediction
explain_features	protocol9_interpretability

Each mapped protocol adopts its pipeline as the one-call spine (intro/headline cell), then drops to the primitives for the deep dive. The remaining protocols (2/4/5/6/7/10) stay as detailed primitive workflows, not 1:1 with a pipeline.

Remaining for this umbrella

• SequenceFeatureTransformer (BaseEstimator + TransformerMixin, leak-free fit-on-train; supersedes the closed Add simple ML pipeline wrapper for AAanalysis features #24) — not yet started.
• Deferred top-level aap / __all__ exposure (CONFIRM-FIRST, ADR-0040 D5).

#241 stays open until the transformer lands.

[breimanntools]

Roadmap: golden-pipeline families (revised)

(Supersedes the earlier flat single-primitive list.) Organized by goal; "family" variants share a goal but use a different method / modality engine.

Core workflow goals:

#	Pipeline	Goal	Engine	Issues
1	obtain_samples ✅	build the training set	AAWindowSampler / dPULearn	#253
2	find_features ✅	identify features — CPP sequence	CPP / CPPGrid	#252
3	predict_samples ✅	train + CV a predictor	TreeModel	#254
4	benchmark_models	test against simple models + benchmarks — repeated-CV + bootstrap CIs + paired ΔMCC + learning-curve	metrics + TreeModel	#91 (prio:1), #25, #93, #55
5	explain_features ✅	explain — CPP-SHAP per residue	ShapModel	#255

benchmark_models design will draw from the ADAMTS7 thorough-benchmarking project (to be shared in a separate session) — the "test vs simple baselines + published benchmarks" methodology.

Family variants + extensions (candidates):

#	Pipeline	Goal — variant	Engine	Issues
6	find_features2	identify features — embeddings / structure / annotations / conservation	Embedding / Structure / Annotation Preprocessors; MSA	#22, #23, #40, #64, #65
7	explain_features2	explain — rule-extraction + example/sample-based	—	#50, #49
8	explain_features3	explain — uncertainty + causal	—	#16, #53, #54
9	engineer_proteins	ML-guided protein design / directed evolution	AAMut / SeqMut / SeqOpt	#37, #261, #59
10	map_structure	paint CPP/SHAP impact onto a 3D structure (pro)	CPPStructurePlot	#119, #120, #40
11	analyse_omics	connect to the omics ecosystem (scverse, proteomics)	new adapters	new issues (relates #210)

Naming (to settle): replace the 2/3 suffixes with a source / method parameter on the base verb — find_features(source="cpp"|"embedding"|"structure"|"annotation"|"conservation"), explain_features(method="shap"|"rules"|"examples") — staying verb_noun (ADR-0040 D1); peel off a new verb only when the goal genuinely differs (e.g. estimate_uncertainty). Final naming decided when each is spun into its own issue.

SequenceFeatureTransformer (the remaining #241 item) is separate from these.