Prototype: demand-driven generics as the only mechanism

[SimonHeybrock]

Consolidated end-state proposal for #235, now targeting main. Draft for discussion — depends on scipp/cyclebane#32; CI is red until that lands and is released. #238 has been merged into this PR; #236 remains open as the coexistence alternative. Full context, derivation, measured evidence, and a decision log are in docs/developer/architecture-and-design/demand-driven-generics.md (part of this PR); the commit history preserves one decision per commit.

What this is

Generic providers are resolved by demand-driven instantiation (structural unification of type patterns with requested keys), replacing eager constraint-based expansion:

import sciline as sl

type A = int
class Raw[Run](float): ...
class Processed[Run](float): ...

def foo[Run](x: Raw[Run]) -> Processed[Run]:
    return Processed[Run](x * 2)

pl = sl.Pipeline([foo], params={Raw[A]: Raw[A](1.2)})
pl.compute(Processed[A])  # Processed[A](2.4)

• PEP 695 generics (classes, functions, type X[T] = ... aliases) work with zero ceremony; the example from Support for Python 3.12 generics #235 runs verbatim. Old-style unconstrained TypeVars work the same way. Fixes the blocker behind Scope does not work with Python 3.12 generics #233.
• The engine is backward-chaining only: a demanded key unifies with generic return types, dependencies are demanded recursively. There is no forward chaining.
• Coherence instead of insertion-order dependence: a template with an equivalent pattern replaces the previous one; a strictly more specific pattern wins; incomparable overlapping matches raise. Concrete providers and params always shadow generic ones.
• Values for generic keys (params={Raw: value}) are templates applied on demand to all requested specializations.
• map() needs no special handling: Derive mapped-node labeling at compile time cyclebane#32 keeps plain node names and derives mapped-node labeling at task-graph build time, so providers instantiated after mapping receive their indices automatically.

Semantic changes vs main

• The constraints= argument is removed. Constraints declared on type variables (TypeVar('T', int, float), class Raw[Run: (A, B)]) still work, as filters during unification.
• Unconstrained type variables no longer raise at insertion; unsatisfiable setups fail at graph build as unsatisfied requirements.
• A specialized provider shadows a generic one regardless of insertion order (restores pre-cyclebane-rewrite semantics; previously a generic provider inserted later replaced specialized ones).
• reduce() requires an explicit key when the reduced sink would come from a generic provider; fully concrete pipelines are unaffected.
• Node names are strictly unique (Derive mapped-node labeling at compile time cyclebane#32): reduce(name=<name of the reduced node>) and pipeline[C] = pipeline[C].map(...).reduce(func=merge) now raise clear errors; use a distinct name for the mapped node vs. its reduction. This also turns a silent self-loop bug on cyclebane main into an error.
• output_keys() returns concrete sinks plus return-type patterns of generic providers not consumed by other generic providers; no-argument visualize() shows the concrete part of the graph. Inspection surfaces show demanded/derived state, not the constraint cross-product.

Evidence

Of the 248 tests before this change, 242 pass unchanged; six tested the removed constraints= argument. All 143 cyclebane tests pass with identical to_networkx() output. mypy clean (sciline), one pre-existing error fewer (cyclebane). New tests cover PEP 695 classes and aliases, chains, target-only instantiation, shadowing, multi-TypeVar providers, coherence errors, mapped pipelines with post-map instantiation, and generic params.

Before this could land

• User-guide rewrite: generic-providers.ipynb still documents constraints=, and the parameter-table docs use the now-forbidden reduce-graft idiom; the docs build fails on this branch.
• Decision on deprecating sl.Scope (Scope does not work with Python 3.12 generics #233) and a migration/deprecation story for constraints= (this branch hard-removes it to show the end state).
• Land and release Derive mapped-node labeling at compile time cyclebane#32; pin the cyclebane version.

🤖 Generated with Claude Code

[SimonHeybrock]

No description provided.

[SimonHeybrock]

An independent fresh-eyes review of both branches found three silent-wrong-answer holes, all where eager semantics were made lazy without being pinned by tests. Fixed in d0b184e (sciline) and scipp/cyclebane@3727304, each with regression tests: reduce without key now raises when the sink could come from a not-yet-instantiated generic provider (a unique sink not matched by any generic argument still works without key); TypeVar bound= is honored as a unification filter (was silently matching anything); demanded-key nesting depth is capped to catch non-terminating rule chains (P[list[T]] -> P[T]); CycleError now carries the cycle; get_mapped_node_names distinguishes 'not mapped' from an index_names mismatch. Cleanups: the four instantiation hooks are unified behind one _demanded helper. The cyclebane side resolves reduce specs eagerly into index-name sets, restoring main semantics for map-after-reduce, and drops stale reduce specs on branch replacement. Suites: sciline 250, cyclebane 146, both green.