Derive mapped-node labeling at compile time - #32
Draft
SimonHeybrock wants to merge 3 commits into
Draft
Conversation
Graph.map no longer relabels the descendants of mapped roots. The graph keeps original node names; which nodes carry which indices is derived in to_networkx from reachability of mapped roots and recorded reduce specs. This makes map() commute with adding branches to the graph, which sciline needs in order to instantiate generic providers on demand after mapping (see scipp/sciline#235). reduce() records what it consumes in a side table instead of computing the remaining indices eagerly; validation still happens at call time. Where a reduce result or an assigned branch shadows a mapped node of the same name (e.g. pipeline[C] = pipeline[C].map(...).reduce(...)), the mapped node is relabeled to an explicit MappedNode alias so both can coexist; this replaces the implicit distinction that relabeling provided. The new named_indices/node_indices/value_keys accessors expose derived indices to users that previously scanned for MappedNode labels. Also makes the derivation tolerant of self-loops, which __setitem__ can create when the new branch contains its destination node; this could already happen before this change and needs a separate fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Node names are now strictly unique: reduce() raises when the result name already exists (including the previously special-cased mapped node), and __setitem__ raises when the new branch contains a node with the destination name. This replaces the MappedNode alias mechanism for idioms like pipeline[C] = pipeline[C].map(...).reduce(func=merge); callers must use distinct names for the mapped node and its reduction. The named_indices accessor and the self-loop tolerance in the index derivation are removed; the strict __setitem__ error also fixes the silent self-loop these idioms produced. Setting a mapped branch at a node name that does not exist yet is now allowed; the mapped/unmapped consistency check only applies when replacing an existing branch, since indices follow from derivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
Follow-up per discussion (a046ffb): the MappedNode alias mechanism is removed in favor of strict name uniqueness — |
This was referenced Aug 21, 2026
An independent review found that applying reduce specs lazily while validating them eagerly diverges from main semantics when a later map adds indices reaching the reduce node: axis-based specs dropped the wrong index and full reductions swallowed the later indices. Specs now resolve into a concrete set of index names to drop when reduce() is called, making validation and application the same computation and letting later indices flow through the reduce node, as with eager relabeling. Also fixes __setitem__ leaving a stale reduce spec when a reduce node is replaced by a plain branch, updates docstrings that still described map-time relabeling, and trims dead generality (_labeled_key, _node_name in __setitem__). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spike supporting the discussion in scipp/sciline#237 (demand-driven generic providers) — see the design document there for context. Draft for discussion.
What changes
Graph.mapno longer relabels the descendants of mapped roots asMappedNode. The stored graph keeps original node names at all times; which nodes carry which indices is derived insideto_networkx()from reachability of the mapped roots plus recorded reduce specs, and theMappedNoderepresentation now exists only transiently during compilation. The observable output ofto_networkx()is unchanged.The practical consequence:
map()commutes with adding branches to the graph. Sciline needs this so that generic providers can be instantiated on demand after mapping — the mapped-labeling of late-added dependents falls out of the derivation instead of requiring eager instantiation beforemap().reduce()records what it consumes (index/axis/all, plus groupby's extra index) in a side table; the remaining indices of the reduce node are derived at compile time. Validation still happens eagerly at call time against the currently derivable indices.The shadowing seam
Idioms like
reduce(name=C, index='x')andpipeline[C] = pipeline[C].map(...).reduce(func=merge)give the reduce result the same public name as the mapped node it reduces. Previously this worked because the mapped node was labeledMappedNode(C)while the result was plainC— the dual identity was implicit and global. With plain names it is a genuine collision. The dual identity turns out to be irreducible exactly at these seams, so it is kept there and only there: when a reduce result or an assigned branch shadows a mapped node, the mapped node is relabeled to an explicitMappedNodealias. New accessors (node_indices,named_indices,value_keys) expose derived indices to callers that previously scanned forMappedNodelabels (sciline'sget_mapped_node_names)._from_orig_keyand the eager relabeling helpers are gone;__getitem__/__delitem__/__setitem__lose theirMappedNodespecial cases.Found along the way
graph1['c'] = graph2['d']onmainmerges the branch's own'c'node with the destination during sink-renaming, producing a self-loop (reproducible onmain; invisible because nothing topologically sorts the stored graph). The index derivation here tolerates self-loops to stay behavior-compatible; the underlying__setitem__issue deserves a separate fix.Verification
All 143 tests pass unchanged. mypy: one error fewer than
main(22 vs 23 pre-existing). The sciline side of the spike (all 245 sciline tests green, forward-instantiation hook removed) is in scipp/sciline — branch235-deferred-mapped-labels.🤖 Generated with Claude Code