Skip to content

Derive mapped-node labeling at compile time - #32

Draft
SimonHeybrock wants to merge 3 commits into
mainfrom
derive-mapped-labels-at-compile
Draft

Derive mapped-node labeling at compile time#32
SimonHeybrock wants to merge 3 commits into
mainfrom
derive-mapped-labels-at-compile

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

Spike supporting the discussion in scipp/sciline#237 (demand-driven generic providers) — see the design document there for context. Draft for discussion.

What changes

Graph.map no longer relabels the descendants of mapped roots as MappedNode. The stored graph keeps original node names at all times; which nodes carry which indices is derived inside to_networkx() from reachability of the mapped roots plus recorded reduce specs, and the MappedNode representation now exists only transiently during compilation. The observable output of to_networkx() is unchanged.

The practical consequence: map() commutes with adding branches to the graph. Sciline needs this so that generic providers can be instantiated on demand after mapping — the mapped-labeling of late-added dependents falls out of the derivation instead of requiring eager instantiation before map().

reduce() records what it consumes (index/axis/all, plus groupby's extra index) in a side table; the remaining indices of the reduce node are derived at compile time. Validation still happens eagerly at call time against the currently derivable indices.

The shadowing seam

Idioms like reduce(name=C, index='x') and pipeline[C] = pipeline[C].map(...).reduce(func=merge) give the reduce result the same public name as the mapped node it reduces. Previously this worked because the mapped node was labeled MappedNode(C) while the result was plain C — the dual identity was implicit and global. With plain names it is a genuine collision. The dual identity turns out to be irreducible exactly at these seams, so it is kept there and only there: when a reduce result or an assigned branch shadows a mapped node, the mapped node is relabeled to an explicit MappedNode alias. New accessors (node_indices, named_indices, value_keys) expose derived indices to callers that previously scanned for MappedNode labels (sciline's get_mapped_node_names).

_from_orig_key and the eager relabeling helpers are gone; __getitem__/__delitem__/__setitem__ lose their MappedNode special cases.

Found along the way

graph1['c'] = graph2['d'] on main merges the branch's own 'c' node with the destination during sink-renaming, producing a self-loop (reproducible on main; invisible because nothing topologically sorts the stored graph). The index derivation here tolerates self-loops to stay behavior-compatible; the underlying __setitem__ issue deserves a separate fix.

Verification

All 143 tests pass unchanged. mypy: one error fewer than main (22 vs 23 pre-existing). The sciline side of the spike (all 245 sciline tests green, forward-instantiation hook removed) is in scipp/sciline — branch 235-deferred-mapped-labels.

🤖 Generated with Claude Code

Graph.map no longer relabels the descendants of mapped roots. The graph
keeps original node names; which nodes carry which indices is derived in
to_networkx from reachability of mapped roots and recorded reduce specs.
This makes map() commute with adding branches to the graph, which
sciline needs in order to instantiate generic providers on demand after
mapping (see scipp/sciline#235).

reduce() records what it consumes in a side table instead of computing
the remaining indices eagerly; validation still happens at call time.
Where a reduce result or an assigned branch shadows a mapped node of the
same name (e.g. pipeline[C] = pipeline[C].map(...).reduce(...)), the
mapped node is relabeled to an explicit MappedNode alias so both can
coexist; this replaces the implicit distinction that relabeling
provided. The new named_indices/node_indices/value_keys accessors expose
derived indices to users that previously scanned for MappedNode labels.

Also makes the derivation tolerant of self-loops, which __setitem__ can
create when the new branch contains its destination node; this could
already happen before this change and needs a separate fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Node names are now strictly unique: reduce() raises when the result name
already exists (including the previously special-cased mapped node), and
__setitem__ raises when the new branch contains a node with the
destination name. This replaces the MappedNode alias mechanism for
idioms like pipeline[C] = pipeline[C].map(...).reduce(func=merge);
callers must use distinct names for the mapped node and its reduction.
The named_indices accessor and the self-loop tolerance in the index
derivation are removed; the strict __setitem__ error also fixes the
silent self-loop these idioms produced.

Setting a mapped branch at a node name that does not exist yet is now
allowed; the mapped/unmapped consistency check only applies when
replacing an existing branch, since indices follow from derivation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@SimonHeybrock

Copy link
Copy Markdown
Member Author

Follow-up per discussion (a046ffb): the MappedNode alias mechanism is removed in favor of strict name uniqueness — reduce() and __setitem__ now raise clear errors when the new node would shadow a like-named node inside the branch (the pipeline[C] = pipeline[C].map(...).reduce(...) idiom and reduce(name=<reduced node>)). This also turns the silent self-loop on main into an error, and deletes named_indices plus the self-loop tolerance (net −47 lines vs. the previous state of this PR). Grafting a mapped branch at a new node name is now allowed, since indices follow from derivation. The explicit escape hatch for subgraph reuse would be a future Graph.rename.

An independent review found that applying reduce specs lazily while
validating them eagerly diverges from main semantics when a later map
adds indices reaching the reduce node: axis-based specs dropped the
wrong index and full reductions swallowed the later indices. Specs now
resolve into a concrete set of index names to drop when reduce() is
called, making validation and application the same computation and
letting later indices flow through the reduce node, as with eager
relabeling.

Also fixes __setitem__ leaving a stale reduce spec when a reduce node
is replaced by a plain branch, updates docstrings that still described
map-time relabeling, and trims dead generality (_labeled_key,
_node_name in __setitem__).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant