Guard the detached instance root on the key it is stored under - #286
Merged
Conversation
`_sync_subgraph_spans` tested `if prefix in inv_state.detached_roots`, the bare prefix, while `_open_detached_fan_out_instance_root` stores under `prefix + (str(fan_out_index),)`. The guard therefore never matched a detached fan-out instance, and the arm fired once per inner NODE EVENT rather than once per instance. Reachable on an ordinary graph: a detached fan-out whose instance subgraph has two nodes. Each node's started event re-opened the root, the second replacing the first in both dicts, so the first root and its detached invocation span were never ended and never exported. Measured: three traces instead of two, the first inner node alone in a trace of its own, and two spans carrying a parent span id nothing emitted. Invisible with a single-node instance subgraph, which is why no existing test saw it: one event, one open. The instance arm now tests its own key. The bare-prefix test above it stays, because a detached SUBGRAPH root really is stored that way; the two shapes share one dict and that is what made the mismatch easy to miss. This is step one of the #279 rework. Nothing else in that path is safe until both callers agree on the key: the previous attempt mirrored the detached arms first and turned this latent mismatch into a double-open, which was one of four blockers an adversarial review found.
This was referenced Aug 29, 2026
There was a problem hiding this comment.
Pull request overview
This PR fixes a detached fan-out observability defect in the OTel observer where a detached fan-out instance root was guarded using the wrong key shape, causing a root to be re-opened per inner node event and leading to abandoned spans and incorrect trace grouping.
Changes:
- Fix
_sync_subgraph_spansto guard detached fan-out instance roots using the same key shape that_open_detached_fan_out_instance_rootstores under. - Add a regression test asserting exported-span trace shape for a detached fan-out instance with a multi-node instance subgraph.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/unit/test_observability_otel.py | Adds a regression test for detached fan-out instance root opening and resulting trace/parentage shape. |
| src/openarmature/observability/otel/observer.py | Corrects detached fan-out instance root guarding to prevent per-inner-node double-open and abandoned roots. |
Suppressed comments (1)
tests/unit/test_observability_otel.py:6455
- Like the rest of this test module, prefer
.context.trace_idon exported spans instead ofget_span_context(), which may not exist on ReadableSpan and makes this test more brittle than necessary.
trace_of = {s.name: cast("Any", s.get_span_context()).trace_id for s in spans}
assert trace_of["a"] == trace_of["b"], (
"the two inner nodes of one fan-out instance landed in different traces"
)
trace_count = len({cast("Any", s.get_span_context()).trace_id for s in spans})
assert trace_count == 2, f"expected the parent trace plus one detached instance trace, got {trace_count}"
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The new test read `cast("Any", s.get_span_context()).span_id`, which
silences pyright without checking anything. This file's idiom is
`assert s.context is not None` followed by `.context.trace_id`, used at
16 sites against 5 for `get_span_context()`.
Switching to `.context` alone does not help: it is equally Optional and
produces the same three errors. So the three casts are now one
`_span_ctx` helper that asserts and returns the context, enforcing the
invariant at runtime rather than asserting it away.
Guard test re-verified after the refactor: removing the instance-key
guard still turns it red.
The comment claimed it stays "for the subgraph shape", implying it is the guard that catches a detached subgraph root. It is not. `_open_detached_subgraph_root` writes `subgraph_spans[prefix]` as well as `detached_roots[prefix]`, and the `subgraph_spans` guard immediately above runs first, so this test can never be the one that fires. Established by mutation: deleting this guard leaves the whole suite green, while deleting the `subgraph_spans` guard is caught by conformance fixture 002. Kept as depth, since a future path could populate `detached_roots` without `subgraph_spans` and the #279 rework may add one. The comment now says that, so nobody keeps it believing it is load-bearing or deletes it believing the stated reason was the real one.
A user-visible fix: three traces where there should be two, one instance's inner nodes split across them, and spans pointing at a parent that was never exported. The entry names the two-node condition, since that is what makes it visible and is not something a reader would guess.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step one of the #279 rework. Small on purpose: nothing else in the detached path is safe until both callers agree on the key.
The defect
_sync_subgraph_spanstestedif prefix in inv_state.detached_roots— the bare prefix — while_open_detached_fan_out_instance_rootstores underprefix + (str(fan_out_index),). The guard therefore never matched a detached fan-out instance, and the arm fired once per inner node event rather than once per instance.Reachable on an ordinary graph, on
maintoday, with nothing from the reverted work: a detached fan-out whose instance subgraph has two nodes.The second open replaced the first in both
detached_rootsanddetached_invocation_spans, so the first root and its detached invocation span were never ended and never exported. One instance's inner nodes ended up in different traces, with the first pointing at a parent span id nothing emitted.It is invisible with a single-node instance subgraph, which is why no existing test saw it: one event, one open.
The fix
The instance arm now tests its own key. The bare-prefix test above it stays, because a detached subgraph root really is stored that way. Both shapes share one dict, which is what made the mismatch easy to miss, and the comment now says so.
Why this lands alone
The previous attempt at #279's gap 3 mirrored the detached arms into orphan synthesis first. That turned this latent mismatch into a live double-open, and it was one of four blockers an adversarial review found, which is why the whole mirroring was reverted.
Remaining steps, in order, recorded on #279:
_open_detached_subgraph_rootan identity source independent of the triggering event, so routing it through the orphan path cannot blankopenarmature.subgraph.name.Tests
test_detached_fan_out_instance_opens_one_root_per_instancefails onmainand passes here; removing the guard turns it red again.It asserts the trace shape a consumer depends on rather than the call count: no span left pointing at an unexported parent, the instance's inner nodes sharing one trace, and exactly two traces.
openarmature.invocationis excluded from the dangling-parent check deliberately and with the reason stated — a detached invocation span is parented under a syntheticNonRecordingSpanthat carries the new trace id and is never exported by design, with the fan-out node span holding a Link to it instead.2196 passing; ruff, format and pyright clean.