Skip to content

Guard the detached instance root on the key it is stored under - #286

Merged
chris-colinsky merged 4 commits into
mainfrom
fix/detached-root-key-shape
Aug 30, 2026
Merged

Guard the detached instance root on the key it is stored under#286
chris-colinsky merged 4 commits into
mainfrom
fix/detached-root-key-shape

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Step one of the #279 rework. Small on purpose: nothing else in the detached path is safe until both callers agree on the key.

The defect

_sync_subgraph_spans tested if prefix in inv_state.detached_roots — the bare prefix — while _open_detached_fan_out_instance_root stores under prefix + (str(fan_out_index),). The guard therefore never matched a detached fan-out instance, and the arm fired once per inner node event rather than once per instance.

Reachable on an ordinary graph, on main today, with nothing from the reverted work: a detached fan-out whose instance subgraph has two nodes.

_open_detached_fan_out_instance_root called 2 time(s) for 1 instance
   from ns=('fo', 'a')
   from ns=('fo', 'b')

traces: 3                     <- expected 2
  ['a']                       <- alone, parent never exported
  ['b', 'fo', 'openarmature.invocation']
  ['fo', 'openarmature.invocation']

The second open replaced the first in both detached_roots and detached_invocation_spans, so the first root and its detached invocation span were never ended and never exported. One instance's inner nodes ended up in different traces, with the first pointing at a parent span id nothing emitted.

It is invisible with a single-node instance subgraph, which is why no existing test saw it: one event, one open.

The fix

The instance arm now tests its own key. The bare-prefix test above it stays, because a detached subgraph root really is stored that way. Both shapes share one dict, which is what made the mismatch easy to miss, and the comment now says so.

Why this lands alone

The previous attempt at #279's gap 3 mirrored the detached arms into orphan synthesis first. That turned this latent mismatch into a live double-open, and it was one of four blockers an adversarial review found, which is why the whole mirroring was reverted.

Remaining steps, in order, recorded on #279:

  1. Give _open_detached_subgraph_root an identity source independent of the triggering event, so routing it through the orphan path cannot blank openarmature.subgraph.name.
  2. Establish the invocation-span precondition in the orphan path, or gate the detached arms on it existing.
  3. Only then mirror the arms, re-checking the instance-index question against every reader.

Tests

test_detached_fan_out_instance_opens_one_root_per_instance fails on main and passes here; removing the guard turns it red again.

It asserts the trace shape a consumer depends on rather than the call count: no span left pointing at an unexported parent, the instance's inner nodes sharing one trace, and exactly two traces. openarmature.invocation is excluded from the dangling-parent check deliberately and with the reason stated — a detached invocation span is parented under a synthetic NonRecordingSpan that carries the new trace id and is never exported by design, with the fan-out node span holding a Link to it instead.

2196 passing; ruff, format and pyright clean.

`_sync_subgraph_spans` tested `if prefix in inv_state.detached_roots`,
the bare prefix, while `_open_detached_fan_out_instance_root` stores
under `prefix + (str(fan_out_index),)`. The guard therefore never
matched a detached fan-out instance, and the arm fired once per inner
NODE EVENT rather than once per instance.

Reachable on an ordinary graph: a detached fan-out whose instance
subgraph has two nodes. Each node's started event re-opened the root,
the second replacing the first in both dicts, so the first root and its
detached invocation span were never ended and never exported. Measured:
three traces instead of two, the first inner node alone in a trace of
its own, and two spans carrying a parent span id nothing emitted.

Invisible with a single-node instance subgraph, which is why no
existing test saw it: one event, one open.

The instance arm now tests its own key. The bare-prefix test above it
stays, because a detached SUBGRAPH root really is stored that way; the
two shapes share one dict and that is what made the mismatch easy to
miss.

This is step one of the #279 rework. Nothing else in that path is safe
until both callers agree on the key: the previous attempt mirrored the
detached arms first and turned this latent mismatch into a double-open,
which was one of four blockers an adversarial review found.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a detached fan-out observability defect in the OTel observer where a detached fan-out instance root was guarded using the wrong key shape, causing a root to be re-opened per inner node event and leading to abandoned spans and incorrect trace grouping.

Changes:

  • Fix _sync_subgraph_spans to guard detached fan-out instance roots using the same key shape that _open_detached_fan_out_instance_root stores under.
  • Add a regression test asserting exported-span trace shape for a detached fan-out instance with a multi-node instance subgraph.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
tests/unit/test_observability_otel.py Adds a regression test for detached fan-out instance root opening and resulting trace/parentage shape.
src/openarmature/observability/otel/observer.py Corrects detached fan-out instance root guarding to prevent per-inner-node double-open and abandoned roots.
Suppressed comments (1)

tests/unit/test_observability_otel.py:6455

  • Like the rest of this test module, prefer .context.trace_id on exported spans instead of get_span_context(), which may not exist on ReadableSpan and makes this test more brittle than necessary.
    trace_of = {s.name: cast("Any", s.get_span_context()).trace_id for s in spans}
    assert trace_of["a"] == trace_of["b"], (
        "the two inner nodes of one fan-out instance landed in different traces"
    )
    trace_count = len({cast("Any", s.get_span_context()).trace_id for s in spans})
    assert trace_count == 2, f"expected the parent trace plus one detached instance trace, got {trace_count}"

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/unit/test_observability_otel.py Outdated
The new test read `cast("Any", s.get_span_context()).span_id`, which
silences pyright without checking anything. This file's idiom is
`assert s.context is not None` followed by `.context.trace_id`, used at
16 sites against 5 for `get_span_context()`.

Switching to `.context` alone does not help: it is equally Optional and
produces the same three errors. So the three casts are now one
`_span_ctx` helper that asserts and returns the context, enforcing the
invariant at runtime rather than asserting it away.

Guard test re-verified after the refactor: removing the instance-key
guard still turns it red.
The comment claimed it stays "for the subgraph shape", implying it is
the guard that catches a detached subgraph root. It is not.
`_open_detached_subgraph_root` writes `subgraph_spans[prefix]` as well
as `detached_roots[prefix]`, and the `subgraph_spans` guard immediately
above runs first, so this test can never be the one that fires.

Established by mutation: deleting this guard leaves the whole suite
green, while deleting the `subgraph_spans` guard is caught by
conformance fixture 002.

Kept as depth, since a future path could populate `detached_roots`
without `subgraph_spans` and the #279 rework may add one. The comment
now says that, so nobody keeps it believing it is load-bearing or
deletes it believing the stated reason was the real one.
A user-visible fix: three traces where there should be two, one
instance's inner nodes split across them, and spans pointing at a
parent that was never exported. The entry names the two-node condition,
since that is what makes it visible and is not something a reader would
guess.
@chris-colinsky
chris-colinsky merged commit b8dd490 into main Aug 30, 2026
6 checks passed
@chris-colinsky
chris-colinsky deleted the fix/detached-root-key-shape branch August 30, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants