Orphan-synthesized dispatch spans lose attributes when the wrapper short-circuits

[chris-colinsky]

Blocks v0.17.0. Own PR — three related gaps in the same repair path, all found reviewing #277.

_synthesize_call_site_wrapper_spans opens a dispatch span from a provider or tool event. Those events carry lineage but not subgraph_identities, and may carry no caller_invocation_metadata. The repair for that is a backfill from the first inner node event — which never arrives if the wrapper short-circuits.

1. Subgraph identity is not repaired when no node event follows

Fan-out node declared with subgraph_identity="leaf_identity" and an instance_middleware that issues a provider call then returns a partial without calling next_call (the FailureIsolation / early-exit shape). No inner node event is emitted, _sync_subgraph_spans never runs for that prefix, and the backfill never executes. The instance dispatch span keeps openarmature.subgraph.name='' despite the declared identity.

2. Caller metadata is never backfilled at all

A dispatch span synthesized from a provider event may lack the §5.6 cross-cutting openarmature.user.* entirely. The event's caller_invocation_metadata is read defensively (so it no longer crashes), but nothing repairs it later. Same short-circuit shape makes it permanent.

3. The detached arms are not mirrored

_synthesize_call_site_wrapper_spans mirrors only the two non-detached arms of _sync_subgraph_spans. For a fan-out registered in detached_fan_outs, the orphan's parent is still decided by drain scheduling — and now differs in trace id, not just parent, which is a larger divergence than the one #277 set out to close.

Direction

(1) and (2) want the same thing: the synthesis path should source what it can from the invocation rather than only from the event, and the backfill should not be the sole repair. (3) is either mirroring the two detached arms or an explicit, tested carve-out — but not silence.

Each needs a test with the wrapper short-circuiting, since every existing test lets the inner node run.

[chris-colinsky]

Gaps 1 and 2 are fixed in #285. Gap 3 is not, and the reason is worth recording here rather than in a closed PR.

Mirroring the two detached arms into _synthesize_call_site_wrapper_spans was attempted and reverted. An adversarial review found four blockers, three of which exist only once that path can reach the detached openers.

The root cause is pre-existing and blocks any attempt. _sync_subgraph_spans guards its detached fan-out arm with if prefix in inv_state.detached_roots — the bare prefix — while _open_detached_fan_out_instance_root stores under prefix + (str(fan_out_index),). That guard has never matched. With one caller it is dead code; with two it is a double-open that overwrites the first root, leaks two unended spans, and strands the orphan in an abandoned trace.

The other three:

• The detached openers assume an invocation span already exists. That holds from the node path, which runs strictly after _open_invocation_span, and not from the orphan path. A call arriving first mints a disconnected trace, and the real parent trace permanently loses its §4.4 Link.
• _open_detached_subgraph_root reads the identity off the triggering event. An identity-less provider event therefore blanks openarmature.subgraph.name on a subgraph that declares one — a measured regression against current behaviour, and the same defect class Recover the subgraph identity and caller metadata a short-circuiting wrapper loses #285 fixes for fan-out.
• Changing that opener's instance index to the chain-at-depth broke agreement with every reader, which still keys by the innermost event.fan_out_index.

Suggested order for the next attempt:

1. Fix the guard/store key-shape mismatch in _sync_subgraph_spans first, on its own, with a test. Nothing else is safe until both callers agree on the key.
2. Give _open_detached_subgraph_root an identity source that does not depend on the triggering event, the way Recover the subgraph identity and caller metadata a short-circuiting wrapper loses #285 did for fan-out, so routing it through the orphan path cannot regress the attribute.
3. Establish the invocation-span precondition in the orphan path, or gate the detached arms on it existing.
4. Only then mirror the arms, and re-check the instance-index question against every reader.

A guard test for step 1 that fails today would be a good first commit.

Consequence of leaving it: for a wrapper-issued call inside a detached wrapper the enclosing span is still decided by drain scheduling, and the divergence is in trace id rather than only in parent.

[chris-colinsky]

Step 1 of the rework is open as #286.

It fixes the key-shape mismatch named in the plan above, and the finding turned out stronger than that plan assumed. The mismatch is not latent, waiting for a second caller to make it a defect — it is live on main today from an ordinary graph.

A detached fan-out whose instance subgraph has two nodes re-opens the instance root once per inner node event, because the guard tests the bare prefix and the opener stores under prefix + (str(fan_out_index),):

_open_detached_fan_out_instance_root called 2 time(s) for 1 instance
traces: 3                     <- expected 2
  ['a']                       <- alone, parent never exported
  ['b', 'fo', 'openarmature.invocation']
  ['fo', 'openarmature.invocation']

The second open replaces the first in both detached_roots and detached_invocation_spans, so the first root and its detached invocation span are never ended and never exported. A single-node instance subgraph hides it entirely, which is why nothing caught it.

test_detached_fan_out_instance_opens_one_root_per_instance fails on main and passes with the fix.

Remaining steps, unchanged:

2. Give _open_detached_subgraph_root an identity source independent of the triggering event, so routing it through the orphan path cannot blank openarmature.subgraph.name.
3. Establish the invocation-span precondition in the orphan path, or gate the detached arms on it existing.
4. Only then mirror the arms, re-checking the instance-index question against every reader.

Status of this issue overall: gaps 1 and 2 shipped in #285 (the subgraph identity now rides FanOutEventConfig; FailureIsolatedEvent carries caller_invocation_metadata). Gap 3, the detached-trace half, is what remains and is what these four steps cover.

[chris-colinsky]

Step 1 landed: #286 is merged (b8dd490).

Verified on merged main rather than trusting CI: the two-node detached instance subgraph now produces 2 traces instead of 3, the instance root opens once instead of twice, and both inner nodes share a trace.

A mutation pass on the merged shape, five mutants, four killed by four different tests:

• remove the instance guard → killed by the new test
• guard on the bare prefix (the original defect shape) → killed by the new test
• guard on the right shape with a constant index → killed by conformance 008
• remove the subgraph_spans guard → killed by conformance 002
• remove the bare-prefix detached_roots guard → survived

That survivor is redundant code, not a coverage gap. _open_detached_subgraph_root writes subgraph_spans[prefix] as well as detached_roots[prefix], and the subgraph_spans guard runs first, so the bare-prefix test can never be the one that fires. It is kept as depth and the comment now says so explicitly, including that it must not be read as load-bearing.

Next is step 2: give _open_detached_subgraph_root an identity source independent of the triggering event, so routing it through the orphan path cannot blank openarmature.subgraph.name. Same move that worked for fan-out in #285, where the declared identity now rides FanOutEventConfig and the observer caches it off the node's own started event.

Steps 3 and 4 unchanged: establish the invocation-span precondition in the orphan path, then mirror the arms and re-check the instance-index question against every reader.

[chris-colinsky]

Gap 3 is parked pending a spec answer. Raised as coord release-v0.17.0/47.

Step 2 turned out to be under-scoped. The plan read like the fan-out fix in #285: give the opener an identity source that does not depend on the triggering event. That worked for fan-out because a fan-out node emits an event at its own namespace, which always precedes its instances, so the declared identity could ride FanOutEventConfig.

A subgraph node emits no event at its own namespace. Its declared identity reaches an observer only through subgraph_identities on the events of nodes inside it. In the shape step 4 creates — a detached subgraph whose entry node's middleware issues a call before next_call — the provider event arrives first, so an observer-side cache is still empty when the opener runs. There is no earlier carrier. An observer cannot know the identity.

The only sound fix is the shape proposal 0045 used for the lineage chains: a ContextVar the engine sets on descent, stamped onto provider events. Measured scope:

• 11 event kinds carry fan_out_index_chain / branch_name_chain and not subgraph_identities; NodeEvent is the only one that has it
• 4 builder sites stamp the chains and would each need the new field
• 1 new ContextVar, set and reset at 3 subgraph-descent sites

That is a fourth event-surface addition in this release, after FanOutEventConfig.subgraph_identity and FailureIsolatedEvent.caller_invocation_metadata in #285. All are additive and permitted, but a second implementation has no reason to add any of them and no fixture can assert them portably. So the question went to spec as a class rather than a fourth instance: when an observer synthesizes a dispatch span from a provider event, which lineage attribution is it entitled to, and which is it required to render?

Either answer resumes this cleanly. If all of it, subgraph_identities belongs on the provider events beside 0045's chains, as a proposal. If best-effort, the empty name is correct and this stops being a gap.

A narrower option was considered and rejected for now: mirror only the detached fan-out arm in step 4 and leave the subgraph arm unmirrored. It regresses nothing — the detached fan-out opener sets no openarmature.subgraph.name today either — and closes most of gap 3. It stays available if the spec answer is slow.

Consequence of the hold: for a wrapper-issued call inside a detached wrapper, the enclosing span is still decided by drain scheduling, and the divergence is in trace id rather than only in parent.

Steps 1 (#286) and the gap 1/2 work (#285) are unaffected and merged.