Rewind after a multi-failed-sibling WhenAll deadlocks the UNSIGNALED sibling subtree (emit-once replay can't reconcile a re-driven orphan with no rewind signal)

[YunchuWang]

Summary

PR #300 (fixes #299) closes the activity-level rewind orphan: after a server-side (Azure Storage / DurableTask.Core) rewind, a Task.all fan-out with multiple failed activity siblings now re-dispatches all of them on the rewound (signaled) path.

However, the extension E2E RewindOrchestratorTests.RewindFailedOrchestration_ShouldSucceed(1|2) (backend = Azure Storage) exercises a two-level version of the same pattern and still deadlocks, for a distinct, deeper reason that #300 does not — and by design cannot safely — address. This issue tracks that residual gap.

VERIFIED (from live Azure Storage worker logs)

Test / topology. The test asserts the root reaches Completed (RewindOrchestratorTests.cs:50). The orchestrator (test/e2e/Apps/.../functions/RewindOrchestration.ts) is two levels deep:

• Root RewindParentOrchestration — Task.all of four sub-orchestrations, two of which fail (RewindOrchestration.ts L26-37): [SucceedSub, FailParentSub(fail_parent_sub_1), FailParentSub(fail_parent_sub_2), SucceedSub].
• Each FailParentSubOrchestration (L48-59) calls one FailChildSubOrchestration.
• Each FailChildSubOrchestration (L61-82) does a Task.all of two FailActivity siblings (+ one succeeding activity) — this inner fan-out is exactly Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299.

Mechanism.

1. The root Task.all fails fast on the first sub-orchestration failure. The second failing sub-orch's failure is a late completion for a now-terminal parent and is dropped (never committed) — the same drop mechanism Fix rewind deadlock when Task.all had multiple failed siblings (#299) #300 describes, one level up.
2. DurableTask.Core server-side rewind therefore rewinds only the committed-failed subtree and wakes only that subtree's grandchild with the revival GenericEvent.
3. The Fix rewind deadlock when Task.all had multiple failed siblings (#299) #300 fix reaches the signaled grandchild and works: it logs Waiting for 2 → Returning 2 (both failed activity siblings re-dispatched) and completes.
4. The other failing subtree is re-driven by the root's replay with no rewind signal. Its grandchild hits the identical Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 activity orphan, but the reconcile is (correctly) gated on the rewind signal (isRewindReplay), so it cannot fire → Waiting for 2 → Returning 0 → deadlock → root never reaches Completed → 30s timeout.

Verbatim per-grandchild evidence (numFailures=2, single-TFM, one clean instance):

grandchild instance	rewind GenericEvent	inner Task.all outcome
…:0002:0002	2	Returning 2 → FailChildSubOrchestration completed ✅
…:0003:0002	0	stuck at Waiting for 2 → Returning 0, never completes ❌

The dropped branch is non-deterministic (a race): in a numFailures=1 run the orphaned branch was …:0002:0002 (GenericEvent=0) while …:0003:0002 was signaled and completed — the opposite of the numFailures=2 run above.

Zero NonDeterminismError in every run (numFailures 1 & 2, pass and fail). The line-50 failures are deadlocks, not throws — so this is distinct from the duplicate-TaskScheduled concern #300's idempotent guard addresses.

Why the reconcile gate cannot simply be removed. The worker cannot distinguish a stale orphan (no completion will ever arrive — must re-dispatch) from a genuinely in-flight scheduled-but-uncompleted task (completion coming later — must not re-dispatch) without a rewind signal. Un-gating the reconcile would re-dispatch in-flight activities during ordinary replay → double execution. The signal (GenericEvent / ExecutionRewound) is the only structural discriminator, and the backend sends it only to the rewound subtree.

Root-cause framing (why v3 / DurableTask.Core pass and this doesn't)

The classic durable-functions v3 SDK (and DurableTask.Core) pass this same E2E because their replay model re-emits the full action set every episode and the backend dedupes via TaskScheduled. An unsignaled re-driven orphan is harmless there — the action is simply re-emitted and de-duplicated.

durabletask-js (and durabletask-python) use an emit-once + gated-reconcile replay model, which is not inherently robust to a missing rewind signal. So the class "rewind after a multi-failed-sibling WhenAll" is only partially closed by #300 (the signaled, activity-level path); the unsignaled sub-orchestration path remains open.

durabletask-python shares this exact gap (see #300's Sister-SDK parity note): the same pop-orphan taskScheduled handler, the same get_actions() = list(_pending_actions.values()), and no rewind-revival recovery.

HYPOTHESIS / candidate directions (need design + cross-SDK review — NOT prescribed here)

• (B) Durably record all WhenAll sibling terminal outcomes before the instance goes terminal, so server-side rewind rewinds every failed path and every subtree receives a revival signal. Removes the drop at the source; needs care that the user-visible fail-fast result semantics are preserved while the sibling failures are still persisted.
• (C) A v3-parity re-emit/dedup replay-model change in the worker, so a missing signal is no longer load-bearing. Larger, cross-cutting change.

Both need design review and should be coordinated with durabletask-python (shared gap). This issue deliberately does not prescribe a direction.

Scope / relationship to #299 and #300

• Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 / Fix rewind deadlock when Task.all had multiple failed siblings (#299) #300: the activity-level orphan on the signaled rewound path — fixed and verified (grandchild Returning 2 + zero NonDeterminismError on live Azure Storage).
• This issue: the sub-orchestration-level, unsignaled orphan exposed by the two-level E2E — remains open.

Refs: #299, #300.

Fidelity note

The live E2E was run with the #300 fix logic ported onto the #282 (gRPC-worker) core — the fix anchors are byte-identical — with callEntities=false and a single target framework (--framework net8.0) to force a single deterministic instance. The residual-failure mechanism is core-level (WhenAll fail-fast drop + backend rewind scoping + gated reconcile), independent of that harness detail.