fix(synapse): rebind retries to the replacement daemon - #65
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
| const classified = classifyError(error); | ||
| if (classified.code !== "module_restarted" || restarted) throw classified; | ||
| restarted = true; | ||
| await this.rebindAfterModuleRestart(deadlineAt, signal); |
There was a problem hiding this comment.
Correctness: if rebindAfterModuleRestart throws here (either "timeout" because the page deadline already elapsed, or "transport" because the re-demand call failed), the exception propagates to the outer catch in embedItems (lines 939-950). Neither "timeout" nor "transport" is a permanent code (isPermanentSynapseCode, lines 188-198), so this.permanentFailure is never set and the break at line 948 never runs — the for loop simply continues to the next page.
But rebindAfterModuleRestart already reset this.compatibleDaemonId = null and this.initialized = false unconditionally as its first lines (799-801) before throwing. Nothing re-checks this.initialized or re-calls this.initialize() per page in embedItems's loop (only signal?.aborted || this.permanentFailure is checked, line 872) or in embedItemsDetailed's group/page loops (line 989) — runDetailedPage itself never calls initialize() either.
Failure scenario: a 3-page embedItems/embedItemsDetailed call hits module_restarted on page 1; the rebind's re-demand fails or the deadline has just elapsed. Pages 2..3 are then dispatched via callWithRetry (line 1592-1594) with expectedDaemonId omitted entirely, since this.compatibleDaemonId is stuck at null — silently bypassing the daemon-compatibility check this PR exists to enforce, for the rest of that top-level call. The only place state gets re-initialized is the single this.initialize() call at the very top of the next external invocation.
| } | ||
| } | ||
|
|
||
| private async rebindAfterModuleRestart( |
There was a problem hiding this comment.
Altitude: this bolts explicit "rebind after module_restarted" calls onto three separate catch sites (lines 906, 1168, 1267) rather than making compatibility self-healing inside initialize()/callWithRetry. module-transport.ts's ensureConnected (lines 1210-1290) already solves this exact problem — "the daemon identity may have changed under us" — lazily, by checking compatibleDaemonId at the top of every call and reconnecting/re-demanding as needed, so no call site needs to know about the failure mode.
Cost: recovery here is scattered and order-dependent. A future call path that can also surface module_restarted won't rebind unless someone remembers to bolt the same three lines on again — which is also the direct cause of the bug flagged on line 906 (a failed rebind leaves state stale for call sites that don't re-check it). The self-healing version one file away would have avoided both the duplication and that bug.
| signal?: AbortSignal, | ||
| ): Promise<void> { | ||
| this.initialized = false; | ||
| this.compatibleDaemonId = null; |
There was a problem hiding this comment.
Simplification: this reset is dead in every path. For connectionOrigin === "managed-default", initialize()'s managed-default branch (line 662) unconditionally sets this.compatibleDaemonId = null again before assigning any new value (line 698) — so this line's write is always overwritten before it matters. For any non-managed origin, compatibleDaemonId is never set anywhere else, so it's already null. Low-cost, but it misleads a reader into thinking this line carries meaning independent of what initialize() already guarantees.
| this.initialized = false; | ||
| this.compatibleDaemonId = null; | ||
| this.managedDemand = null; | ||
| const remainingMs = deadlineAt - Date.now(); |
There was a problem hiding this comment.
Reuse: this hand-rolls deadlineAt - Date.now() / "throw timeout if ≤0" arithmetic (repeated again at lines 879-888 and elsewhere in this file) instead of reusing the Deadline abstraction (remainingMs(), stageBudgetMs(...)) that module-transport.ts already factors this exact pattern through (lines ~1155-1183). This diff threads a caller deadline through three new call sites — a natural opportunity to converge on the shared abstraction instead of adding more copies of the same signed-remaining-time edge case that each need to get ≤ 0 right independently.
Summary
Synapse retry paths now discard a stale daemon identity after
module_restarted, rerun managed compatibility, and resubmit the same request key against the replacement daemon. Ephemeral and durable paths retain their original absolute page deadline through rebind, resubmission, and polling.Verification
bun test packages/plugin/src/features/magic-context/memory/embedding-synapse.test.ts: 47 passedbun run typecheckbun run lintStack
Layer 7 of 7 above #46. Parent: #64.