fix: recover orchestrators stuck stopping

[sihanwang94]

Problem

An orchestrator could remain indefinitely in Stopping after the Host and owning Node restarted. The persisted session retained stopRequested=true, but the previous recovery path only reconciled sessions whose state was exactly offline. Older Stop intents could therefore be skipped, Resume stayed blocked, and the UI did not explain which Node still owed the acknowledgement.

The live affected example was UX E2E Tests. Its lead session belonged to Node Sihan, which had not reconnected after the Host restart. The owned worker sessions were already stopped, while the lead remained offline with a pending Stop request.

Root cause

Stop completion requires authoritative knowledge from the Node that owns the process:

• A Host restart parks active sessions offline but cannot know whether their operating-system processes still exist.
• A reconnecting Node inventory can prove whether a session is present or absent.
• If the owning Node never reconnects, automatically claiming the session stopped would be unsafe because the disconnected process could still be running.
• The original reconciliation query did not include every persisted, non-queued Stop intent, leaving some restart-era rows outside recovery.

Changes

Reconcile persisted Stop intents

• Reconciliation now includes sessions that are offline or have stopRequested=true, except queued launches.
• Queued sessions are deliberately excluded because a newly dispatched process may not have appeared in Node inventory yet; treating temporary absence as stopped could orphan a process that starts moments later.
• Terminal sessions clear stale Stop intent without redispatching Stop.

Recover from authoritative Node inventory

When the owning Node reconnects:

1. If the session is present in inventory, Host restores its live state and reissues Stop once.
2. If the session is absent, Host settles it as stopped and clears stopRequested.
3. Subsequent heartbeats do not repeatedly resend Stop after the offline-to-live reconciliation has completed.

This preserves Node inventory as the primary source of truth and keeps Stop handling idempotent.

Recover when the Node remains unavailable

• Offline sessions with a pending Stop now expose Mark stopped.
• Orchestrator detail exposes Mark orchestrator stopped for an offline lead with pending Stop.
• Confirming the action settles the offline lead and its offline owned workers, then clears their Stop intents.
• The recovery path does not guess that online sessions or sessions on an available Node have stopped.
• Resume becomes available only after the Stop intent is actually settled.

Make the pending state explicit

• The conversation UI names the unavailable Node, for example: Stop is waiting for the offline node Sihan.
• The warning explains the two safe paths: reconnect the Node for authoritative reconciliation, or use Mark stopped only when the operator knows the old process is gone.
• Resume remains hidden while acknowledgement is genuinely pending.

Recovery precedence and safety

1. Reconnected Node inventory is authoritative.
2. Present sessions receive one Stop reissue.
3. Absent sessions settle automatically as stopped.
4. A Node that never reconnects does not cause Host to invent a successful stop.
5. Manual operator confirmation is available for the unavailable-Node case.

The implementation intentionally does not use Host startup's reset activeSessions=0 value as proof that work ended, because that value describes Host connectivity rather than the remote process lifecycle.

User-visible result

Orchestrators no longer remain permanently stranded in Stopping after normal reconnect reconciliation. When the owning Node is unavailable, the UI clearly identifies that Node and gives the operator a safe, explicit way to settle known-dead work and Resume the conversation.

Validation

• Added lifecycle coverage for restart recovery, missing/present inventory, terminal Stop intents, and repeated heartbeat idempotency.
• Added route coverage for unavailable-Node confirmation, lead/worker settlement, Stop/Resume idempotency, and refusal to guess while a Node is online.
• Added Terminal and orchestrator-page coverage for warning text, Node identity, Mark stopped actions, and Resume gating.
• Ran npm run verify successfully, including lint, formatting, type checking, the full test suite, and production Host/Node builds.

[coderabbitai]

Warning

Review limit reached

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: b9112575-3ac9-40e5-9cf7-7fe47a889e2f

📥 Commits

Reviewing files that changed from the base of the PR and between c551bc3 and 6cb1aaa.

📒 Files selected for processing (9)

• apps/host/src/fleet-service.ts
• apps/host/src/orchestrator/lifecycle.test.ts
• apps/host/src/routes/orchestrators.test.ts
• apps/host/src/routes/orchestrators.ts
• apps/host/src/store.ts
• apps/host/ui/src/components/TerminalView.test.tsx
• apps/host/ui/src/components/TerminalView.tsx
• apps/host/ui/src/components/orchestration/OrchestratorPage.test.tsx
• apps/host/ui/src/components/orchestration/OrchestratorPage.tsx

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}