feat(workflows): add dynamic declarative workflows - #618
Conversation
Add durable sequential workflow execution with bounded outputs, human review gates, cancellation, restart recovery, lineage-scoped locking, and SSE updates. Expose workflow creation and monitoring in the right panel and through the CodeNomad plugin bridge. Keep approval mutations on the authenticated user API so plugin credentials cannot bypass a human gate, and preserve workspace lineage across desktop restore. Cover runtime transitions, restart rebinding, abort containment, route scoping, client reconciliation, restore identity, and plugin messaging with focused tests.
Resolve right-panel customization conflicts by registering Workflows through the modular tab registry while retaining keyboard tab navigation and ARIA panel relationships. Combine workflow and customization translations, keep generic persisted tab IDs, and preserve all upstream settings, auth, filesystem, permission, and Linux launch fixes. Validated with the UI typecheck and 78 focused registry, workflow, reconciliation, snapshot, and codec tests.
Add versioned JSON/YAML workflow definitions with branching, bounded parallelism, loops, nested workflows, gates, retries, structured outputs, budgets, pause/resume, crash recovery, and managed-worktree execution. Keep workflow mutation behind authenticated host APIs, scope plugin callbacks with per-workspace capabilities, restrict agent tools and plugin operations, and harden cancellation, persistence, CORS, process cleanup, ownership leases, and stale revision handling. Extend the right-panel workflow UI with declarative authoring, localized execution monitoring, durable drafts, recovery controls, safe output rendering, and retained-workspace session navigation while preserving the simple builder. Validated with server typechecks and the full 335-pass server suite, UI/plugin typechecks and builds, focused workflow/UI/Markdown tests, and a zero-finding gatekeeper review.
pascalandr
changed the title
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30383020047 Artifacts expire in 7 days.
|
Merge the background-process client-factory refactor from dev while preserving workflow cancellation semantics. The shared instance client now supports explicit session/worktree directories and composes Request, caller, and fallback timeout signals. Combined tests cover loopback routing, authorization, directory scoping, pre-dispatch cancellation, active cancellation, and timeouts. Validated with server and UI typechecks plus the full server suite: 345 passed, 4 platform-skipped, 0 failed.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30665339657 Artifacts expire in 7 days.
|
pascalandr
changed the title
Merge upstream/dev at c16cc00 into the workflow host branch. This adopts the first-party right-panel manifest runtime from NeuralNomadsAI#617 and registers Workflows through that contract while preserving stable tab IDs, keyboard navigation, localization, and existing customization behavior. The merge also carries the latest invalid OpenCode configuration diagnostics and retains both workspace lifecycle test paths. Resolutions were validated with server and UI typechecks, the UI production build, the full server suite, focused right-panel/workflow/workspace tests, and an independent regression review.
Keep the event loop referenced while the mocked stuck fetch waits for AbortSignal.timeout. Real fetch requests retain a socket handle, but the test double did not, allowing Node's test runner to cancel the suite before the unrefed timeout fired in CI. The timer is cleared after each test. The focused instance-client test and full server suite pass with no failures or cancellations.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30809608845 Artifacts expire in 7 days.
|
Expose saved-definition create and update operations to the workspace plugin, and attach plugin-started runs to the calling OpenCode session so existing family permissions and Yolo state apply naturally. Let omitted agent tool lists inherit normal OpenCode access, retain explicit installed-tool allowlists, and add an opt-in repeat exhaustion failure policy for bounded self-correction loops. Cover schema parsing, runtime behavior, plugin payloads, and route boundaries. Validated with both TypeScript projects, 62 focused tests, and the full 355-test server suite.
A concurrently starting modern Electron client can publish its legacy running marker just before its cross-host participant. On slower Windows runners, Tauri could observe that transient state and both clients could remain secondary. Recheck a blocking legacy marker once after a bounded 50 ms grace period while preserving PID, process identity, and executable validation. Add a deterministic participant-publication race test. Validated with rustfmt, five cross-host startup runs, and the complete 85-test Tauri suite on Windows.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30886836822 Artifacts expire in 7 days.
|
Allow agent nodes to declare an optional sessionKey. Nodes and repeat iterations sharing a key now continue the same OpenCode conversation, while nodes without a key retain the existing one-session-per-action behavior. Persist bounded key-to-session bindings in the workflow journal and serialize prompts per key so parallel branches cannot mutate one conversation concurrently. Existing crash recovery and confirmed session abort handling remain the side-effect boundary. Cover portable schema validation, persisted-state validation, repeat and cross-node reuse, parallel serialization, and restored bindings. Validated with server/UI/plugin typechecks and the complete server suite: 354 passed, 4 skipped.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30890442848 Artifacts expire in 7 days.
|
1 similar comment
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30890442848 Artifacts expire in 7 days.
|
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper review
Findings
-
High: Shared
sessionKeycancellation can falsely confirm termination without aborting the active OpenCode generation.packages/server/src/workflows/interpreter.ts:192,packages/server/src/workflows/interpreter.ts:657-670,packages/server/src/workflows/manager.ts:1085-1089,packages/server/src/workflows/manager.ts:1222-1229. The session limiter is released before the prior node callssessionFinished(). A waiting node can start another prompt on the same session, then the prior node deletes that session ID from the sharedSet. Cancellation subsequently treats the active session as already finished and skips the explicitsession.abort, allowing remote work or side effects to continue after the run is persisted as cancelled. -
High: Caller-controlled ancestry can borrow another session family's Yolo permissions.
packages/server/src/server/routes/workflows.ts:71-75,packages/server/src/server/routes/workflows.ts:247-251,packages/server/src/workflows/interpreter.ts:98-103,packages/server/src/permissions/auto-accept-store.ts:29-55. The plugin start route accepts anyinitiatorSessionIdunder the workspace bearer and uses it directly asparentID. A callback-token holder can nominate a different Yolo-enabled session, causing workflow descendants and their inherited tools to be auto-approved under that family. -
High: A workspace-scoped plugin capability can replace workflows used by every workspace.
packages/server/src/server/http-server.ts:247-269,packages/server/src/server/routes/workflows.ts:184-230,packages/server/src/workflows/manager.ts:128-131,packages/server/src/workflows/manager.ts:431-445. Plugin definition routes ignorerequest.params.idand mutate one global store. Because omitted tool lists inherit all plugin tools, a compromised or prompt-injected agent in workspace A can replace a trusted definition that later executes in workspace B. -
High: Worktree deletion races unpublished workspace startup.
packages/server/src/server/routes/worktrees.ts:165-176,packages/server/src/workspaces/manager.ts:166-174,packages/server/src/workspaces/manager.ts:392,packages/server/src/workspaces/manager.ts:496. Starting workspaces are reserved before publication but excluded fromlist()andget(). DELETE can therefore remove the worktree whileruntime.launchis using it; a new startup can also begin after the route's scan and beforegit worktree remove. -
High: A recovery marker with completed actions permanently reserves the workspace.
packages/server/src/workflows/manager.ts:991-1007,packages/server/src/workflows/manager.ts:1470-1476,packages/server/src/workflows/manager.ts:1612-1624,packages/server/src/workflows/manager.ts:661-666,packages/server/src/workflows/manager.ts:750-767. If terminal persistence fails after all actions complete, restart createsrecovery_requiredwith no ambiguous session IDs. Resume rejects it for lacking IDs, and cancellation also refuses to confirm termination, leaving no API path to release the reservation. -
Medium: Crashes before action admission are misclassified as ambiguous side effects.
packages/server/src/workflows/interpreter.ts:144-147,packages/server/src/workflows/interpreter.ts:451-455,packages/server/src/workflows/manager.ts:1627-1658. An action is persisted asrunningbefore its attempt is incremented. A crash while waiting for a limiter or listing tools leavesattempt === 0and no session, yet restart marks it unrecoverable; resume and cancel then cannot release the run despite no side effect having started. -
Medium: Repeat results and
onExhaustedoutcomes are not recovery-stable.packages/server/src/workflows/interpreter.ts:251-266,packages/server/src/workflows/manager.ts:1646-1654. The loop always resumes from index zero and evaluateswhilebefore consulting completed iterations. If the last completed body changes the condition to false and the process crashes before the repeat parent is persisted, recovery can omit that body's output and turn an exhaustion failure into success. A normal run also fails immediately after the last body without checking whether that body satisfied the exit condition. -
Medium: Confirmed abort/recovery resends a named prompt into the interrupted conversation.
packages/server/src/workflows/interpreter.ts:327-332,packages/server/src/workflows/interpreter.ts:404-422,packages/server/src/workflows/interpreter.ts:742-746,packages/server/src/workflows/manager.ts:1488-1499. Abort handling removes execution-node session references but retainssessionBindings. Retry or confirmed recovery therefore reuses the aborted session and appends the identical prompt again, potentially duplicating instructions or side effects already represented by the first user message. -
Medium: Human confirmations can authorize a different state from the one displayed.
packages/ui/src/components/instance/shell/right-panel/tabs/WorkflowRunList.tsx:49-71,packages/server/src/server/routes/workflows.ts:98,packages/server/src/workflows/manager.ts:639-668. Approval reads the reactive current gate only after the dialog closes; if another client advances from gate A to gate B, the stale dialog approves B. Recovery confirmation carries neither an expected revision nor interrupted-node identity, so a stale dialog can authorize a later, unrelated ambiguous side effect. -
Medium: The Tauri 50 ms retry does not close the participant-publication race.
packages/tauri-app/src-tauri/src/client_state/cross_host.rs:229-276,packages/tauri-app/src-tauri/src/client_state/cross_host.rs:296-320,packages/tauri-app/src-tauri/src/client_state/cross_host.rs:726-751. If modern Electron publishes its legacy marker but is suspended for longer than 50 ms before publishing its participant, Tauri remains permanently secondary. If Electron then crashes before acquiring ownership, Tauri never retries acquisition and client-state ownership remains vacant until restart. -
Medium: Side-effect checkpoints are atomic but not crash-durable.
packages/server/src/workflows/manager.ts:1328-1334,packages/server/src/workflows/manager.ts:1706-1720,packages/server/src/workflows/definition-store.ts:205-214. Temporary files are renamed without syncing the file or containing directory. A power or OS crash afterpersist()returns but after a prompt starts can restore an older checkpoint and cause recovery to repeat an effect it believes never started. -
Medium: Valid definition-store limits permit multi-gigabyte concurrent reads.
packages/server/src/workflows/definition-store.ts:24-25,packages/server/src/workflows/definition-store.ts:125-140. The store permits 1,000 histories of approximately 4 MiB each, whilelist()reads and parses every file concurrently. A catalog built entirely through valid plugin requests can make ordinary definition listing allocate several gigabytes and terminate the server. -
Medium: Workflow SSE delivery has unbounded buffering and quadratic snapshot traffic.
packages/server/src/workflows/manager.ts:1725-1729,packages/server/src/server/routes/events.ts:39-50,packages/server/src/workflows/definition-schema.ts:16,packages/server/src/workflows/interpreter.ts:18. Every checkpoint publishes the complete accumulated run, potentially containing 10,000 nodes and 4 MiB of output. The SSE writer ignores backpressure, so slow clients retain increasingly large snapshots in memory while large runs repeatedly serialize and transmit the entire history.
Open questions / assumptions
- Callback credentials are assumed not to confer arbitrary session-family identity.
- Per-workspace callback tokens are assumed not to be host-wide workflow-administrator credentials.
- OpenCode abort is assumed not to transactionally remove the original user message and partial response.
onExhausted: failis assumed to mean the exit condition remains true after all permitted attempts.- Power/OS-crash recovery is assumed to be in scope.
- Electron is assumed to have no enforced upper bound guaranteeing participant publication within 50 ms.
Validation gaps
- No cancellation test covers the handoff between two serialized prompts sharing one
sessionKey. - No restart test covers an action persisted as
runningwithattempt === 0. - No recovery test covers a completed repeat body whose parent node was not persisted.
- No test resolves or cancels marker-backed recovery containing only completed actions.
- No integration test combines plugin-start ancestry with the real
AutoAcceptManager. - No cross-workspace test defines plugin definition-mutation scope.
- No mounted UI test changes gate/recovery state while confirmation dialogs are open.
- The Tauri retry test does not cover publication after 50 ms or producer crash.
- No power-loss/fsync, maximum-catalog, slow-SSE-client, or unpublished-workspace deletion-race test exists.
Existing checks at review time: 354/358 server tests with 4 skipped, 85/85 Tauri tests, server/UI typechecks, plugin build, and all PR CI jobs green.
Remove plugin-controlled workflow ancestry and global definition mutations so workspace capabilities cannot borrow Yolo families or rewrite host-wide definitions. Make workflow session handoff, retry cleanup, repeat recovery, stale confirmations, terminal checkpoints, recovery markers, definition catalogs, and SSE delivery fail safely across cancellation, restart, slow clients, and power-loss boundaries. Serialize managed-worktree create/delete/start operations and retain failed cleanup ownership. Extend Tauri cross-host election with bounded late promotion, authoritative state reload and migration, and renderer ownership notifications. Validated with 369 passing server tests plus 4 platform skips, 91 Tauri tests, focused UI/plugin tests, server/UI/desktop typechecks, and the plugin build.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head 897c3cd1
Merge remains blocked. The previous 13 findings are materially improved, but this pass found the following critical/high issues:
- Critical - Windows command injection through managed-worktree slugs. A
git.cmd/git.batshim plus a slug containing shell metacharacters can escape the generatedcmd.execommand. Files:packages/server/src/workspaces/spawn.ts,git-worktrees.ts. - High - plugin HTTPS disables certificate verification while sending the callback capability.
rejectUnauthorized: falsepermits capability interception. File:packages/opencode-plugin/plugin/lib/request.ts. - High - workflow and definition admission remain process-local. Two supported hosts sharing one config directory can start conflicting workflows or acknowledge the same definition revision concurrently. Files:
workflows/manager.ts,definition-store.ts. - High - confirmed abort cleanup resets the durable admission marker too early. A crash before the node-failure checkpoint can make an already-run side effect look pre-admission and repeat without confirmation. File:
workflows/interpreter.ts. - High - compact recovery events expose a confirmable revision before details hydrate. The UI can confirm the new revision while showing stale recovery details. Files:
ui/src/stores/workflows.ts,WorkflowRunList.tsx. - High - node references can leak across foreach/repeat siblings when the local producer is skipped or absent. File:
workflows/interpreter.ts. - High - managed-worktree lifecycle races remain. Pending guarded deletion can allow a duplicate live workspace, incomplete stopped cleanup can permit deletion, and stale DELETE state can target a recreated worktree.
- High - cross-host promotion is incomplete in both directions. Tauri can promote without applying the authoritative renderer snapshot and lacks a normal polling trigger; Electron cannot late-promote after Tauri exits.
- High - SSE backpressure drops events before reconnect because no replay cursor exists. File:
server/routes/events.ts.
Additional medium findings cover coalesced workflow refresh trailing state, deferred cleanup exhaustion, legacy shutdown approval, worktree quarantine, catalog history I/O, and UI remount/navigation behavior.
Validation at this head remains green locally: server 369 passed / 4 skipped, Tauri 91/91, focused UI 39/39, typechecks and plugin build. CI is running.
The critical/high items are the next correction pass before another Gatekeeper review.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30937160275 Artifacts expire in 7 days.
|
Reject unsafe managed-worktree slugs, quote Windows wrapper arguments, and restore normal TLS certificate verification for plugin capability requests. Serialize workflow admission and definition mutations across server processes, preserve retry admission checkpoints, isolate dynamic node references by iteration, and hydrate revision-bound workflow details before enabling confirmations. Revalidate worktrees under deletion leases, retain incomplete cleanup ownership, add bounded SSE replay, and let Tauri and Electron late-promote only after authoritative state reconciliation. Validated with 380 passing server tests plus 4 platform skips, 93 Tauri tests, 119 Electron native tests, server/UI/Electron typechecks, and the plugin build.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/30941160924 Artifacts expire in 7 days.
|
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head bfe3ad67
Merge remains blocked.
Fixed in this pass: plugin TLS verification, Windows wrapper injection, single-critical-section definition updates, single-host retry checkpoints, stale worktree DELETE validation, and basic SSE replay storage.
Remaining critical/high blockers:
- Critical - a second live host can recover a workflow still executing on the first host. Run checkpoints lack a durable executor lease/fencing token, so startup recovery and retry execution can overlap across hosts.
- High - stale filesystem-lock reclamation can delete a newly acquired live lock. Competing stale removers do not CAS the observed owner.
- High - cross-host pause/cancel can report success while the executing host continues. The executor does not observe a durable transition/fencing generation.
- High - workflow/worktree ownership checks can be stale across hosts. A manager initialized before another host starts a run can permit deletion beneath that run.
- High - workspace path leases remain process-local. Another server can remove a managed worktree used by a live ordinary workspace.
- High - browser and Tauri reconnect paths do not transmit the SSE cursor. The server replay buffer exists, but clients still reconnect without
Last-Event-ID. - High - Tauri promotion reload currently invalidates renderer access, while Electron promotion still misses legacy migration and some recovery-claim transitions.
- High - ordinary workspace reuse can later be deleted by retrying an older cancelled creation owner.
Medium findings include replay-window gap signaling, list hydration fences, outer-loop node references, deferred cleanup exhaustion, definition list read-after-write, and plugin tool cancellation.
Local validation at this head: server 380 passed / 4 skipped, Tauri 93/93, Electron native 119/119, all typechecks and plugin build green. CI is running.
The next pass will add durable executor fencing/CAS-safe locks first, then complete cursor and promotion behavior before another push/review.
Persist executor leases and fencing tokens so workflow admission, recovery, pause, cancellation, and ownership refresh fail closed across server processes. Add CAS-safe filesystem locks and cross-process workspace leases to prevent stale owners from deleting or replacing live work. Carry SSE replay cursors and overflow resync through browser and native transports, and make Electron/Tauri late promotion reload and migrate authoritative client state before writes. Finish workflow hydration, plugin cancellation, navigation, and shutdown cleanup edges. Cover lock reclamation, multi-manager admission, workspace deletion races, SSE replay, desktop promotion, and workflow refresh behavior. Validated server, UI, Electron, plugin typechecks/builds plus full server, Electron, Tauri, and focused UI/plugin tests.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head b431f131
Merge remains blocked. The prior executor-lease, CAS-claim, ownership-refresh, workspace-lease, cursor-propagation, promotion, hydration, and plugin-cancellation findings are materially addressed, but this pass found these remaining high-severity gaps:
- Filesystem lock reclamation does not fence the stale writer or unlocker. A writer suspended beyond the stale timeout can resume after a successor commits and overwrite the same expected revision; release can also remove a successor lock after pathname replacement.
packages/server/src/workflows/filesystem-lock.ts:62-70,99-118,127-141. - A crash after publishing a
.reclaimclaim can permanently wedge the protected lock. Future reclaimers time out without retiring the abandoned claim.packages/server/src/workflows/filesystem-lock.ts:41-58,97-103,127-134. - A fenced workflow executor can begin a prompt or shell after remote cancellation. The local signal is checked before the effect, but durable fence loss is only observed by the later heartbeat.
packages/server/src/workflows/interpreter.ts:339-349,383-391,packages/server/src/workflows/manager.ts:1139-1152. - Workspace lease expiry remains split-brain rather than fenced. A suspended host continues operating after another host retires its lease; the lease also tracks the server PID rather than the detached workspace process, so a killed server can leave a live process that a replacement duplicates.
packages/server/src/workspaces/process-lease.ts:12-20,86-89,119-130,172-176,packages/server/src/workspaces/manager.ts:453-456. - Plugin abort has unsafe indeterminate edges. An aborted start may launch privileged workflow actions without returning a run ID, and a pre-aborted request destroys the request before its error listener is installed.
packages/opencode-plugin/plugin/lib/workflows.ts:241-248,packages/opencode-plugin/plugin/lib/request.ts:136-152. - Replay overflow advances the cursor without authoritative resynchronization. Reset currently refreshes workspace descriptors, not active messages, permissions, questions, and workflow/session state, so interaction requests can remain permanently absent.
packages/ui/src/lib/server-events.ts:72-76. - Browser and native transports keep independent cursors. Transport switching and Tauri renderer reload can duplicate deltas or drop handoff events by restarting from an unrelated or undefined cursor.
packages/ui/src/lib/browser-event-transport.ts:18,packages/ui/src/lib/native/desktop-events.ts:18,194-196,packages/ui/src/lib/server-events.ts:39-50. - Malformed/truncated SSE frames can acknowledge undelivered events, and event IDs have no server epoch. EOF after an
idbut before a complete JSON frame retains the cursor; server restart reuses low numeric IDs and silently skips new-process events.packages/ui/src/lib/browser-event-transport.ts:97-107,140-141,packages/tauri-app/src-tauri/src/desktop_event_transport/stream.rs:189-210,252-262,packages/server/src/events/bus.ts:14,51-55. - Re-entrant event publication can deliver IDs out of order. A synchronous subscriber can publish ID 2 before downstream subscribers receive ID 1, allowing reconnect from 2 to strand ID 1.
packages/server/src/events/bus.ts:25-43. - Replay retention is count-bounded but not byte-bounded. One thousand valid multi-megabyte events can exhaust the Node heap.
packages/server/src/events/bus.ts:13,16,25-27. - Tauri trusts any navigated localhost page for renderer authority. A hostile localhost origin loaded in the main window can become claimable and invoke authoritative client-state commands.
packages/tauri-app/src-tauri/src/main.rs:212-215,613-619,packages/tauri-app/src-tauri/src/client_state/commands.rs:54-55. - Electron writes are not ownership-epoch fenced. A save started before ownership loss can later regain ownership and rename its stale temporary snapshot over newer Tauri state.
packages/electron-app/electron/main/client-state.ts:521. - Desktop ownership transition gaps remain. Electron legacy migration can race a newly starting legacy Tauri host, ownership loss is not reported to renderers, and a surviving host-local Electron secondary never retries its local election.
packages/electron-app/electron/main/client-state.ts:268,272,454-458,packages/tauri-app/src-tauri/src/client_state.rs:286-288,606-615. - Failed compact-boundary hydration can leave confirmations disabled indefinitely. Hydration errors are swallowed while the marker remains and no retry is scheduled.
packages/ui/src/stores/workflows.ts:330-341.
Local validation on this head: server 400 passed / 5 skipped, Electron native 122/122, Tauri 97/97, focused UI/plugin 50/50, all relevant typechecks and plugin build green. CI is running.
The next correction pass should first make lock/lease ownership epoch-fenced, then unify replay cursors and reset hydration, and finally close desktop ownership and plugin-abort races.
Fence filesystem writers, workflow effects, workspace processes, and desktop snapshots against ownership replacement. Keep stale reclaim claims recoverable, revalidate executor ownership before effects, retain live detached workspace identities, and retry local desktop promotion safely. Make SSE cursors epoch-scoped and shared across browser/native transports, preserve publication order under re-entrancy, bound replay bytes, and avoid acknowledging incomplete frames. Resynchronize on replay resets and preserve the native cursor across renderer reloads. Handle plugin aborts without orphaning accepted runs or emitting unhandled request errors, retry compact workflow hydration, and bind Tauri renderer authority to trusted document origins. Add race coverage and widen only loaded-CI workflow polling budgets. Validated server 411 passed with 5 skipped, Electron 126 passed, Tauri 99 passed, focused UI/plugin 43 passed, all relevant typechecks, plugin build, and diff checks.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head 61da7b2f
Merge remains blocked. No critical findings remain, and the prior epoch/replay, stale-owner, plugin pre-abort, Electron local-promotion, and renderer-origin issues are materially improved. This fresh pass found these remaining high crash/race windows:
- Filesystem lock acquisition and commit are not generation-atomic. A contender can reclaim after
mkdirbut before the original owner writesowner.json, allowing the original owner to overwrite the successor owner and enter. Durable writers also check ownership before a later rename, leaving a reclaim window between check and commit.packages/server/src/workflows/filesystem-lock.ts:49-55,packages/server/src/workflows/definition-store.ts:265-267,packages/server/src/workflows/manager.ts:2160-2162. - Executor fencing is checked before, not held through, the external effect. A host suspended after revalidation can resume after lease expiry/recovery and submit its prompt or shell.
packages/server/src/workflows/interpreter.ts:340-345,386-391. - Read-only peers do not trigger expired-run recovery.
list()/get()use the one-time initializer and can retain stale running state until an unrelated mutation enters admission.packages/server/src/workflows/manager.ts:458-488,1007-1013. - A refreshed workspace heartbeat can be retired from a stale observation. Retirement compares serialized owner data but not heartbeat generation/mtime, so a successful refresh between observation and rename does not block takeover.
packages/server/src/workspaces/process-lease.ts:176-189,231-245. - Detached workspace identity has an uncovered publication boundary. The process detaches before its identity is persisted into the lease; a server crash in that interval leaves a live child that a replacement can duplicate. Foreign-host retirement also ignores persisted child identity.
packages/server/src/workspaces/runtime.ts:239-246,packages/server/src/workspaces/manager.ts:596-607,packages/server/src/workspaces/process-lease.ts:176-189. - Replay overflow is acknowledged without authoritative cache resynchronization. No production reset consumer invalidates storage/session caches, so stale state can overwrite changes that are now outside replay.
packages/ui/src/lib/server-events.ts:73-79,packages/ui/src/lib/storage.ts:43-57,112-116. - Browser cursor state is lost across renderer reload. The shared cursor is module-memory only, so browser/Electron/fallback transports reconnect fresh and can miss state changes between REST hydration and SSE bootstrap.
packages/ui/src/lib/event-transport-cursor.ts:1,packages/ui/src/lib/browser-event-transport.ts:37-38. - Tauri host-local secondaries still cannot promote. The local process election is not retried and the watcher drops candidacy, leaving persistence disabled after the local primary exits.
packages/tauri-app/src-tauri/src/client_state/process.rs:228,258,packages/tauri-app/src-tauri/src/client_state.rs:605. - Legacy migration cleanup remains racy in both desktop hosts. Electron and Tauri can delete a newer legacy snapshot written after their last ownership check.
packages/electron-app/electron/main/client-state.ts:498-499,packages/tauri-app/src-tauri/src/client_state.rs:812-820. - An accepted plugin start can still orphan on a truncated response. Compensation starts only after complete JSON decoding; if the server accepts then truncates/resets the response, no run ID is available and no cancellation is sent.
packages/opencode-plugin/plugin/lib/workflows.ts:248. - Compact hydration can remain blocked after stale successful retries. Two successful responses below the announced revision leave the hydration marker set with no later retry.
packages/ui/src/stores/workflows.ts:347-353.
Local validation on this head: server 411 passed / 5 skipped, Electron 126/126, Tauri 99/99, focused UI/plugin 43/43, all relevant typechecks and plugin build green. CI is running.
The next pass should make lock commit ownership atomic, close process/heartbeat publication windows, persist browser cursor/reset state, and finish desktop migration/local-promotion plus plugin response recovery.
Publish fully initialized filesystem locks atomically and use same-machine process identity to prevent reclaiming suspended live writers or executors. Recover dead peer runs through coalesced reads, CAS workspace heartbeat retirement, and persist pre-spawn cleanup anchors so detached children cannot be duplicated. Give plugin starts caller-known idempotent run IDs so aborted or truncated responses can always compensate. Persist server-scoped replay cursors across renderer reloads and require authoritative storage, session, message, permission, and question resynchronization before acknowledging replay overflow. Retry Tauri host-local election, guard Electron and Tauri legacy cleanup by authority and exact file identity, and bound stale workflow hydration retries without leaving confirmations permanently disabled. Validated server 419 passed with 5 skipped, Electron 128 passed, Tauri 101 passed, focused UI/plugin 53 passed, all relevant typechecks, plugin build, Cargo formatting, and diff checks.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head 7ca814e1
Merge remains blocked. Distributed/shared-storage findings outside the supported same-machine CodeNomad-host scope are excluded. No critical findings remain, but these same-machine high issues remain:
- Workspace process probe failure bypasses a live launch anchor. When persisted process identities exist but discovery fails, the lease is declared stale before checking the live cleanup token, allowing a duplicate workspace process.
packages/server/src/workspaces/process-lease.ts:208-215,362-366. - Native replay reset can drop live events emitted during hydration. Events after reset cursor N are discarded while hydration runs; later cursor advancement can acknowledge past them.
packages/ui/src/lib/native/desktop-events.ts:96-107,157-180. - Replay reset can acknowledge incomplete session/interaction state. Session-status failures are swallowed as idle, and failed legacy permission/question lists become empty snapshots despite
propagateErrors.packages/ui/src/stores/session-api.ts:412-451,packages/ui/src/stores/instances.ts:489-502,547-559,693-727. - Replay reset omits authoritative YOLO and workflow state. YOLO is cleared but not refetched, while workflow loading remains fire-and-forget and outside the reset acknowledgement barrier.
packages/ui/src/stores/instances.ts:813-823,1575-1594,packages/ui/src/stores/workflows.ts:124-149,394-397,packages/ui/src/lib/server-events.ts:73-88. - Tauri promotion can deadlock with two surviving local secondaries. One survivor holds the local lock while stale cross-host recovery requires a claim from the other, which cannot claim without the local lock.
packages/tauri-app/src-tauri/src/client_state/process.rs:234-248,client_state/cross_host.rs:617-650. - Unexpected Tauri CLI exit leaves its exact-origin authority trusted. The exit monitor clears state but retains the trusted URL, so another process reusing the port can claim client-state authority.
packages/tauri-app/src-tauri/src/cli_manager.rs:1168-1183,client_state/commands.rs:28-33,80-89. - Disabled legacy snapshots can be resurrected after migration. Legacy
restoreEnabled:falsehides but retains snapshot/window payloads; later enabling restore exposes old private state. Electronclient-state.ts:150-158,421-429,510-518; Tauriclient_state.rs:445-453,721-737,848-870. - Legacy cleanup still has a final validation-to-unlink race. A replacement written after content/authority validation but before unlink is deleted. Electron
client-state.ts:531-536; Tauriclient_state.rs:908-921. - A torn pre-spawn launch token can wedge recovery forever. Malformed token data is treated as unknown/potentially live rather than a dead incomplete publication after the manager is gone.
packages/server/src/workspaces/process-lease.ts:149-160,201-215,330-347. - Malformed successful plugin start responses bypass compensation. A 2xx
{}response reaches summary parsing and throws without cancelling the caller-known run ID.packages/opencode-plugin/plugin/lib/workflows.ts:263-285. - Cancellation can hold global workflow admission indefinitely.
cancelOwned()awaits an unbounded active completion while inside the admission lock, blocking all starts/cancels/definition mutations if a prompt ignores abort.packages/server/src/workflows/manager.ts:747-760,792-797. - Stale compact hydration now fails open for recovery. After bounded stale responses, hydration clears while the compact newer recovery revision retains old execution details, enabling confirmation of unseen ambiguous state.
packages/ui/src/stores/workflows.ts:347-356,375-382,WorkflowRunList.tsx:73-82,160.
Local validation on this head: server 419 passed / 5 skipped, Electron 128/128, Tauri 101/101, focused UI/plugin 53/53, all relevant typechecks and plugin build green. CI is running.
Fail workspace takeover closed when detached-process discovery is inconclusive, consult live launch anchors before reclaim, and recover stale torn anchors safely. Move unbounded workflow completion waits outside global admission so a provider ignoring abort cannot block unrelated mutations. Buffer native events across replay reset, propagate authoritative session and interaction failures, and include YOLO and workflow hydration before acknowledging overflow. Keep compact workflow confirmations closed until current details hydrate while retrying with bounded backoff. Resolve three-process Tauri promotion, revoke trusted CLI origin on unexpected exit, destructively strip disabled legacy payloads, and quarantine legacy files before validated cleanup in both desktop hosts. Compensate malformed successful plugin starts by caller-known run ID. Validated server 423 passed with 5 skipped, Electron 129 passed, Tauri 105 passed, focused UI/plugin 41 passed, all typechecks, plugin build, Cargo checks, and diff checks.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head 51206827
Merge remains blocked. No critical findings remain. The previous process-probe fallback, reset hydration barriers, migration privacy/cleanup, malformed start compensation, and basic Tauri cohort fixes are improved, but these high same-machine races remain:
- Workflow control/cancel lock inversion can deadlock global admission. Answer/pause/resume hold admission then wait for run transition; cancel holds run transition then waits for admission.
packages/server/src/workflows/manager.ts:623,666,696,760-776,795-801,1105-1113. - Bounded cancellation can finalize before unresolved session creation settles. With no session ID/admitted attempt yet, cancellation releases ownership; a late session can become orphaned if abort is unconfirmed.
manager.ts:854-901,1528-1544,1786-1791,interpreter.ts:102-119,338-340,457-482. - Windows launch-token probing can wedge dead workspace leases. Token-first recovery uses the POSIX
shprobe onwin32; probe failure is treated as potentially live and persisted process identities are never checked.process-lease.ts:212-220,408-411,process-identity.ts:541-557. - Cancelling a recovered confirmed-retry checkpoint can permanently reserve its workspace. The safe retry checkpoint is still classified as ambiguous, repeatedly returning to
recovery_required.run-state.ts:39-41,manager.ts:816-844,872-889,1786-1791. - A newer native replay reset retains pre-reset buffered events. R1 events are replayed after R2 authoritative hydration and can duplicate deltas or overwrite unversioned state; the older cursor can replace R2.
packages/ui/src/lib/native/desktop-events.ts:178-215,232-236. - Three-process local promotion can still deadlock in both desktop hosts. Electron local non-primary survivors do not publish the claim required by the local-lock holder; Tauri can split local-lock and recovery-winner roles between different survivors. Electron
client-state.ts:318-330,client-state-cross-host.ts:249-255; Tauriclient_state/process.rs:238-247,client_state/cross_host.rs:699-703. - Tauri CLI authority revocation is not generation-fenced. Buffered output can republish a dead endpoint after unexpected exit, and explicit restart advances generation before the old monitor revokes the outgoing origin.
cli_manager.rs:784-789,1127-1207,1230-1320,main.rs:139-146.
Local validation on this head: server 423 passed / 5 skipped, Electron 129/129, Tauri 105/105, focused UI/plugin 41/41, all relevant typechecks and plugin build green.
Use a single workflow lock order, settle session creation before terminal cancellation, and allow confirmed retry checkpoints to cancel without recurring recovery. Probe Windows launch tokens with native process discovery and safely fall through to immutable identities. Version native replay resets so superseding hydration drops older buffered events and cannot regress cursors. Let same-cohort desktop secondaries publish recovery-only claims while preserving a single authority winner across three-process Electron and Tauri elections. Generation-fence Tauri endpoint publication and revoke outgoing renderer authority on unexpected exit, stop, restart, and restart failure so buffered output cannot restore a dead origin. Validated server 428 passed with 5 skipped, Electron 131 passed, Tauri 108 passed, replay UI 7 passed, all typechecks, Cargo checks, and diff checks.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head 63e37b51
Gate: reject. No critical findings; four high same-machine blockers remain:
- Confirmed resume during cancellation can retain global admission indefinitely. Cancellation exposes transient
recovery_required; resume accepts it and waits on unresolved completion while holding admission, even after cancellation later persistscancelled.packages/server/src/workflows/manager.ts:697-704,795-841,1104-1112. - Windows launch-token probing does not match production environment-only tokens. The probe searches command lines, while the token is inherited only through environment; fallback can duplicate surviving descendants or permanently block wrapper recovery.
packages/server/src/workspaces/process-identity.ts:378-400,runtime.ts:179-181,273-299,process-lease.ts:213-230. - Electron recovery can deterministically select an ineligible local secondary. A non-local-primary publishes a recovery claim and may sort before the local-lock holder, preventing either from retiring the stale owner.
packages/electron-app/electron/main/client-state.ts:318-332,client-state-cross-host.ts:261-314,459-468. - Tauri launcher exit can revoke readiness while descendants survive. The endpoint generation advances when the root exits before the process tree is empty; descendant output and the watchdog retain the revoked generation, leaving startup stuck.
packages/tauri-app/src-tauri/src/cli_manager.rs:1122-1129,1164-1199,1256-1263.
Validated head: server 428 passed / 5 skipped, Electron 131/131, Tauri 108/108, replay UI 7/7, typechecks and diff checks green.
Keep workflow control settlement outside admission while fencing concurrent cancellation, and track Windows launch trees with immutable identities so wrapper descendants prevent duplicate recovery without relying on unobservable environment tokens. Make Electron recovery claims target the verified host-local lock holder so ineligible secondaries cannot win. Preserve Tauri readiness generation while owned descendants survive, while complete exit and explicit restart still revoke endpoint authority immediately. Replace the timing-dependent stale-lock stress test with deterministic reclaim-then-contention coverage. Validated server 434 passed with 5 skipped, Electron 131 passed, Tauri 111 passed, all relevant typechecks, Cargo checks, and diff checks.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head b36da30e
Gate: reject. The workflow resume/cancellation admission race is fixed. Three high boundaries remain:
- Incomplete Windows launch snapshots can permit duplicate workspace ownership. A failed readiness-time CIM snapshot retains an earlier wrapper-only identity set; recovery treats it as complete and can miss a surviving child.
packages/server/src/workspaces/runtime.ts:421-431,manager.ts:605-607,process-lease.ts:216-264. - Tauri can retain a dead ready endpoint while workspace descendants keep the job nonempty. CLI exit leaves stale URL/cookies/renderer authority trusted when unrelated descendants survive; containment-probe failure also does not revoke.
packages/tauri-app/src-tauri/src/cli_manager.rs:874-883,1191-1241,client_state.rs:642-655. - Electron cross-host ownership lacks machine identity. On unsupported separate-machine shared storage, local PID probing can classify a live remote owner stale; the new eligible-claim handoff then retires it instead of failing closed.
packages/electron-app/electron/main/client-state-cross-host.ts:87-104,170-181,309-320,459-476.
Validated head: server 434 passed / 5 skipped, Electron 131/131, Tauri 111/111, typechecks and diff checks green.
Persist complete Windows launch-tree identity generations atomically and fail readiness closed when the final native snapshot is unavailable, preventing wrapper-only recovery from duplicating surviving children. Revoke a dead ready Tauri endpoint immediately even when unrelated workspace descendants remain, while preserving the starting-only launcher handoff. Add shared Electron/Tauri machine identity so separate-machine records fail closed and same-machine cohorts still recover. Validated server 437 passed with 5 skipped, Electron 133 passed, Tauri 116 passed, focused workspace 30 passed, typechecks, Cargo checks, and diff checks.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper re-review - head e347835d
Gate: reject. Three high blockers remain:
- Pre-upgrade workspace leases are misread. Legacy
{token}launch files and per-process identity files are ignored by the new generation reader, risking duplicate POSIX launches or permanent Windows blockage after upgrade.packages/server/src/workspaces/process-lease.ts:231,412-428. - Transient readiness CIM failure can permanently wedge a direct Windows workspace. Cleanup keeps returning unknown even after the initially captured process is proven gone, preventing stop/delete/restart.
packages/server/src/workspaces/runtime.ts:424-430,551-555,738-740,manager.ts:635-649. - Tauri CLI authority revocation does not stop native event transport. The transport retains the old URL/session cookie and reconnects to a reused port after ready CLI death.
packages/tauri-app/src-tauri/src/cli_manager.rs:1200-1238,desktop_event_transport.rs:267-287,transport.rs:54-139.
Other prior highs are cleared. Validated head: server 437 passed / 5 skipped, Electron 133/133, Tauri 116/116, typechecks and diff checks green.
Read complete legacy workspace launch and process identity records so upgrades preserve live-process fencing and reclaim dead trees without treating malformed persisted state as safe. Allow direct Windows cleanup to recover after transient readiness capture failure while wrapper trees remain fail closed. Revoke and erase native Tauri event transport authority alongside CLI URL and cookies so a reused port cannot receive stale credentials or events. Validated server 444 passed with 5 skipped, focused workspace 37 passed, Tauri 119 passed, typechecks, Cargo checks, and diff checks.
After transient Windows identity-capture failure, clear the safety fence only after positively recapturing the original live immutable leader and its descendants. If the leader exits before complete recapture, retain unknown ownership so an unobserved child cannot be duplicated. Validated server 445 passed with 5 skipped, focused workspace runtime 14 passed, typecheck and diff checks.
pascalandr
left a comment
pascalandr
left a comment
There was a problem hiding this comment.
Gatekeeper final re-review - head 52800aad
Gate: ACCEPT
No critical or high findings remain within the supported same-machine multi-process scope.
Verified closure of the prior blockers:
- durable workflow executor fencing, recovery, cancellation, retry checkpoints, and lock ordering;
- CAS-safe filesystem/workspace leases, complete Windows launch-tree identity generations, legacy lease upgrades, and fail-closed orphan handling;
- epoch-scoped SSE replay, cursor persistence, overflow hydration barriers, and superseded-reset buffering;
- plugin trust, TLS, caller-known idempotent run IDs, abort compensation, and malformed-response handling;
- Electron/Tauri ownership epochs, three-process promotion, stable machine identity, legacy migration cleanup, exact-origin authority, and endpoint/native-transport revocation.
Final local validation:
- server:
445 passed / 5 skipped; - Electron native:
133/133; - Tauri:
119/119; - focused UI/plugin and replay suites green;
- server/UI/Electron typechecks, plugin build, Cargo checks, and
git diff --checkgreen.
Residual medium risk: on Windows, a direct launch whose leader exits before complete identity recapture remains fail-closed and may require operator cleanup rather than risking a duplicate process.
Populate the conflicting retirement tombstone so POSIX rename cannot replace it as an empty directory. This preserves the intended retry assertion on Linux while retaining the same Windows behavior. Validated the process lease suite with 24 passing tests; broader server runs also passed this case and exposed separate pre-existing timing flakes.
Give asynchronous workspace removal a bounded 500 ms polling window after lease loss instead of exhausting immediate callbacks before filesystem cleanup can settle under CI load. The assertions still require the port, runtime process, and workspace record to be removed. Validated the focused lease-loss test and all 24 process-lease tests.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/31001762992 Artifacts expire in 7 days.
|
Key app-tab content by stable tab IDs so instance descriptor refreshes no longer dispose and recreate the entire conversation shell. Replay recovery now invalidates stale requests while retaining visible sessions, messages, and timeline segments until authoritative data arrives. Authoritative empty message responses still clear stale messages once the response commits, preserving replay correctness without a blank intermediate state. Validated UI typechecking, a production Vite build, 51 transport and hydration tests, and 26 focused workflow/message tests.
Trigger the owner heartbeat directly after publishing the successor instead of relying on an unref timer to keep Node alive. This prevents the test runner from cancelling the remaining process-lease tests when the event loop drains first. Validated all 24 process-lease tests.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/31085953524 Artifacts expire in 7 days.
|
Use the stable browser EventSource transport unless the user explicitly enables the Rust-native Tauri transport. This prevents native reconnect loops from globally cycling connection state, workspace reconciliation, sessions, and Status UI when no workflow module is active. Explicit stored opt-in remains supported. Validated UI typechecking, production build, and 20 browser/native transport and connection-status tests.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/31096536912 Artifacts expire in 7 days.
|
Queue event frames while the response is backpressured and resume them in order after drain instead of closing every healthy stream. The queue is bounded to 8 MiB and still destroys genuinely stalled or overflowing connections so replay recovery remains available. This stops global connecting/connected loops and repeated session reconciliation regardless of whether Tauri native transport or browser streaming is selected. Validated server typechecking, all 445 server tests, six SSE route tests, and 13 browser transport/status tests.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/31099434498 Artifacts expire in 7 days.
|
1 participant
Summary
Implements Claude Code-style dynamic workflows through a safe declarative YAML/JSON runtime backed by OpenCode agents. It provides comparable dynamic orchestration without evaluating generated JavaScript or TypeScript.
User experience
Users can build and operate workflows from the Workflows tab in the customizable right panel:
Example:
Compared with Claude Code dynamic workflows
evalor generated-code execution; commands cross an explicit shell-node boundaryThe user outcome is intentionally similar: workflows are created dynamically and can branch, loop, fan out, invoke agents, pause for people, and compose other workflows. The execution model is deliberately different: CodeNomad favors predictable validation, visual operation, persistence, and recovery over arbitrary generated-code flexibility.
Tradeoffs
toolsis omitted; an explicittoolslist remains an installed-tool allowlist. Shell nodes remain the deterministic command primitive.Safety
Persistent named sessions and probable follow-ups
Agent nodes may declare an optional
sessionKey. The first use creates an OpenCode session; later nodes and repeat iterations with the same key continue that conversation. Bindings are persisted for restart recovery, and prompts sharing a key are serialized so parallel branches cannot mutate one conversation concurrently. Omitting the key preserves one session per action.Probable follow-ups remain intentionally separate: per-session worktrees for independently isolated workers, provider-specific reasoning controls such as a max-thinking variant, and richer PR authorization primitives if generic human gates prove insufficient.
Validation
Closes #435