Skip to content

fix(daemon): don't crash-loop on headless Linux when a routine is overdue#1409

Merged
muqsitnawaz merged 1 commit into
mainfrom
fix-daemon-notify-crash
Jul 23, 2026
Merged

fix(daemon): don't crash-loop on headless Linux when a routine is overdue#1409
muqsitnawaz merged 1 commit into
mainfrom
fix-daemon-notify-crash

Conversation

@muqsitnawaz

Copy link
Copy Markdown
Contributor

What

The agents-daemon systemd user service crash-loops on headless Linux whenever a routine is overdue, which also takes down the browser IPC socket — agents browser start then fails with Error: Timeout waiting for browser daemon socket.

Root cause

On startup the daemon detects overdue routines and fires a best-effort desktop notification (notifyOverdue in apps/cli/src/lib/overdue.ts) via notify-send on Linux / osascript on macOS. A headless box has no notify-send (no libnotify-bin), so the spawn fails with ENOENT.

spawn() reports a missing binary as an asynchronous 'error' event, not the synchronous throw the surrounding try/catch was written to catch. With no 'error' listener, Node re-throws it as an uncaught exception and the whole daemon exits (status=1). systemd Restart= brings it back every ~10s, it re-detects the same overdue routine, and crashes again — a permanent loop. The daemon never lives long enough to bind browser.sock.

Observed on yosemite-m0 (Ubuntu 24.04, headless):

Error: spawn notify-send ENOENT
  syscall: 'spawn notify-send'
  spawnargs: ['Routine overdue: check-updates', 'Missed 7/20/2026, 9:00:00 AM...']
agents-daemon.service: Main process exited, code=exited, status=1/FAILURE
agents-daemon.service: Scheduled restart job, restart counter is at 13.

Fix

Attach an 'error' listener to both notifier spawns so the failure is swallowed — which is exactly what the function's Best-effort … failures … are swallowed doc comment already promised. The macOS branch had the same latent flaw (never hit because osascript always exists there).

Test

apps/cli/src/lib/overdue.test.ts reproduces the crash by emptying PATH (deterministic ENOENT on every platform) and asserting the process survives. Verified it fails without the fix — the async spawn osascript ENOENT surfaces as an Uncaught Exception and the run exits non-zero — and passes with it.

Notes

  • Impacts every headless Linux box running the daemon (the whole yosemite-* worker fleet) once any routine goes overdue.
  • A headless box driving a browser separately needs Chromium launched with --no-sandbox (Ubuntu 24.04 AppArmor blocks unprivileged user namespaces) — that's environment config, out of scope for this PR.

…rdue

The overdue-routine desktop notification spawns notify-send (Linux) /
osascript (macOS) best-effort. A missing notifier binary — the default on a
headless box without libnotify-bin — surfaces as an async spawn 'error'
event, not the synchronous throw the surrounding try/catch expected, so Node
re-threw it as an uncaught exception and killed the daemon. systemd then
restart-looped it every ~10s, which also tore down the browser IPC socket
(agents browser start -> 'Timeout waiting for browser daemon socket').

Attach an 'error' listener to both notifier spawns so the failure is
swallowed as the best-effort contract already documented. Add overdue.test.ts
reproducing the crash with an emptied PATH (deterministic ENOENT on every
platform).
@muqsitnawaz
muqsitnawaz merged commit c624ef6 into main Jul 23, 2026
9 checks passed
@muqsitnawaz
muqsitnawaz deleted the fix-daemon-notify-crash branch July 23, 2026 00:09
@prix-cloud

prix-cloud Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Code Reviewer

Verdict: Ready to merge

Build: Pass — bun install && bun run build (tsc) clean, no errors.
Tests: Pass — bun test src/lib/overdue.test.ts → 2 pass / 0 fail. Scoped run over src/lib/{daemon,routines}.test.ts + src/lib/__tests__/ surfaced pre-existing failures unrelated to this diff (shim-path resolution and a detached-daemon socket-liveness test that don't survive this sandboxed container — no systemd, restricted process daemonization). Confirmed these aren't regressions from this PR: real GitHub Actions CI on this PR is all green — test, test-shard (1/2/3), typecheck, gitleaks, compiled-smoke, bench, test-windows.

Read CLAUDE.md (repo root) and apps/cli/AGENTS.md before reviewing — build/test commands (bun install && bun run build && bun test) came from the latter.

Changes that work well

  • Correct root-cause fix. spawn() reports a missing binary via an async 'error' event, not a synchronous throw, so the surrounding try/catch in notifyOverdue (apps/cli/src/lib/overdue.ts:108-137) never caught it — Node re-threw as an uncaught exception and killed the daemon. Attaching child.on('error', () => {}) on both the darwin (line 121) and linux (line 132) branches closes that gap and matches the function's existing doc comment ("Best-effort ... failures ... are swallowed").
  • The macOS branch got the same listener even though the bug only reproduces on headless Linux (no osascript is ever missing on macOS) — good defensive symmetry rather than a Linux-only patch that leaves a landmine.
  • overdue.test.ts reproduces the crash deterministically by emptying PATH (guarantees ENOENT on whichever binary the platform branch spawns) rather than hardcoding a platform assumption — passes on this Linux sandbox and would pass on macOS CI too.
  • Changelog fragment added under .changelog/next/, consistent with repo convention.

Things to verify manually

  • The regression test asserts survival by sleeping 300ms after calling notifyOverdue and trusting the async 'error' fires within that window. spawn ENOENT typically resolves in low single-digit ms, so this is very unlikely to flake, but worth keeping an eye on if CI runners are ever heavily loaded — a 'error' handler that fires after the test process exits wouldn't be caught by this test.
  • Can't reproduce the original headless-Linux crash-loop live from this sandbox (no systemd, no real agents-daemon.service). The fix logic is sound and the unit test isolates the exact failure mode described in the PR, but confirming the yosemite-* fleet actually stops restart-looping after this ships is worth a quick spot-check post-deploy.

Reviewed by Code Reviewer — actually ran the build and tests on this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant