feat: recover from repeated crashes by going into safe mode

[mateoalfaro]

Summary

Today the session scripts only know one recovery trick: if the compositor dies
early, retry with software rendering once. If the shell crashes, the session
just restarts it forever, and a genuine crash loop (e.g. an assertion failure
in labwc, or a bad shell update) produces an endless restart cycle with no way
for the user to get to a working desktop or fix their settings.

This adds a crash-loop detection layer to the session that escalates to
Safe Mode (see the companion singularity-shell PR): after repeated
failures the session relaunches in a recovery configuration with a persisted
marker, and leaves safe mode automatically once the binaries change.

singularity-labwc-session

• Build identity: computes a build id from the sha256 of the labwc,
  session, and desktop binaries (SINGULARITY_SESSION_BUILD_ID overrides it
  for tests). Markers and requests are scoped to it.
• Escalation ladder: normal attempt → early-failure software-rendering
  retry (existing behavior, preserved) → write a Safe Mode marker → safe-mode
  attempt. Non-zero exits that are neither crashes nor short-lived (130/143,
  ≥30s runtime) don't trigger recovery.
• Safe mode keeps the default (hardware) rendering stack. Forcing the
  software stack (GSK_RENDERER=cairo, WLR_RENDERER=pixman, llvmpipe) made
  the recovery session itself CPU-bound and unusably slow on machines with a
  working GPU, and in the escalation ladder the software path has already
  failed by the time safe mode starts. Operators who suspect the GPU stack
  can opt back in with SINGULARITY_SAFE_MODE_SOFTWARE=1.
• Marker protocol: $XDG_STATE_HOME/singularity/safe-mode (versioned,
  records build id, reason, exit status, elapsed time, timestamp; written
  atomically with 0600). A marker only applies to the same build id — after
  an update the session tries normal mode again, so a fixed build can never be
  trapped in recovery.
• Cross-process request: safe-mode-request lets the shell supervisor
  (below) escalate; if the outer supervisor was killed before consuming it,
  the next display-manager launch still honors it.
• Manual entry: SINGULARITY_SAFE_MODE=1 runs a safe session directly;
  SINGULARITY_FORCE_NORMAL=1 bypasses an existing marker. Compositor binary
  paths are overridable (SINGULARITY_LABWC_BINARY, …) for testing/packaging.

singularity-desktop-session

• Wraps the shell (singularity-desktop) in a supervisor: clean exits restart
  immediately; crashes restart with a 2s backoff. 3 crashes within a 5
  minute window escalates: writes safe-mode-request (with build id) and
  terminates the session so the outer script relaunches in safe mode.
• SINGULARITY_SESSION_SUPERVISOR_ONLY=1 provides a side-effect-free entry
  point for black-box testing (placed before all service/user mutations).

Testing

• tests/session_safe_mode_test.sh (meson test('session-safe-mode')):
  exercises the supervisor crash-window logic and the labwc-session marker /
  build-id / escalation behavior end-to-end via the env-var entry points.
• Manual: kill singularity-desktop three times → session escalates to safe
  mode and the shell's Safe Mode panel appears; clearing it via "Restart
  Normal Session" returns to a normal session.

Related

• Companion singularity-shell change implementing the recovery session
  itself (feature suppression + Settings repair surface).

Tested on NixOS 26.11