feat(observability): add observability crate and unify process init

[Bnjoroge1]

New preloop-observability crate: env-driven logging (RUST_LOG defaults to info; PRELOOP_LOG_FORMAT pretty/json/auto), task heartbeats with criticality, limit registry, VM telemetry registry, and the Observability handle that later PRs clone into the server and pool. All three binaries install the unified subscriber so preloop serve and the CLI stop disagreeing about log level. Absent OTEL_* env means disabled — no default localhost:4318 socket.

Part of a stacked series (merge bottom-up):

1. log hardening
2. this PR (foundation crate)
3. health/readiness/status endpoints
4. HTTP and store metrics
5. OTLP export
6. review-fix sweep

---

Summary by cubic

Introduces preloop-observability and unifies process-level logging init across preloop, preloop-runner-server, and preloop-runner. The server’s default log level changes from “silent unless RUST_LOG is set” to info, matching the CLI; PRELOOP_LOG_FORMAT now controls pretty/json/auto output, and OTLP export stays disabled unless an OTEL endpoint is provided (no implicit localhost:4318).

• Adds preloop-observability:
  • ObservabilityConfig::from_env (RUST_LOG default info; PRELOOP_LOG_FORMAT auto/pretty/json; OTEL vars parsed; Debug redacts secrets).
  • Observability handle and ObservabilityRuntime with a bounded 2s shutdown window.
  • Task heartbeat registry (critical vs non-critical; Drop deregisters; staleness detection) and limit registry (register/record_drop/record_reject).
• Replaces ad-hoc tracing_subscriber setup in all binaries with one subscriber install via ObservabilityRuntime::install_fmt_subscriber; the handle is kept for later wiring into server/pool.
• Operator notes:
  • Server log volume increases by default; set RUST_LOG to tune if needed.
  • Use PRELOOP_LOG_FORMAT=pretty|json|auto (auto = pretty on TTY, JSON when piped).
  • OTLP export remains off unless an OTEL_EXPORTER_OTLP_* endpoint is set; setting the endpoint to “none” explicitly disables export.

^{Written for commit 454c45e. Summary will update on new commits.}

Note

Add preloop-observability crate and unify logging init across CLI, runner, and runner-server

• Introduces preloop-observability with ObservabilityConfig, Observability, and ObservabilityRuntime types. Config is built from env: RUST_LOG defaults to info, PRELOOP_LOG_FORMAT selects pretty/json/auto (auto maps to pretty on TTY, json otherwise).
• Adds TaskHeartbeat registry for tracking long-lived background tasks with staleness detection on critical tasks, and LimitRegistry for recording dropped/rejected event counts per named limit.
• ObservabilityRuntime::install_fmt_subscriber installs a global tracing fmt subscriber honoring the resolved log format and filter. shutdown() is bounded to 2 seconds (currently a no-op placeholder).
• preloop-cli, preloop-runner, and preloop-runner-server replace direct tracing_subscriber init with the new crate. All three now default to info when RUST_LOG is unset.
• Risk: ObservabilityConfig custom Debug impl redacts OTLP headers and endpoint secrets; if redaction logic is incomplete, sensitive values could leak in logs. The global subscriber is installed once per process, so concurrent or repeated install_fmt_subscriber calls will panic.

📊 Macroscope summarized 454c45e. 9 files reviewed, 7 issues evaluated, 7 issues filtered, 0 comments posted

🗂️ Filtered Issues

crates/preloop-observability/src/lib.rs — 0 comments posted, 7 evaluated, 7 filtered

• line 102: from_env applies the empty/none filter only after the .or_else(...) chain. Therefore, if OTEL_EXPORTER_OTLP_ENDPOINT exists but is empty/whitespace, it prevents a valid signal-specific endpoint such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT from being considered, and otlp_enabled becomes false even though an endpoint is present. Filter each candidate before selecting the fallback (while preserving the intentional global none override if desired). [ Out of scope (post-validation triage) ]
• line 109: The headers fallback has the same post-selection filtering problem: an empty OTEL_EXPORTER_OTLP_HEADERS value wins the .or_else(...) chain and is then removed, so a non-empty signal-specific headers variable is ignored. Consequently has_otel_headers() incorrectly reports false despite usable OTLP headers being configured. [ Out of scope (post-validation triage) ]
• line 196: A critical task's heartbeat entry is owned only through HeartbeatHandle, whose unconditional Drop removes the map entry. When a spawned task panics, Rust unwinds and drops the handle too, so the dead critical task disappears from TaskHeartbeat; any_critical_stale then cannot detect it and /readyz can remain healthy despite loss of a critical background task. The registry must distinguish clean completion from panic (or retain a failed/stale entry). [ Out of scope ]
• line 196: TaskHeartbeat stores one HeartbeatEntry per task name, but register returns independent guards and silently replaces an existing same-name entry. If two overlapping tasks register the same static name, dropping the older guard removes the newer task's entry; subsequent beats become no-ops and readiness can incorrectly report healthy because the critical task is no longer registered. The map needs per-registration identity/ref-counting, or duplicate registration must be rejected. [ Out of scope (post-validation triage) ]
• line 210: register silently replaces an existing entry with the same name, but both returned handles remain able to deregister by name. If a replacement/restarted task is registered before the old handle drops, dropping the old handle removes the replacement's entry; subsequent beats become no-ops and readiness/status stop monitoring the still-running task. The handle needs a per-registration identity (or duplicate registration must be rejected) so Drop only removes its own entry. [ Out of scope (post-validation triage) ]
• line 289: HeartbeatHandle::drop always deregisters the task, including when the task is being unwound after a panic (Rust runs destructors during panic unwinding). A panicked critical background task therefore disappears from snapshot and any_critical_stale, so /readyz can continue reporting healthy after the critical task has died. Check std::thread::panicking() and preserve/mark the entry on panic rather than removing it. [ Out of scope (post-validation triage) ]
• line 556: absent_endpoint_means_disabled_not_localhost mutates process-global OTEL environment variables without synchronizing with none_disables_signal and debug_redacts_headers_and_endpoint_userinfo. Rust tests run concurrently by default, so either test can set an endpoint between these removals and ObservabilityConfig::from_env(), making this test flaky (and its removals can likewise break the other tests). Serialize all environment-mutating tests and restore prior values. [ Out of scope (post-validation triage) ]

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ede9f9bf-013d-426c-bbe9-af02bae16fc8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}