From efdf00d708599b1d9c1ffc900d2d17aceaa18616 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 11:55:16 -0400 Subject: [PATCH 01/22] docs(plans): track observability strategy plan, revised at 673bdfa0 Plan 002 was authored at 84d92cfd; the 37 commits since inserted ~12k lines into in-scope files. Re-verify every current-state claim against live code and close the coverage gaps the first draft missed. Drift corrected: - three of four unsafe-logging anchors moved (results_twirp 477->713, distributed_task 324->327, blob_store range); TraceLayer 775->805; CLI subscriber 433-445->743-745; preloop status 2194-2260->2831; AppState::emit 754->819; all broker.rs and runner_lifecycle ranges - SmolVM floor 1.7.7 -> 1.8.1; re-cite machine status/ls --json, data-dir, state_probe, and the vm- cgroup leaf at the new tag - docs/cli.md does not exist; the file is docs/cli_reference.md Coverage added: - bounded-buffer and hard-limit drops: nine caps discard data with no counter, including a 64 MiB per-job live-log tail-drop, MAX_SESSION_AUDIT ring eviction, and QUEUE_MAX_PENDING cancelling a user's job - scheduled workflows, concurrency-group contention, GitHub rate-limit and token budget, and per-component persistent storage growth - TaskHeartbeat registry replacing two ad-hoc heartbeats; fifteen long-lived tasks currently die silently - thirteen HTTP path families, not eight; exclude the live-log WebSocket from the request-duration histogram so it cannot corrupt the SLI Design corrections: - consolidate the four existing ad-hoc RunnerPoolConfig shared handles into one PoolStatus instead of adding a fifth - sample the fleet with one `machine ls --json` call, which returns the same object SmolVmProvider::list already parses, rather than one subprocess per VM - add MachineState::Unreachable so a wedged golden stops silently poisoning every fork taken from it - document that treating an absent OTEL endpoint as disabled deviates from the spec default of http://localhost:4318 Also track plans/README.md, which 001 already referenced. Entire-Checkpoint: 01M0FY4G03TFAM8TXQEVEXV3TZ --- plans/002-observability-strategy.md | 1841 +++++++++++++++++++++++++++ plans/README.md | 51 + 2 files changed, 1892 insertions(+) create mode 100644 plans/002-observability-strategy.md create mode 100644 plans/README.md diff --git a/plans/002-observability-strategy.md b/plans/002-observability-strategy.md new file mode 100644 index 00000000..29057fe3 --- /dev/null +++ b/plans/002-observability-strategy.md @@ -0,0 +1,1841 @@ +# Plan 002: Make Preloop observable without making a backend mandatory + +> **Executor instructions**: Implement this plan as the ordered PR-sized steps below. Run every +> verification command and confirm the expected result before moving to the next step. Preserve the +> official runner wire exactly. If a STOP condition occurs, stop and report it instead of improvising. +> +> **Drift check (run first)**: +> +> ```sh +> git diff --stat 673bdfa0..HEAD -- \ +> Cargo.toml Cargo.lock \ +> crates/preloop-observability crates/preloop-cli crates/preloop-runner-server \ +> crates/preloop-orchestrator crates/preloop-vm crates/preloop-runner \ +> docs contrib/openobserve scripts rules justfile versions.toml CHANGELOG.md +> ``` +> +> If any in-scope file changed since this plan was written, compare the current-state excerpts and +> named symbols below with live code. A semantic mismatch is a STOP condition. + +## Status + +- **Priority**: P1 +- **Effort**: L, split into seven independently reviewable changes +- **Risk**: MED; the HTTP and runner lifecycle paths are protocol-critical, while telemetry must be fail-open +- **Depends on**: none +- **Category**: direction, architecture, operations, security, DX +- **Planned at**: commit `84d92cfd`, 2026-08-17 +- **Revised at**: commit `673bdfa0`, 2026-08-20. Every current-state excerpt, file:line anchor, and + external version claim below was re-verified against live code at that commit. The revision also + closed six coverage gaps the first draft missed: bounded-buffer/limit drops, the scheduled-workflow + subsystem, concurrency-group queueing, GitHub rate-limit budget, persistent-storage growth, and a + general background-task heartbeat registry (the first draft named only two loops out of fifteen). + +## Executive decision + +Build observability in three layers, in this order: + +1. **Zero-dependency operator diagnostics**: truthful liveness/readiness, an authenticated aggregate + status endpoint, `preloop status --json`, structured stderr/journald logs, and authenticated + Prometheus text at `/metrics`, including host-observed resource use for Preloop-owned microVMs. + These work with no sidecar and answer “why is this job not moving?” even when no telemetry backend + exists. +2. **Vendor-neutral telemetry**: OpenTelemetry metrics, logs, and short-lived traces, exported through + bounded OTLP/HTTP batches only when standard `OTEL_*` configuration is present. Export failure must + never reject, delay, cancel, or change a workflow. +3. **OpenObserve as an optional reference backend**: a pinned, private single-node deployment plus + importable dashboards and alerts. Preloop must neither embed OpenObserve nor require it. Any OTLP + backend remains interchangeable. + +OpenObserve is a good fit for the optional third layer: its single-node mode is one binary/container, +uses SQLite plus local disk by default, accepts logs/metrics/traces over OTLP, and includes dashboards +and standard alerts. It is not a sound architectural dependency and its HA mode is not minimal. Keep +the product boundary at OTLP and Prometheus. + +The highest-priority deliverable is not a dashboard. It is `preloop status`: backend telemetry is +least useful during exporter, network, credential, or storage failures, which are exactly when an +operator needs a direct answer. + +## Why this matters + +Today Preloop can accept work while the pool repeatedly fails to provision, can leave an operator +unable to distinguish “no compatible runner” from “runner has stopped polling,” and can report a +healthy process while critical background behavior is degraded. The only built-in aggregate view is +a recent-runs table. That makes incident diagnosis depend on manually correlating unstructured logs, +HTTP endpoints, server state, and VM state. + +This plan makes every important control-plane question answerable: + +- Is the process alive and is its critical event loop making progress? +- How many jobs are ready, dependency-blocked, concurrency-blocked, expanding, or unclaimable? +- Is compatible capacity absent, preparing, provisioning, idle, busy, paused, or stale? +- Are runners polling and renewing leases? +- Are VM CPU, host memory, throttling, sparse-disk allocation, or runtime health limiting capacity? +- Are durable-state writes and GitHub check updates succeeding? +- Is telemetry itself exporting, dropping, or failing? +- Which run/job/session/machine was involved, without putting identifiers into metric labels? + +## Current state + +### Logging is local, duplicated, and not export-ready + +`crates/preloop-cli/src/main.rs:743-745` initializes a plain formatting subscriber: + +```rust +tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")), + ) + .init(); +``` + +The standalone server (`crates/preloop-runner-server/src/main.rs:78-80`, which uses +`EnvFilter::from_default_env()` and therefore has no `info` fallback) and the Rust runner +(`crates/preloop-runner/src/main.rs:17-21`) each initialize their own subscriber. Three binaries, +three slightly different filter defaults, no JSON selection, no OTLP pipeline, no metrics provider, +no exporter-health state, and no coordinated flush. + +`tracing-subscriber` is already a workspace dependency at `0.3` with features +`["env-filter", "json"]`, so `PRELOOP_LOG_FORMAT=json` needs a layer switch, not a new dependency. + +**Security gate**: current INFO/WARN events are unsafe to export unchanged: + +- `crates/preloop-runner-server/src/artifact_twirp.rs:94` — + `info!(token, name = request.name, "artifact v2 create")` logs an artifact upload capability token. +- `crates/preloop-runner-server/src/results_twirp.rs:713` — + `info!(token, "cache v2 create entry")` logs a cache upload capability token. +- `crates/preloop-runner-server/src/blob_store.rs:63, 78, 95, 113, 121, 127, 131` — seven + `warn!`/`info!` sites carry `kind, token` blob capability tokens. +- `crates/preloop-runner-server/src/distributed_task.rs:327` — + `info!(?body, "agent_request_patch received")` logs the complete runner PATCH JSON body. +- `crates/preloop-runner-server/src/recording.rs:1-90` deliberately records every header and both + bodies for conformance capture, including authorization material. + +Do not enable log export until the first four are removed or reduced to safe fields. Flow recording +must remain an explicit local conformance facility, stored mode 0600, and must never pass through the +normal logging/OTLP pipeline. + +The step-1 audit must be a scan, not a fixed list: the four sites above are the ones that exist at +`673bdfa0`, and the churn between `84d92cfd` and `673bdfa0` moved three of them. Re-run the scan +rather than trusting these anchors: + +```sh +grep -rnE '(info|warn|error)!\(' crates/preloop-runner-server/src crates/preloop-orchestrator/src \ + | grep -E '\b(token|authorization|cookie|headers|body|payload|signed_url|secret|password)\b' +``` + +### Health and status do not diagnose the control plane + +`crates/preloop-runner-server/src/runs.rs:4-9` always returns `ok: true`: + +```rust +pub(crate) async fn healthz(State(shared): State>) -> Json { + Json(json!({ + "ok": true, + "protocol_version": PROTOCOL_VERSION, + "shutdown_requested": shared.shutdown.is_cancelled(), + })) +} +``` + +The router exposes only `GET /healthz` for health and installs +`TraceLayer::new_for_http()` at `crates/preloop-runner-server/src/routes.rs:805`. The default trace +span records the request URI; exporting it would leak query strings and create unbounded URI values. +Use Axum's matched route template, never raw URI/path-and-query, in telemetry. There is no +`/readyz`, no `/metrics`, and no Prometheus dependency anywhere in the workspace today; `/healthz` +at `routes.rs:256` is the only health surface. + +`preloop status` at `crates/preloop-cli/src/main.rs:2831` calls only +`GET /api/v1/runs?limit=20` and renders recent runs. The native runner API can already report queued +versus claimable work for a run (`runner_lifecycle.rs:145-164`), but there is no one-call operational +snapshot. `wait_for_engine_socket` (`crates/preloop-cli/src/main.rs:1458-1475`) probes +`http://localhost/healthz` with a 500 ms per-attempt timeout over a 30 s window, so CLI startup +currently treats "process accepts connections" as "server is usable". + +`Command::Status` is a unit variant. The repository already has a `--json` convention to copy: +`PlanArgs { #[arg(long)] json: bool }` at `crates/preloop-cli/src/main.rs:706-707`. + +### The state needed for useful diagnostics already exists + +`AppState` (`crates/preloop-runner-server/src/state.rs:358-503`) and `InnerState` +(`state.rs:1067-1153`) already contain: + +- ready, dependency-held, concurrency-blocked, and expansion queues; + `pending_jobs`, `pending_expansions`, `expanding`, `queued_at`; +- registered runners, sessions, last-poll timestamps, claims, active requests, and leases; + `session_last_seen`, `runner_liveness_timeout`, `inflight_messages`, `broker_messages`, + `claimed_jobs`, `cancellation_queue`; +- pool assignment/provision reservations and the `queue_depth` / `pool_preparing` atomics; + `job_assignments`, `pool_pending`, `pool_proven_runners`, `pool_assignments_enabled`, + `require_job_assignments`; +- debug sessions and scheduler state, plus the GitHub App surface added since the first draft: + `github_app`, `github_apps`, `dispatch_token_cache`, `dispatch_actor_cache`, `github_pat`, + `github_urls`, `pr_config`, `action_sha_cache`, `pending_registrations`, `secrets`. + +`AppState::emit` (`state.rs:819`, lock released at `state.rs:827-832`) deliberately releases the state lock before persistence, +logs store failure, and broadcasts regardless. In-memory state is authoritative; the database is a +restart source. `docs/architecture.md:39-56` explicitly says two servers sharing SQLite or Postgres +still diverge. Every signal must include `service.instance.id`; this plan does not imply Preloop HA. + +Core lifecycle boundaries are centralized enough to instrument without scattering counters: + +- session create/delete: `broker.rs:368-410`; +- broker polling, claim, acquire, renew, and complete: `broker.rs:637-1202`; +- restart orphan reconciliation: `broker.rs:959-975`; +- no-matching-runner, job timeout, expired lease, and deaf-runner reaping: the reaper loop at + `bootstrap.rs:396-410`, which ticks on a 10-second `tokio::time::interval`; +- store abstraction: the private `Store` trait at `store.rs:33-55`, declared `#[async_trait]` and + `pub(crate) trait Store: Send + Sync` with exactly seven methods — `load_into`, `store_inner`, + `store_meta_only`, `store_run_event`, `store_workflow_run_counter`, `store_log_chunk`, + `append_event`. + +Instrument `Store` once with a decorator in `store.rs`; do not duplicate measurement in SQLite and +Postgres implementations. The decorator is cheap because `#[async_trait]` already erases the futures +and the trait is consumed as `Arc` (`state.rs:360`), constructed in one factory at +`store.rs:261-267`. Wrap the value that factory returns; no call site changes. + +`store_pg.rs:95-103` spawns a detached `tokio-postgres` connection task that logs one ERROR and exits +if the connection dies. Every subsequent store call then fails. Instrument the connection task's +liveness directly (`preloop.store.connection.up`), not only the per-operation failures it causes. + +### Embedded pool state is not visible to the server + +`RunnerPool::run` in `crates/preloop-orchestrator/src/lib.rs:1880-2100` owns its idle, building, +provisioning, golden-registry, and slot state. + +`RunnerPoolConfig` does not merely expose "selected shared atomics" — it already carries **four +independent ad-hoc shared handles** wired one at a time as needs arose: + +| Field | Type | Purpose | +|---|---|---| +| `pending_jobs` | `Option>` | queue depth pushed into the pool | +| `preparing_signal` | `Option>` | golden/image warm state pushed out | +| `next_job_runs_on` | `Option>>>` | label hint pushed into the pool | +| `pending_registrations` | `Option>>>` | in-flight registrations | + +Therefore the directive is **consolidation, not addition**: introduce one neutral `PoolStatus` handle +in `preloop-observability`, move these four channels onto it, and delete the ad-hoc fields. Adding a +fifth parallel channel beside four existing ones is a regression, and four `Option>` fields +that may each independently be `None` already make "what is the pool doing" unanswerable. Do not add a +server-to-orchestrator dependency and do not make the status endpoint shell out to SmolVM. + +### VM resource telemetry has stable host-side inputs but no Preloop surface + +`VmProvider` in `crates/preloop-vm/src/lib.rs:290-331` exposes lifecycle plus `MachineState` only +(`create`, `start`, `start_forkable`, `fork`, `stop`, `delete`, `status`, `list`, `exec`, +`exec_with_secret_env`, `exec_stream`, `rearm_fork_base`, `copy`, `pack`). +`SmolVmProvider::status` at `crates/preloop-vm/src/lib.rs:1018-1043` shells out to `smolvm machine status` and substring- +matches lowercased human text, discarding the PID and configured resource data needed for telemetry: + +```rust +let text = String::from_utf8_lossy(&output.stdout).to_ascii_lowercase(); +Ok(if text.contains("running") { + MachineState::Running +} else if text.contains("stopped") { + MachineState::Stopped +} else { + MachineState::Unknown +}) +``` + +The supported SmolVM floor is **`1.8.1`** (`versions.toml:8`, `smolvm_min_version`); the golden pin is +`1.8.2` (`versions.toml:16`). The v1.8.1 contract is stronger than the first draft assumed: + +- `machine status --json` (`src/cli/machine.rs:3783-3789` → `vm_common::status_vm_json`) and + `machine ls --json` (`src/cli/machine.rs:3857-3862`) emit **the same per-machine object**, built by + one shared `machine_status_json` helper (`src/cli/vm_common.rs:2360-2412`) explicitly so "the two + outputs never drift apart". +- Fields: `name`, `state`, `cpus`, `memory_mib`, `pid`, `mounts`, `ports`, `created_at`, `storage_gb`, + `overlay_gb`, `image`, `ephemeral`, `forkable`, `forkpoint_held`, `labels`, `restart_policy`, + `restart_max_retries`, `restart_count`, `health_cmd`, `health_interval_secs`, `health_timeout_secs`, + `health_retries`, `health_startup_grace_secs`, `network`, plus GPU/CUDA fields. +- `state` is **not** the persisted record state. `machine_status_json` resolves it through + `agent::state_probe::resolve_state`, a vsock liveness probe that yields a distinct + **`Unreachable`** state when the record says `Running` and the VMM PID is alive but the guest agent + does not answer (`src/agent/state_probe.rs:6-76`). That is a first-class "the VM is wedged" signal + Preloop currently throws away by substring-matching for `running`. +- `machine data-dir` is a public command (`src/cli/machine.rs:365, 4290-4310`) exposing the + hash-derived machine path; the JSON object does not carry it, so disk-path resolution still needs + this call. + +Because `machine ls --json` returns the identical object for every machine in one process, the fleet +sampler needs **one subprocess per pass, not one per VM** — and Preloop already parses exactly this +output today in `SmolVmProvider::list` (`crates/preloop-vm/src/lib.rs:1045-1060`). Use `machine ls +--json` for periodic fleet inspection and `machine status --json` only for a single machine on a +lifecycle/reconciliation path. Never call either from `/metrics` or `/api/v1/status`. + +On Linux, Preloop already establishes a delegated cgroup-v2 root and enables `cpu`, `memory`, and +`pids` before the pool starts (`init_vm_cgroup_delegation`, `preloop-vm/src/lib.rs:1360` ff., and +`preloop-cli/src/main.rs:1057-1067`). SmolVM's own supervisor then creates a capped per-VMM +`vm-` leaf under that delegated root — this is upstream-documented behavior at the supported +floor (`smolvm v1.8.1 src/process.rs:375-393`, `place_in_cgroup`), not a Preloop-side invariant, so a +missing leaf must degrade to the process fallback rather than be treated as an error. Those +host files provide CPU time/throttling, host memory current/limit/events, and VMM/helper PID counts. +On macOS and cgroup-unavailable Linux, fall back to host process CPU time and RSS with PID-start-time +validation. + +These are **host-observed VM/VMM metrics**, not guest-kernel metrics. Host RSS is not guest “memory +used”; cgroup PID counts are VMM/helper processes, not processes inside the guest; host cgroup OOM +events do not detect every guest-kernel OOM. Never run periodic commands inside the guest to fill the +gap: that changes the workload being measured and can hang behind a sick guest. + +### Local and self-hosted topology is deliberately small + +`docs/self-hosting.md` defines `preloop serve` as one process containing the control plane and VM pool. +The native API is bearer-authenticated and loopback/private by default; only the GitHub webhook needs +a public route. The systemd unit captures stdout/stderr in journald. Therefore the default profile +must add no process and must retain stderr/journald output even when OTLP is configured. + +### Six blind spots the first draft missed + +Re-auditing at `673bdfa0` found six classes of operator-visible state with no surface at all. They are +not refinements of the sections above; each can independently make Preloop behave incorrectly in a way +no log line, metric, or status field currently reports. + +#### 1. Bounded buffers and hard limits drop data silently + +Preloop enforces several caps. Most discard user-visible data with no counter and, in the worst cases, +no log line at all: + +| Limit | Location | Value | Observable today? | +|---|---|---|---| +| per-job live-log retained bytes | `live_logs.rs:25` `DEFAULT_MAX_BYTES` | 64 MiB | **No.** Tail-drops oldest wrappers silently (`live_logs.rs:46`) | +| oversized live-log batch | `live_logs.rs:250` | > per-job cap | WARN only, no counter | +| `concurrency: queue max` pending holders | `concurrency.rs:255` `QUEUE_MAX_PENDING` | 100 | **No.** Overflow sets `cancel_arrival` and cancels a user's job (`concurrency.rs:281`) | +| archived debug sessions | `debug_sessions.rs:64` `MAX_ARCHIVED_SESSIONS` | 64 | **No.** Ring eviction | +| debug session events | `debug_sessions.rs:68` `MAX_SESSION_EVENTS` | 512 | **No.** Ring eviction | +| debug session **audit** entries | `debug_sessions.rs:71` `MAX_SESSION_AUDIT` | 512 | **No.** Ring eviction — silent audit-trail loss | +| completed debug operations | `debug_sessions.rs:74` `MAX_COMPLETED_OPS` | 256 | **No.** Ring eviction | +| git request body | `snapshots.rs:20` `MAX_GIT_REQUEST_BYTES` | 16 MiB | Rejected, not counted | +| reusable-workflow nesting | `remote_workflows.rs:6` `MAX_REUSABLE_WORKFLOW_DEPTH` | 4 | Rejected, not counted | + +A cap that silently discards data is worse than an outage: the operator sees a plausible but wrong +answer. `queue max` overflow is the sharpest case — a user's job is cancelled by policy and nothing +distinguishes that from any other cancellation. Silent audit eviction is a compliance problem, not a +telemetry nicety. + +This plan therefore adds one bounded instrument family, `preloop.limit.*`, whose `limit` attribute is +the **constant's name** (a compile-time-finite set), never a value or an identifier. Every constant in +the table above must be registered through it, and any future cap must be added with it. + +#### 2. Scheduled workflows have a history endpoint and no telemetry + +`scheduler.rs` drives cron/scheduled workflows, spawns its scan from `bootstrap.rs:479` and +`bootstrap.rs:485`, and exposes `GET /api/v1/scheduler/history` (`routes.rs:419`). Nothing reports +whether a schedule fired, fired late, was skipped because the previous instance was still running, or +silently stopped firing because the scan task died. "My nightly job did not run" is a first-order +operator question with a zero-coverage answer today. + +#### 3. GitHub dependency budget is untracked + +`AppState` holds `dispatch_token_cache` and `dispatch_actor_cache` (60 s TTL, +`dispatch_auth.rs:46, 49`), `action_sha_cache`, `github_pat`, and `github_app`/`github_apps`. +`actions.rs:32` sets `ACTION_TICKET_TTL_SECS` to six hours; `oidc.rs:25` sets `TOKEN_TTL_SECS` to 300. +Nothing reads `x-ratelimit-remaining`/`x-ratelimit-reset`, tracks installation-token expiry, or reports +cache hit rate. GitHub secondary-rate-limit exhaustion is the single most common integration outage +for a control plane of this shape, and it is currently indistinguishable from "GitHub is slow". + +#### 4. Persistent storage growth is unbounded and unreported + +Preloop writes to the state dir (`preloop.db` plus WAL, checkpointed every +`store.rs:211` `WAL_CHECKPOINT_INTERVAL` = 128 commits), the cache and artifact stores, the run-log +store, the snapshot object cache (`snapshots.rs:1187` `ObjectCache`, GC spawned at `snapshots.rs:1896`), +replay results (pruned at `distributed_task.rs:895`), and VM images/overlays. Only the VM volume gets a +free-space signal in the first draft. A full state-dir filesystem is a hard outage, and the first +symptom today is a store write failure with no capacity context. + +Plan 001 owns cache quotas and eviction policy. Plan 002 owns the measurement contract: 001 must emit +through the `preloop.storage.*` family defined here rather than inventing its own. + +#### 5. Fifteen background tasks, two proposed heartbeats + +The first draft made `/readyz` depend on "the state sampler and reaper heartbeats". The real inventory +of long-lived tasks whose death is currently silent: + +| Task | Location | Cadence | +|---|---|---| +| reaper (timeouts, leases, deaf runners) | `bootstrap.rs:396-410` | 10 s interval | +| scheduler scan | `bootstrap.rs:479`, `bootstrap.rs:485` | timer | +| GitHub App event loop | `bootstrap.rs:497`, `bootstrap.rs:517` | event-driven | +| shutdown/lifecycle supervisor | `bootstrap.rs:568` | event-driven | +| listener accept loops (TCP + unix) | `bootstrap.rs:601, 615, 660, 679, 720` | per connection | +| snapshot object-cache GC | `snapshots.rs:1896` | periodic | +| replay-result prune | `distributed_task.rs:895` | periodic | +| Postgres connection task | `store_pg.rs:95-103` | lifetime | +| GitHub check dispatch | `github.rs:1064` | event-driven | +| auto-PR gate | `github_pr.rs:397` | event-driven | +| `RunnerPool::run` supervisor | `preloop-orchestrator/src/lib.rs:1880-2100` | loop | +| guest pause watchers | `preloop-orchestrator/src/lib.rs:3027, 4395, 4450` | per VM | +| pool run/watch tasks | `preloop-orchestrator/src/lib.rs:3367, 3532, 5854, 5875` | per operation | +| key rotation | `preloop-orchestrator/src/keys.rs:74` | periodic | + +Hand-wiring two heartbeats and leaving twelve silent is the same failure mode this plan exists to fix. +Replace the ad-hoc approach with one `TaskHeartbeat` registry in `preloop-observability`: a task +registers a stable name at spawn, beats each iteration, and deregisters on clean exit. Readiness and +status read the registry; a task that stops beating or panics is visible generically. Only the tasks +marked "critical" in the registry gate `/readyz`, so an event-driven task idling is not a failure. + +#### 6. HTTP surfaces are wider than the first draft's classification + +`routes.rs` serves thirteen path families, not the eight the first draft enumerated: `/_apis`, +`/broker`, `/runner`, `/runner/server`, `/api/v1`, `/twirp`, `/twirp-blob`, `/internal/test`, +`/ws/live-logs`, `/snapshots`, `/.well-known`, `/oidc`, `/repos`, plus `/healthz`. Routers are +assembled by merge (`routes.rs:39` `protected_apis`, `:209` `results_metadata`, `:233` `dispatch_api`, +merged at `:349, 797, 799, 813`). Four middlewares carry the auth contract: `require_native_bearer`, +`require_results_bearer`, `require_job_runtime_bearer`, and `resolve_runner_identity`, with +`auth.rs::runner_surface_only` layered only on the unix router. + +Two of these need different treatment from a request-duration histogram: + +- `/ws/live-logs` is a long-lived WebSocket. A duration histogram over its lifetime measures nothing + useful and pollutes the latency SLI. Use a connection gauge plus a close-reason counter. +- `/snapshots` and `/repos` serve git objects to VMs. These are the path by which workflow source + reaches a job; a failure here fails every job with a misleading in-workflow error. + +## Architecture and invariants + +```mermaid +flowchart LR + CLI[preloop status] -->|native bearer| STATUS[/api/v1/status] + PROBE[systemd / operator probes] --> LIVE[/healthz + /readyz] + PROM[Prometheus-compatible scraper] -->|native bearer| METRICS[/metrics] + + SERVER[Control plane] --> OBS[preloop-observability] + POOL[Runner pool] --> OBS + VMHOST[Host VM sampler: cgroup/process/sparse disk] --> OBS + OBS --> STDERR[stderr / journald] + OBS --> METRICS + OBS -. optional bounded OTLP/HTTP .-> O2[OpenObserve or any OTLP backend] + COLLECTOR[Optional OTel Collector for host telemetry] -.-> O2 + + SERVER --> RUNLOGS[Existing workflow run-log store] + CAPTURE[Explicit protocol flow capture] --> LOCALFILE[Local 0600 capture file] +``` + +### New crate + +Create `crates/preloop-observability` with a small, explicit API and no dependency on server or +orchestrator internals: + +- `ObservabilityConfig::from_env()` parses logging and standard OTel configuration without exposing + header values in `Debug` or errors. +- `Observability::noop()` is allocation-light and makes unit tests and library-only consumers perform + no network I/O. +- `Observability` is a cloneable handle containing pre-created instruments, the cached operational + snapshot, pool/VM-status handles, critical-task heartbeats, and telemetry-export health. +- `TaskHeartbeat` registry: `register(name, criticality) -> HeartbeatHandle`, `beat()`, + `Drop` deregisters. Names come from a compile-time-finite set; readiness reads only entries marked + critical. This replaces per-task ad-hoc `AtomicU64` timestamps. +- `LimitRegistry`: `record_drop(limit, count)` / `record_reject(limit)` where `limit` is a + `&'static str` constant name, backing the `preloop.limit.*` family and the status `limits` block. +- `ObservabilityRuntime` owns subscriber/provider guards and performs bounded shutdown/flush. +- `OperationalSnapshot`, `Condition`, `PoolSnapshot`, `VmFleetSnapshot`, `VmSample`, and their bounded + enums are serializable DTOs shared by server and CLI. +- OpenTelemetry API/instrument types are always available; SDK, OTLP/HTTP, Prometheus, tracing bridge, + and formatting layers are host-only Cargo features. Disable default features on exporter crates and + do not pull the tonic/gRPC stack in the first implementation. + +Both `preloop` and standalone `preloop-server` construct one handle/runtime before building +`ServerConfig`; the same handle is cloned into `AppState` and `RunnerPoolConfig`. The guest +`preloop-runner` gets only the structured local logger and never exports directly by default. + +### Non-negotiable invariants + +1. **Fail open**: telemetry creation or export failure produces a sanitized warning and status + condition, never a failed request or process exit. It cannot change queue, runner, or run state. +2. **No request-path export**: logs, metrics, and spans enqueue into bounded nonblocking SDK batches. + No handler awaits an exporter. Queue overflow drops telemetry and increments local drop health. +3. **Bounded shutdown**: attempt flush for at most two seconds after server/pool shutdown. Then exit. +4. **No backend by default**: absent explicit `OTEL_EXPORTER_OTLP_*` endpoint configuration, Preloop + opens no telemetry connection. Local metrics/status/logging still work. This is a **deliberate + deviation from the OTel specification**, whose default `OTEL_EXPORTER_OTLP_ENDPOINT` is + `http://localhost:4318`. Preloop treats an absent variable as "disabled", not "localhost", because + a CI control plane must not emit background connection attempts on an operator's machine. Document + the deviation in `docs/observability.md`; an operator who wants spec behavior sets the variable. +5. **Always retain stderr**: OTLP augments, never replaces, terminal/journald logs. +6. **No wire changes**: do not add fields, alter status codes, or change bodies on `/_apis`, `/broker`, + `/runner`, or Twirp/result-service routes. Instrument around existing behavior. +7. **No high-cardinality metric attributes**: IDs and user-controlled strings are logs/traces only. +8. **No workflow output export**: workflow stdout/stderr stays in the existing run-log store. +9. **No raw HTTP capture**: do not collect headers, bodies, raw URI, query strings, or error text into + metrics. Trace attributes are allowlisted, not denylisted. +10. **No state-lock callbacks**: OTel observable callbacks cannot await or lock `InnerState`. A periodic + async sampler updates a small cached snapshot; exporters read that snapshot. +11. **No avoidable hot-path allocation**: instruments and attribute arrays are prebuilt where static; + poll/renew success is metrics-only and DEBUG-filtered, not an INFO log allocation. +12. **No guest polling**: VM sampling reads host cgroup/process/filesystem state only. It never executes + `free`, `df`, `ps`, `dmesg`, or another command inside a runner VM. +13. **No fake zeroes**: an unsupported or stale VM metric is absent and its capability is reported + unavailable. Zero means the source measured zero. +14. **No silent drop**: any code path that discards, evicts, truncates, or rejects data because of a + cap MUST record it through `LimitRegistry`. A cap without a counter is a defect. This applies to + telemetry's own bounded queues as much as to live-log buffers and concurrency queues. +15. **No unregistered long-lived task**: every `tokio::spawn` that outlives a request registers a + `TaskHeartbeat`. A background loop whose death is invisible is the failure this plan exists to + remove; the fifteen-task inventory above is the acceptance baseline, not an example list. + +## Operator surfaces + +### `/healthz`: liveness only + +Keep it unauthenticated and intentionally shallow. Return 200 while the process can serve requests and +503 once shutdown begins. Response fields: schema version, `ok`, protocol version. It must not call +SQLite/Postgres, GitHub, OpenObserve, SmolVM, or acquire `InnerState`. + +### `/readyz`: critical-loop readiness + +Keep it unauthenticated but reveal only boolean state and stable reason codes. Return 200 after durable +state restoration and router startup, while every `TaskHeartbeat` marked critical is fresh and +shutdown has not started. Return 503 for `starting`, `shutting_down`, or `task_stale` with the stale +task's registry name as the reason code. The critical set is exactly: the state sampler, the reaper +(`bootstrap.rs:396`), the scheduler scan (`bootstrap.rs:479`), and — when the backend is Postgres — +the connection task (`store_pg.rs:95`). Everything else is reported in `/api/v1/status` but does not +gate readiness. + +Reason codes are the registry names, so adding a critical task adds a code without a schema change. +Non-critical task staleness must never return 503: an event-driven task with no events is healthy. + +Do **not** make readiness depend on runner capacity, GitHub reachability, store write success, workflow +success, or telemetry export. This is a single-authoritative-process design; making a dependency +failure fail readiness would hide the only control plane that can explain it and could create a +restart loop. Rich degradation belongs in `/api/v1/status`. + +Change `wait_for_engine_socket` (`crates/preloop-cli/src/main.rs:1458-1475`) to probe `/readyz`, not +`/healthz`. Keep its 500 ms per-attempt timeout and 30 s window; on window expiry, report the last +`/readyz` reason code instead of a generic timeout, so "server started but the scheduler never came +up" is distinguishable from "server never bound". + +### `/api/v1/status`: authenticated operational diagnosis + +Add an authenticated native endpoint returning a stable, versioned snapshot. It reads the most recent +sampler snapshot without waiting on `InnerState` and reports `snapshot_age_seconds`; sampling every +five seconds is sufficiently current and remains responsive when the state lock is the incident. +Limit problem exemplars to five per condition. + +Required shape (field names may be Rust snake_case internally but the JSON contract is fixed): + +```json +{ + "schema_version": 1, + "observed_at": "RFC3339 timestamp", + "snapshot_age_seconds": 0.4, + "overall": "ok|degraded|blocked|shutting_down", + "service": { + "version": "0.x.y", + "instance_id": "uuid-per-process", + "uptime_seconds": 123, + "shutdown_requested": false + }, + "runs": {"queued": 0, "in_progress": 1, "completed": 12}, + "jobs": { + "ready": 2, + "dependency_blocked": 1, + "concurrency_blocked": 0, + "pending_expansion": 0, + "expanding": 0, + "claimable": 1, + "unclaimable": 1, + "oldest_ready_seconds": 14.2 + }, + "concurrency": { + "groups_active": 3, + "groups_contended": 1, + "pending_holders": 4, + "deepest_group_pending": 4, + "queue_max_pending": 100, + "overflow_cancellations": 0 + }, + "scheduler": { + "enabled": true, + "schedules": 4, + "last_scan_at": "RFC3339 timestamp", + "next_fire_at": "RFC3339 timestamp", + "fired": 12, + "skipped_overlapping": 1, + "late_fires": 0, + "max_fire_delay_seconds": 2.1 + }, + "runners": { + "registered": 2, + "sessions": 2, + "idle": 1, + "busy": 1, + "stale": 0, + "max_poll_age_seconds": 3.1, + "max_lease_age_seconds": 8.0 + }, + "pool": { + "mode": "warm|on_demand|external|disabled", + "desired": 2, + "preparing": false, + "building": 0, + "provisioning": 0, + "idle": 1, + "busy": 1, + "paused": 0, + "consecutive_provision_failures": 0, + "last_transition_at": "RFC3339 timestamp" + }, + "vms": { + "source": "cgroup_v2|process|mixed|unavailable", + "sample_age_seconds": 1.2, + "capabilities": { + "cpu": true, + "memory": true, + "cpu_throttling": true, + "host_oom_events": true, + "host_pids": true, + "sparse_disk_allocation": true, + "block_io": false, + "network_io": false, + "guest_os": false + }, + "count": {"runner": 2, "golden": 1, "unavailable": 0}, + "configured": { + "vcpus": 10, + "memory_bytes": 21474836480, + "storage_bytes": 85899345920, + "overlay_bytes": 21474836480 + }, + "host_usage": { + "cpu_cores": 2.4, + "memory_bytes": 7516192768, + "sparse_disk_allocated_bytes": 12884901888 + }, + "top_consumers": [ + { + "machine_name": "preloop-runner-0", + "role": "runner", + "activity": "busy", + "run_id": "authenticated-detail-only", + "job_id": "authenticated-detail-only", + "cpu_cores": 1.8, + "memory_bytes": 4294967296, + "memory_limit_bytes": 7516192768, + "sparse_disk_allocated_bytes": 5368709120, + "host_cpu_throttled_seconds": 0.0, + "host_oom_kills": 0, + "sample_age_seconds": 1.2 + } + ] + }, + "store": { + "backend": "sqlite|postgres", + "consecutive_failures": 0, + "last_success_at": "RFC3339 timestamp", + "last_failure_at": null + }, + "github": { + "configured": true, + "last_webhook_at": "RFC3339 timestamp", + "pending_check_updates": 0, + "last_check_success_at": "RFC3339 timestamp", + "last_check_failure_at": null + }, + "debug": {"active_sessions": 0, "oldest_session_seconds": null}, + "storage": { + "state_dir": "/var/lib/preloop", + "state_fs_free_bytes": 41231234560, + "state_fs_free_ratio": 0.42, + "components": [ + {"store": "database", "bytes": 184549376}, + {"store": "cache", "bytes": 2147483648}, + {"store": "artifacts", "bytes": 536870912}, + {"store": "run_logs", "bytes": 268435456}, + {"store": "snapshots", "bytes": 1073741824}, + {"store": "vm_images", "bytes": 68719476736} + ], + "last_gc_at": "RFC3339 timestamp" + }, + "limits": [ + { + "limit": "LIVE_LOG_MAX_BYTES", + "value": 67108864, + "dropped": 0, + "rejected": 0, + "last_at": null + } + ], + "tasks": [ + { + "name": "reaper", + "critical": true, + "heartbeat_age_seconds": 3.2, + "state": "running|idle|stale|exited" + } + ], + "telemetry": { + "otlp_enabled": false, + "last_export_success_at": null, + "last_export_failure_at": null, + "dropped_records": 0 + }, + "conditions": [] +} +``` + +The `github` block gains dependency-budget fields, which are the difference between "GitHub is slow" +and "we are rate limited": + +```json +"github": { + "configured": true, + "last_webhook_at": "RFC3339 timestamp", + "pending_check_updates": 0, + "last_check_success_at": "RFC3339 timestamp", + "last_check_failure_at": null, + "rate_limit": { + "resource": "core", + "limit": 5000, + "remaining": 4812, + "reset_at": "RFC3339 timestamp", + "observed_at": "RFC3339 timestamp" + }, + "installation_token_expires_in_seconds": 2841, + "token_cache": {"hits": 91, "misses": 4, "ttl_seconds": 60} +} +``` + +`limits` and `tasks` are arrays, not fixed objects, because both sets grow with the code. Each entry's +`limit`/`name` is a compile-time constant identifier, so the array length stays bounded and the JSON +contract does not change when a cap or task is added. `limits` reports every registered cap, including +those with zero drops — an operator must be able to see the ceiling before hitting it. + +Conditions use stable codes and safe messages. Initial codes: + +- `state_sampler_stale` +- `task_stale` +- `task_exited` +- `queue_no_registered_runner` +- `queue_label_mismatch` +- `concurrency_queue_overflow` +- `concurrency_group_starved` +- `scheduler_scan_stale` +- `scheduler_fire_late` +- `scheduler_skipped_overlapping` +- `pool_preparing` +- `pool_provisioning_deficit` +- `pool_repeated_provision_failure` +- `runner_poll_stale` +- `runner_lease_stale` +- `vm_sampler_stale` +- `vm_sample_unavailable` +- `vm_unreachable` +- `vm_host_memory_pressure` +- `vm_host_cpu_throttled` +- `vm_host_oom_kill` +- `vm_sparse_disk_pressure` +- `store_write_failure` +- `store_connection_down` +- `storage_capacity_pressure` +- `limit_drop_active` +- `limit_reject_active` +- `github_check_update_failure` +- `github_terminal_check_pending` +- `github_rate_limit_low` +- `github_installation_token_expiring` +- `debug_session_stale` +- `debug_audit_evicted` +- `telemetry_export_failure` + +Authenticated condition exemplars may contain run/job/runner/session/machine IDs and `runs-on` labels, +but never tokens, environment values, request bodies, or URLs with query strings. Metrics must contain +only the condition code, never the exemplar fields. + +### `preloop status` + +Change `Status` to `Status(StatusArgs)` with `--json`. Human output must show, in order: + +1. service and snapshot age; +2. ready/blocked queue classes and oldest wait; +3. concurrency-group contention and scheduler state; +4. pool capacity and runner/session freshness; +5. VM fleet capacity, current host usage, capability gaps, and bounded top consumers; +6. store, storage capacity, GitHub budget, debug, and telemetry state; +7. any limit with a nonzero drop/reject count (omit clean limits from human output; keep all of them + in `--json`); +8. background tasks that are stale or exited (omit healthy ones from human output); +9. typed conditions with one-line actions; +10. the existing recent-runs table. + +`--json` prints the endpoint response exactly and no prose, so operators can use `jq` and monitoring +scripts. Preserve native bearer behavior. Do not add a watch loop in this plan. + +Follow the existing flag convention (`PlanArgs { #[arg(long)] json: bool }`, +`crates/preloop-cli/src/main.rs:706-707`) rather than inventing a new one. + +## Signal contract + +### Resource attributes on every exported signal + +- `service.name=preloop` (overridable only through standard `OTEL_SERVICE_NAME`) +- `service.version=` +- `service.instance.id=` +- `deployment.environment.name` only when supplied through standard resource attributes +- host/OS attributes only from an explicitly enabled resource detector + +Do not attach repository, workflow, branch, SHA, runner name, or machine name as a resource attribute. + +### Metric attribute policy + +Allowed metric values are bounded enums or finite route templates: + +- HTTP method, Axum matched route, protocol surface, status-code class; +- execution state/conclusion and stable termination reason; +- queue kind, dispatch outcome, pool mode/state, operation, backend, result; +- VM role/activity/runtime state, host sample source, storage kind, capability, and stable sample error; +- webhook/check/cache/artifact/debug operation and stable outcome. +- `&'static str` constant names from a compile-time-finite set: `limit` (cap identifiers), + `task` (heartbeat registry names), `store` (storage component identifiers). These are code + identifiers, not data, so their cardinality is bounded by the source, not by traffic. + +Forbidden metric attributes: + +- run/job/request/runner/session/machine IDs; +- repository/workflow/ref/SHA, `runs-on` label values, cache keys, artifact names; +- raw URL/path/query, webhook delivery ID, GitHub installation ID; +- error message/type text, filenames, user agent, IP address; +- any token, header, body, secret, or environment value. + +Add a cardinality test that drives at least 1,000 distinct IDs/names through instrumentation, gathers +the Prometheus registry, asserts a fixed upper bound on series count, and asserts none of the unique +values occur in exposition text. + +### Metrics catalog + +Use seconds for durations and bytes for sizes. Histograms count observations; do not add duplicate +request counters when histogram count answers the same question. + +| Instrument | Type | Required attributes | Purpose | +|---|---|---|---| +| `http.server.request.duration` | histogram | method, matched route, surface, status class | API latency/error rate using OTel HTTP semantics | +| `http.server.active_requests` | up/down counter | method, matched route, surface | current HTTP concurrency | +| `preloop.broker.poll` | counter | surface, outcome=`job|cancel|empty|error` | distinguish healthy empty long-polls from dispatch failures | +| `preloop.run.active` | observable gauge | state | runs by current state | +| `preloop.run.completed` | counter | conclusion, termination reason | terminal runs without user identifiers | +| `preloop.run.duration` | histogram | conclusion | run wall time | +| `preloop.job.active` | observable gauge | state | jobs by current state | +| `preloop.job.completed` | counter | conclusion, termination reason | terminal jobs and bounded failure taxonomy | +| `preloop.job.duration` | histogram | conclusion | execution time after claim | +| `preloop.job.queue.depth` | observable gauge | queue kind | ready/dependency/concurrency/expansion depth | +| `preloop.job.queue.oldest_age` | observable gauge | queue kind | detect stalls, not just depth | +| `preloop.job.queue.wait` | histogram | outcome | ready-to-claim or terminal-unclaimed wait | +| `preloop.job.claimability` | observable gauge | reason | claimable vs temporary/permanent unclaimability | +| `preloop.concurrency.group.active` | observable gauge | state=`holding|pending` | concurrency-group occupancy without group names | +| `preloop.concurrency.pending.depth` | observable gauge | none | deepest pending queue across groups | +| `preloop.concurrency.decision` | counter | queue mode, action=`park|cancel_pending|cancel_arrival|admit` | why a job was parked or cancelled by `concurrency:` policy | +| `preloop.scheduler.fire` | counter | outcome=`fired|skipped_overlapping|error` | did a schedule actually run | +| `preloop.scheduler.fire.delay` | histogram | none | scheduled time to actual dispatch | +| `preloop.scheduler.schedules` | observable gauge | none | registered schedule count | +| `preloop.runner.count` | observable gauge | state=`registered|idle|busy|stale` | runner inventory | +| `preloop.runner.session.count` | observable gauge | state | active sessions | +| `preloop.runner.session.transition` | counter | operation, reason, outcome | create/delete/reap/reconcile lifecycle | +| `preloop.runner.poll.max_age` | observable gauge | none | deaf-runner leading indicator | +| `preloop.runner.lease.max_age` | observable gauge | none | expired-lease leading indicator | +| `preloop.pool.runner.count` | observable gauge | mode, state=`desired|idle|busy|building|provisioning|paused` | capacity and deficit | +| `preloop.pool.preparing` | observable gauge | mode | image/golden warm state | +| `preloop.pool.operation.duration` | histogram | operation, outcome, reason | prepare/provision/register/assign/delete/replace | +| `preloop.vm.count` | observable gauge | role, activity, runtime state, pool mode | active runner/golden VM inventory | +| `preloop.vm.configured.vcpus` | observable gauge | role, pool mode | configured virtual CPU capacity | +| `preloop.vm.configured.memory` | observable gauge | role, pool mode | configured guest-memory ceiling in bytes | +| `preloop.vm.configured.storage` | observable gauge | role, storage kind=`root|overlay` | configured logical disk capacity in bytes | +| `preloop.vm.host.cpu.time` | counter | role, pool mode | sampled host CPU seconds attributable to VMM cgroups/processes | +| `preloop.vm.host.cpu.cores` | observable gauge | role, activity, pool mode | current host CPU-seconds/second, expressed as cores | +| `preloop.vm.host.cpu.throttled_time` | counter | role, pool mode | Linux cgroup throttle seconds; absent elsewhere | +| `preloop.vm.host.cpu.throttled_periods` | counter | role, pool mode | Linux cgroup throttled-period count; absent elsewhere | +| `preloop.vm.host.memory.usage` | observable gauge | role, activity, source | current cgroup memory or process RSS in bytes | +| `preloop.vm.host.memory.limit` | observable gauge | role, source | cgroup/configured host memory limit in bytes | +| `preloop.vm.host.memory.events` | counter | role, event=`low|high|max|oom|oom_kill` | Linux host-cgroup events; not guest-kernel OOM | +| `preloop.vm.host.pids.current` | observable gauge | role | Linux VMM/helper process count; not guest process count | +| `preloop.vm.host.pids.limit` | observable gauge | role | Linux VMM/helper process limit | +| `preloop.vm.host.pids.events` | counter | role, event=`max` | Linux host-cgroup PID-limit events | +| `preloop.vm.sparse_disk.allocated` | observable gauge | role, storage kind | physically allocated host blocks for VM-private/shared files | +| `preloop.vm.storage.available` | observable gauge | storage class=`smolvm_data|preloop_state` | authoritative free bytes on the filesystem holding VM state | +| `preloop.vm.age.max` | observable gauge | role, activity | oldest active VM age for leaked-machine detection | +| `preloop.vm.sampler.available` | observable gauge | capability, source | whether each VM signal can be measured on this host | +| `preloop.vm.sampler.age` | observable gauge | source | age of the last successful fast sample | +| `preloop.vm.sampler.errors` | counter | source, reason | bounded sampler failures such as permission, parse, PID reuse, process gone | +| `preloop.store.operation.duration` | histogram | backend, operation, outcome | one decorator around all store methods | +| `preloop.store.consecutive_failures` | observable gauge | backend | restart-durability risk | +| `preloop.store.connection.up` | observable gauge | backend | Postgres connection task alive; SQLite always 1 | +| `preloop.storage.bytes` | observable gauge | store=`database|cache|artifacts|run_logs|snapshots|vm_images` | persistent footprint per component | +| `preloop.storage.fs.available` | observable gauge | mount=`state_dir|smolvm_data` | authoritative free bytes | +| `preloop.storage.gc` | counter | store, outcome | eviction/prune passes; Plan 001 emits through this | +| `preloop.limit.dropped` | counter | limit | records discarded by a cap (live-log tail-drop, ring eviction) | +| `preloop.limit.rejected` | counter | limit | requests/arrivals refused by a cap (queue max, body size, nesting depth) | +| `preloop.limit.value` | observable gauge | limit | the configured ceiling, so alerts compare against it without hardcoding | +| `preloop.task.heartbeat.age` | observable gauge | task | seconds since last beat; the generic dead-loop detector | +| `preloop.task.exited` | counter | task, outcome=`clean|error|panic` | background task termination | +| `preloop.github.operation.duration` | histogram | operation, outcome | token/check/API dependencies | +| `preloop.github.check.propagation_delay` | histogram | outcome | terminal run to GitHub acknowledgement | +| `preloop.github.rate_limit.remaining` | observable gauge | resource | `x-ratelimit-remaining` from the last response | +| `preloop.github.rate_limit.limit` | observable gauge | resource | `x-ratelimit-limit` | +| `preloop.github.rate_limit.reset_in` | observable gauge | resource | seconds until the window resets | +| `preloop.github.token.expires_in` | observable gauge | kind=`installation|oidc|action_ticket` | credential expiry countdown | +| `preloop.github.token.cache` | counter | kind, outcome=`hit|miss|expired` | dispatch/actor/action-SHA cache effectiveness | +| `preloop.webhook.duration` | histogram | event class, outcome, dedup outcome | webhook processing without delivery/repo labels | +| `preloop.cache.operation.duration` | histogram | operation, outcome | cache behavior | +| `preloop.cache.transfer.bytes` | counter | direction, outcome | payload volume | +| `preloop.artifact.operation.duration` | histogram | operation, outcome | artifact behavior | +| `preloop.artifact.transfer.bytes` | counter | direction, outcome | payload volume | +| `preloop.debug.session.count` | observable gauge | state | active/paused/detached sessions | +| `preloop.debug.session.duration` | histogram | terminal reason | leaked-session detection | +| `preloop.snapshot.operation.duration` | histogram | operation, outcome | git object serving that feeds every job | +| `preloop.snapshot.cache` | counter | outcome=`hit|miss|evicted` | snapshot object-cache effectiveness | +| `preloop.livelog.connections` | up/down counter | none | open `/ws/live-logs` WebSockets; not a duration histogram | +| `preloop.livelog.connection.closed` | counter | reason | WebSocket termination taxonomy | +| `preloop.livelog.buffer.bytes` | observable gauge | none | retained live-log bytes against the 64 MiB per-job cap | +| `preloop.telemetry.export` | counter | signal, outcome | exporter self-health, visible locally at `/metrics` | +| `preloop.service.uptime` | observable gauge | none | no-data heartbeat and process continuity | + +Use explicit bucket views: + +- HTTP/store/GitHub: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds. +- queue wait: 0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 900 seconds. +- pool preparation/provision: 1, 5, 10, 30, 60, 120, 300, 600 seconds. +- scheduler fire delay: 1, 5, 15, 30, 60, 300, 900, 3600 seconds. A cron job is not late at 10 ms + resolution, and reusing the HTTP buckets would waste eight of eleven boundaries. + +`/ws/live-logs` is deliberately excluded from `http.server.request.duration`. A long-lived WebSocket +would otherwise dominate the p99 and silently break the availability SLI's denominator. + +The five-second sampler captures counts and bounded condition exemplars under `InnerState` once, then +releases the mutex before updating the shared snapshot. It must not copy log buffers, artifacts, +cache bytes, full run records, or bodies. + +### VM telemetry contract + +VM metrics are first-class Preloop application telemetry because the embedded pool owns these +processes and already knows their lifecycle/assignment. General node telemetry remains optional. + +#### Collection architecture + +1. Replace the human-text `VmProvider::status` contract with one typed inspection contract that all + provider implementations and test fakes implement. For SmolVM at the `1.8.1` floor: + - single machine on a lifecycle path (start/fork/adoption/reconciliation): `machine status --json`; + - whole fleet on the slow sampler pass: **one** `machine ls --json` call, which returns the + identical per-machine object for every machine (upstream `smolvm src/cli/vm_common.rs:2360` + builds both outputs from one helper). `SmolVmProvider::list` + (`crates/preloop-vm/src/lib.rs:1045-1060`) already parses this response — + extend it to a full `VmRuntimeInfo` instead of discarding everything but `name`. + - `machine data-dir` only when a disk path is needed; the JSON object does not carry it. + + Parse into a versioned internal `VmRuntimeInfo` and cache machine role, configured resources + (`cpus`, `memory_mib`, `storage_gb`, `overlay_gb`), `pid`, process start identity, `created_at`, + `restart_count`, `ephemeral`, `forkable`/`forkpoint_held`, and relevant disk paths. Migrate every + status caller; leave no text-parser alias. + + Map `state` faithfully, including **`Unreachable`**. SmolVM resolves `state` through a vsock + liveness probe (`src/agent/state_probe.rs:41-76`), so `Unreachable` means "record says Running, + VMM PID alive, guest agent not answering" — a wedged VM. Today's `text.contains("running")` cannot + even represent it. Surface it as `MachineState::Unreachable`, metric attribute + `runtime_state=unreachable`, and status condition `vm_unreachable`. A pool that keeps forking from + an unreachable golden is a real and currently invisible failure. +2. Register/deregister that runtime info with a `VmTelemetryRegistry` owned by the runner pool and + exposed through the neutral observability handle. Paused debug VMs remain registered until they are + actually deleted. +3. Run one host sampler task, not one task or CLI subprocess per VM. Fast cadence: five seconds for + process identity, CPU, memory, cgroup events, and PID counts. Slow cadence: 60 seconds for sparse + allocated blocks, filesystem capacity, and the single `machine ls --json` fleet reconciliation. + The fast path touches only host cgroup/proc files and spawns no subprocess at all. +4. Validate PID start identity on every fast sample. PID reuse invalidates the cache, records + `pid_reused`, and triggers one bounded re-inspection; never attribute a new process's resources to + an old VM. +5. Publish one immutable fleet snapshot to status and observable gauges after a complete pass. A + partial/error sample preserves the last good values with explicit age and availability; it never + overwrites a missing field with zero. + +Do not run `smolvm machine status` during `/metrics` or `/api/v1/status`. Those endpoints read the +cached snapshot only. + +#### Source priority and exact semantics + +| Signal | Linux preferred source | macOS/fallback source | Meaning | +|---|---|---|---| +| configured CPU/memory/storage | cached SmolVM status JSON | same | requested VM capacity, not current use | +| CPU time | `cpu.stat` `usage_usec` | cumulative VMM process CPU time | host CPU consumed by the VM runtime | +| current CPU cores | delta CPU seconds / delta monotonic wall seconds | same | host cores currently consumed; divide by configured vCPUs for a utilization ratio | +| CPU throttling | `cpu.stat` throttle fields | unavailable | host cgroup quota pressure, not guest scheduler steal time | +| host memory | `memory.current` / `memory.max` | VMM RSS plus configured limit | host memory charged to the VM runtime; not guest free/used memory | +| host memory events | deltas from `memory.events` | unavailable | host cgroup events; `oom_kill` means the host killed VMM/helper work, not necessarily a guest process | +| host PID count | `pids.current` / `pids.max` / `pids.events` | unavailable | VMM/helper host processes; not the guest process table | +| disk allocation | filesystem allocated blocks (`st_blocks × 512`) for known VM files | same | physical blocks attributed to VM files, not sparse logical length | +| host free disk | `statvfs`/equivalent on the VM state filesystem | same | authoritative capacity pressure for the host volume | + +Counter instruments record positive deltas between samples so they remain monotonic across VM +deletion. A first sample establishes a baseline; PID change/restart establishes a new baseline rather +than adding the new process's cumulative value. + +Sparse/CoW accounting has two caveats: + +- Count shared golden/image files once under `role=golden`; runner entries count only their private + overlay/runtime files. +- `st_blocks` may still double-count shared extents on filesystems that do not expose exclusive-block + ownership. Therefore filesystem free bytes are authoritative for alerts; per-VM allocated bytes are + a comparative diagnostic, not a billing total. + +Network byte/packet/drop counters and block-I/O operation/byte counters are not available through the +current stable SmolVM/Preloop boundary on every supported platform. Omit those series and report their +capabilities false. Do not emit zero. Add them only after SmolVM exposes virtio counters or Preloop +deliberately enables and validates a cgroup-IO/interface source at the supported runtime floor. + +Guest OS signals—guest memory free, guest load average, guest process count, guest filesystem free, +guest OOM killer events, and guest network I/O—are likewise absent in this phase. A future +implementation must use an out-of-band SmolVM agent push/snapshot API, not periodic guest shell +commands and not a modified official Actions runner. + +#### Cardinality and diagnostics + +Metrics aggregate by bounded `role=runner|golden`, `activity=idle|busy|paused|unassigned`, runtime +state, pool mode, source, capability, and storage kind. Never label a metric with machine, slot, +runner, run, or job identity; ephemeral VM names would create permanent time-series churn. + +`/api/v1/status` returns aggregate totals plus at most five top consumers, ordered deterministically by +pressure and carrying authenticated machine/run/job correlation. Emit transition-based, +hysteresis/rate-limited structured events for historical attribution: + +- `vm.host.memory.pressure` / `.recovered` +- `vm.host.cpu.throttled` / `.recovered` +- `vm.host.oom_kill` +- `vm.sparse_disk.pressure` / `.recovered` +- `vm.sampler.unavailable` / `.recovered` + +Threshold events name the machine/run/job in logs/traces only. They must not log environment, +commands, mounted paths, image credentials, or guest data. + +### Structured log catalog + +Use `event.name` plus stable fields. OTel trace/span IDs provide request correlation; do not allocate a +second UUID on every HTTP poll. + +| Event name | Level | Required fields | +|---|---|---| +| `server.started` / `server.stopping` | INFO | instance ID, version, listen scheme/address, store backend, pool mode | +| `run.accepted` / `run.completed` | INFO | run ID, workflow path/repository only when known, conclusion, duration | +| `job.ready` / `job.claimed` / `job.completed` | INFO | run ID, job ID, request/runner ID when relevant, conclusion/reason | +| `job.requeued` / `job.unclaimable` | WARN | run ID, job ID, stable reason, safe labels | +| `job.concurrency.cancelled` | WARN | run ID, job ID, queue mode, group hash (not the raw group key), action | +| `schedule.fired` / `schedule.skipped` | INFO/WARN | schedule ID, workflow path, delay seconds, reason | +| `runner.registered` | INFO | runner ID, runner name, safe labels | +| `runner.session.created` / `runner.session.deleted` | INFO | runner ID, session ID, reason | +| `runner.session.reaped` | WARN | runner/session ID, `deaf|lease_expired|startup_orphan` | +| `pool.prepare.started` / `pool.prepare.completed` | INFO | machine/golden name, mode, duration, outcome | +| `pool.provision.started` / `pool.provision.completed` | INFO/WARN | machine name, slot, duration, stable outcome/reason | +| `pool.supervisor.exited` | ERROR | stable reason; full safe error as log text | +| `task.exited` | WARN/ERROR | registry task name, outcome, uptime; ERROR when the task is critical | +| `limit.exceeded` | WARN | limit constant name, configured value, dropped/rejected delta; rate-limited per limit | +| `vm.host.memory.pressure` / `vm.host.memory.recovered` | WARN/INFO | machine/run/job IDs, current bytes, limit bytes, ratio, source | +| `vm.host.cpu.throttled` / `vm.host.cpu.recovered` | WARN/INFO | machine/run/job IDs, bounded interval/ratio, source | +| `vm.host.oom_kill` | WARN | machine/run/job IDs, host event delta, source | +| `vm.sparse_disk.pressure` / `vm.sparse_disk.recovered` | WARN/INFO | machine/run/job IDs, allocated bytes, host free bytes | +| `vm.sampler.unavailable` / `vm.sampler.recovered` | WARN/INFO | capability, source, stable reason, sample age | +| `vm.unreachable` / `vm.reachable` | WARN/INFO | machine name, role, run/job IDs, PID, age since last reachable | +| `store.operation.failed` | ERROR | backend, operation, consecutive failures, safe error | +| `store.connection.lost` | ERROR | backend, safe error; the Postgres connection task exiting | +| `storage.pressure` / `storage.recovered` | WARN/INFO | mount, free bytes, free ratio, largest component | +| `github.webhook.processed` | INFO/WARN | event class, delivery ID, dedup/outcome, duration | +| `github.check.updated` | INFO/WARN | run ID, operation, outcome, duration | +| `github.rate_limit.low` | WARN | resource, remaining, limit, reset-in seconds; never the token | +| `debug.session.created` / `debug.session.closed` | INFO/WARN | run/job/session ID, terminal reason, duration | +| `debug.audit.evicted` | WARN | session ID, evicted count, retained cap | +| `telemetry.export.failed` | WARN | signal, failure class, consecutive failures; no endpoint/header | + +`job.concurrency.cancelled` logs a **hash** of the concurrency group key, never the key itself: the +group expression is user-controlled and routinely interpolates branch names, PR titles, and inputs. +The hash still correlates all jobs in one group without exporting user data. + +Level policy: + +- INFO: low-frequency control-plane, pool, and lifecycle transitions. +- WARN: recoverable degradation requiring operator attention. +- ERROR: process/supervisor terminal failure, invariant break, or sustained durability failure. +- DEBUG: successful long-poll/renew/detail paths. Never INFO-log every poll or lease renewal. +- A workflow's own failure is data, not an ERROR about Preloop. + +Export control-plane and pool/VM lifecycle logs. Keep workflow step output in its current run-log store. +Keep protocol flow capture isolated. Do not add request/response body logging as a troubleshooting +shortcut. + +### Trace policy + +Trace synchronous work, not an hours-long workflow: + +- HTTP server request spans using matched route and OTel HTTP semantic attributes; +- workflow submission/build, webhook processing, broker claim/acquire; +- SQLite/Postgres operations through the decorator; +- GitHub token/check/API requests; +- image/golden preparation and individual VM provision/register/assign/delete phases; +- cache/artifact operations where they perform storage/network I/O. + +A run/job crosses many requests and processes. Correlate those transitions with structured +`run.id`, `job.id`, `request.id`, `runner.id`, `session.id`, and `machine.name` fields on logs/traces. +Do not retain one span for the complete run and do not persist span context in `InnerState` merely to +force a single trace. + +Suppress successful health/metrics probes, successful renewals, and empty broker long-polls from +normal trace export. Always record their metrics; trace errors and job/cancellation deliveries. The +reference OpenObserve profile should set a standard parent-based ratio sampler for ordinary HTTP +traffic and document how to raise it temporarily. + +## SLOs and alerts + +Instrument first, collect a two-week baseline, then ratify thresholds. Initial objectives are starting +points, not promises: + +| SLI | Initial objective | Denominator/exclusions | +|---|---|---| +| Control API/broker availability | 99.9% successful | exclude user/auth 4xx and expected empty long-polls | +| Dispatch latency | 99% within 30s | only ready jobs for which compatible capacity exists; pool preparation is reported separately | +| Terminal propagation | 99% within 30s | job terminal to run finalization and configured GitHub check acknowledgement | +| Durable-state writes | no sustained failure over 5m | all store operations; one transient failure warns, sustained failure pages | +| Runner liveness | no deaf/leaked active session beyond configured bound | active sessions only; completed sessions excluded | +| VM telemetry freshness | no stale fast sample beyond three intervals | Preloop-owned active VMs on hosts where the relevant source is supported | +| Scheduled-trigger fidelity | 99% of schedules fire within 60s of their slot | excludes deliberate overlap skips and periods where the service was down | +| Critical-task liveness | no critical `TaskHeartbeat` stale beyond three of its intervals | tasks marked critical in the registry | +| Data retention honesty | zero unreported drops | every `preloop.limit.dropped` increment must have a matching visible condition | + +Reference alerts: + +1. **Telemetry absent**: no `preloop.service.uptime` for five minutes when the service is expected. +2. **Dispatch stalled with capacity**: oldest claimable ready job exceeds the baseline threshold, + compatible idle/provisioning capacity exists, and pool preparation is false. +3. **Unclaimable queue**: ready unclaimable jobs persist beyond grace with no preparing/provisioning + capacity; distinguish no registered runner from label mismatch. +4. **Pool deficit**: desired exceeds idle+busy+building+provisioning while work is queued, or provision + failures repeat. +5. **Runner deaf/lease stale**: max poll/lease age approaches its configured timeout or reap events occur. +6. **Store failing**: consecutive failures or error rate persists; restart would risk losing live state. +7. **GitHub terminal check pending**: a terminal run lacks successful check propagation beyond 30s. +8. **Debug session stale**: a session remains active beyond its configured/operator-approved lifetime. +9. **Exporter failing**: OTLP was configured but has not succeeded and local drop/failure counts rise. +10. **VM host OOM event**: any increase in host-cgroup `oom_kill`; identify the VM/job from the + corresponding structured event. +11. **VM memory pressure**: sustained aggregate or top-consumer host memory above 90% of its measured + limit while busy; baseline before paging because lazy/ballooned mappings differ by platform. +12. **VM CPU throttling**: sustained throttled-period/time ratio while work is queued or running, not + a one-sample burst. +13. **VM disk pressure**: VM-state filesystem has both low percentage and low absolute free space; + filesystem free bytes, not summed CoW allocation, is authoritative. +14. **VM sampler stale/unavailable**: active owned VMs exist but the supported fast source is stale for + three intervals or errors persist. +15. **Background task dead**: `preloop.task.heartbeat.age` for any critical task exceeds three of its + intervals, or `preloop.task.exited{outcome!="clean"}` increments. This is the generic replacement + for writing one bespoke alert per loop. +16. **Schedule did not fire**: a registered schedule's slot passed with no `preloop.scheduler.fire` + increment, or `fire.delay` p99 exceeds the objective. Page separately from dispatch latency: a + cron that never fires produces no queued job and therefore trips no queue alert. +17. **Concurrency overflow cancelling users' jobs**: `preloop.concurrency.decision{action="cancel_arrival"}` + increments. This is policy working as designed, so warn rather than page, but it must be visible — + the user sees an unexplained cancellation. +18. **Data being dropped**: any `preloop.limit.dropped` or `preloop.limit.rejected` increase. Page on + `MAX_SESSION_AUDIT` eviction specifically; warn on the rest. +19. **GitHub budget exhaustion**: `rate_limit.remaining / rate_limit.limit` below 10%, or + `github.token.expires_in` below twice the refresh interval. Both precede a total integration + outage by minutes and are currently invisible. +20. **Storage capacity**: `preloop.storage.fs.available{mount="state_dir"}` low in both ratio and + absolute terms, or a single `preloop.storage.bytes` component growing monotonically across a full + GC interval. +21. **VM unreachable**: any VM in `runtime_state=unreachable` for more than one slow-sample interval. + A wedged golden silently poisons every fork taken from it. + +Do not page on workflow failure rate. User code fails legitimately. VM-attributable host +CPU/memory/disk pressure is part of the built-in application telemetry minimum. Whole-node load, +unrelated processes, network interfaces, kernel health, and hardware remain optional collector +signals. + +## OpenObserve evaluation + +### Verdict + +Use OpenObserve as the documented, optional “batteries available” backend. Do not make it a runtime +dependency, do not embed/link it, and do not shape Preloop signals around OpenObserve-specific fields +or APIs. + +| Criterion | Assessment | Decision | +|---|---|---| +| Minimal local deployment | Good: one native binary or container in single-node mode | Provide opt-in pinned compose/binary instructions, never auto-start it | +| Unified signals | Good: OTLP/HTTP and OTLP/gRPC ingest logs, metrics, and traces | Preloop implements OTLP/HTTP first to avoid tonic/gRPC dependency | +| Dashboards/alerts | Good: dashboards plus scheduled/realtime/composite standard alerts | Ship importable reference assets after signal names stabilize | +| Storage | Good for small installs, but it is another stateful data volume | Short default retention, persistent volume, backup guidance | +| HA | Poor fit for “minimal”: Kubernetes, object storage, PostgreSQL, NATS, and five roles | Do not ship an HA profile; link upstream docs | +| Security | Basic private deployment is usable; SSO, RBAC, and audit trail are enterprise features | Bind UI to loopback/private network and front it with operator auth if shared | +| Licensing | OSS repository is AGPL-3.0 | Keep it a separate process; do not vendor or relicense; legal review before distributing assets/binary bundles | +| Failure isolation | Co-hosting can compete with runner VMs for CPU/RAM/disk | Opt-in, resource-capped; prefer a separate host/volume for durable self-hosting | + +OpenObserve's own documentation says single-node local mode uses SQLite and local disk (or object +storage), while HA requires Kubernetes/Helm, object storage, PostgreSQL, NATS, and Router, Ingester, +Compactor, Querier, and Scheduler roles. Its storage guide says losing SQLite metadata makes the +installation inoperable. The optional profile must persist and back up both metadata and stream data; +for stronger durability, use object storage and a separately protected metadata store according to +upstream guidance. + +Do not put the OpenObserve UI on the public webhook origin. The OSS/enterprise feature split makes a +private network or an external auth proxy the conservative default. + +### Deployment profiles + +1. **Default local**: no backend, pretty logs on a TTY, JSON/compact logs when noninteractive, + authenticated `/metrics`, direct status/health. Zero external process and zero export network. +2. **Local enhanced**: one OpenObserve binary/container bound to loopback with persistent local volume, + short retention, resource caps, and direct OTLP/HTTP. Intended for debugging and baselining. +3. **Self-hosted single node**: Preloop and OpenObserve may share a host only at measured low volume; + use separate persistent volumes, explicit CPU/memory/disk budgets, private networking, backups, and + preferably object storage for OpenObserve stream data. +4. **Existing observability estate**: point the same OTLP output at the operator's backend. No + OpenObserve assets required. +5. **Advanced host telemetry**: optional OpenTelemetry Collector receives Preloop OTLP and adds + whole-node CPU/memory/filesystem/network/kernel metrics unrelated to a specific Preloop VM, then + forwards. Built-in VM-attributable metrics do not require the collector. +6. **OpenObserve HA**: operator-owned upstream deployment only. Preloop documents compatibility but + does not provision or support its dependencies. + +### Primary sources + +- Architecture and deployment modes: +- Single-node binary/container quickstart: +- OTLP logs/metrics/traces over HTTP and gRPC: + +- Trace endpoint: +- Prometheus remote-write support: +- Dashboards: +- Alerts: +- Storage and SQLite metadata warning: + +- Enterprise-only SSO/RBAC/audit features: +- OpenObserve repository license: +- OTel Rust exporter guidance (Collector recommended for larger production topologies): + +- SmolVM minimum-version (`v1.8.1`, `versions.toml:8`) JSON status contract — `machine_status_json` + at `src/cli/vm_common.rs:2360` and `status_vm_json` at `:2416`, shared with `machine ls --json`: + +- SmolVM `machine status --json` / `machine ls --json` flags and the public `machine data-dir` + command (`src/cli/machine.rs:365, 3783, 3857, 4290`): + +- SmolVM vsock state probe defining the `Unreachable` state (`src/agent/state_probe.rs:6-76`): + +- SmolVM per-VMM `vm-` cgroup-v2 leaf creation (`src/process.rs:375-393`, `place_in_cgroup`), + the upstream basis for Preloop's Linux VM metrics: + +- `sysinfo` 0.39.6 process API (`memory`, `start_time`, `cpu_usage`, and + `accumulated_cpu_time`) and feature controls: + , + + +## Configuration contract + +Add only one Preloop-specific setting: + +- `PRELOOP_LOG_FORMAT=auto|pretty|json`; `auto` means human-readable on a TTY and structured JSON when + noninteractive, without ANSI in journald/files. + +Honor standard variables rather than inventing backend-specific ones: + +- `RUST_LOG` +- `OTEL_SERVICE_NAME` +- `OTEL_RESOURCE_ATTRIBUTES` +- `OTEL_EXPORTER_OTLP_ENDPOINT` and signal-specific endpoint variants +- `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` +- `OTEL_EXPORTER_OTLP_HEADERS` and signal-specific header variants +- `OTEL_EXPORTER_OTLP_TIMEOUT` +- `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, `OTEL_LOGS_EXPORTER` +- `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` +- standard batch/log/metric interval variables supported by the selected OTel Rust release + +Export is disabled unless an OTLP endpoint is explicitly present. `none` disables the corresponding +signal. Reject unsupported protocol values from the telemetry pipeline with a sanitized status +condition while continuing the control plane. Never print endpoints containing userinfo/query data or +any header values. + +Absent-means-disabled is a documented deviation from the OTel default of `http://localhost:4318` +(invariant 4). State it in `docs/observability.md` next to the variable list, because an operator who +knows the spec will otherwise assume a local collector is being contacted. + +For OpenObserve, docs should show placeholder-only examples using the generic base endpoint without a +trailing slash and signal-specific authorization/stream headers. Credentials belong in the protected +systemd environment/credential mechanism already documented by Preloop, not committed compose files. + +## Commands the executor will need + +| Purpose | Command | Expected on success | +|---|---|---| +| Format check | `cargo fmt --all --check` | exit 0 | +| Workspace check | `cargo check --locked --workspace` | exit 0 | +| Observability crate tests | `cargo test --locked -p preloop-observability` | all pass | +| Server observability tests | `cargo test --locked -p preloop-runner-server observability -- --nocapture` | all matching tests pass | +| Pool observability tests | `cargo test --locked -p preloop-orchestrator observability -- --nocapture` | all matching tests pass | +| VM telemetry tests | `cargo test --locked -p preloop-vm observability -- --nocapture` | cgroup/process/disk fixtures and capability cases pass | +| CLI status tests | `cargo test --locked -p preloop-cli status -- --nocapture` | all matching tests pass | +| Structural security rules | `just sg-scan-strict` | exit 0 | +| Full local gate | `just test-ci` | ends with `CI: all checks passed` | +| Official-server protocol check | `just conform-server-light` | no protocol diff | +| Real runner smoke | `just dogfood` | workflow reaches expected terminal success | +| OpenObserve integration | `just observability-openobserve-smoke` | one correlated log, metric, and trace query succeeds; assets import | + +All recipes above except `observability-openobserve-smoke` exist in the `justfile` at `673bdfa0` +(`test-ci` at line 55, `sg-scan-strict` at 74, `dogfood` at 79, `serve` at 94, `conform-server-light` +at 121). `observability-openobserve-smoke` is created by step 6. Note that `just test-ci` runs +`fmt-check clippy zizmor test`, so the zizmor workflow gate applies to any CI workflow this plan adds. + +Match existing conventions: `anyhow` at binary boundaries, `ApiError` in HTTP handlers, +`thiserror` in libraries, `Arc>` plus atomics/Notify for shared state, and +`SecretString::expose()` only at protocol boundaries. Never await or export while holding the global +state mutex. + +Toolchain and existing dependencies verified at `673bdfa0`: + +- `rust-toolchain.toml` pins channel `1.97`; workspace `rust-version = "1.97"`. +- `tracing = "0.1"`, `tracing-subscriber = "0.3"` with `["env-filter", "json"]` already present. +- `reqwest = "0.12"` with `["json", "rustls-tls", "stream"]` — reuse this exact client for OTLP + `http/protobuf` through `opentelemetry_http::HttpClient` rather than adding a second HTTP stack. +- No `opentelemetry*`, `prometheus`, or `sysinfo` dependency exists yet; all are net-new. +- The pinned versions named in step 2 were re-checked against crates.io on 2026-08-20 and are still + the current stable releases (`opentelemetry*` 0.32.x, `tracing-opentelemetry` 0.33.0, + `prometheus` 0.14.0, `sysinfo` 0.39.6). `opentelemetry-prometheus` 0.32.0 shipped the same day as + the core 0.32.0 release, so the historical lag that made that crate risky does not apply to this + release line. Re-verify before implementing; if it has fallen behind again, that is a STOP. + +## Scope + +**In scope**: + +- `Cargo.toml`, `Cargo.lock`, `versions.toml` (only if the SmolVM floor must move) +- new `crates/preloop-observability/**` +- `crates/preloop-cli/src/main.rs`, `crates/preloop-cli/src/server_install.rs`, its Cargo manifest/tests +- `crates/preloop-runner-server/src/{main.rs,lib.rs,bootstrap.rs,routes.rs,runs.rs,state.rs,store.rs,store_pg.rs,broker.rs,runner_lifecycle.rs,scheduler.rs,concurrency.rs,live_logs.rs,snapshots.rs,github.rs,github_app.rs,github_pr.rs,github_push.rs,dispatch_auth.rs,oidc.rs,actions.rs,remote_workflows.rs,debug_sessions.rs,cache_artifacts.rs,artifact_twirp.rs,results_twirp.rs,blob_store.rs,distributed_task.rs,openapi.rs,lib_tests.rs}` and Cargo manifest +- `crates/preloop-orchestrator/src/lib.rs` and its tests/Cargo manifest +- `crates/preloop-vm/src/lib.rs` and its tests/Cargo manifest +- `crates/preloop-runner/src/main.rs` only for safe local logging initialization +- `docs/{architecture.md,self-hosting.md,cli_reference.md}` — note the file is `cli_reference.md`; + there is no `docs/cli.md` — plus new `docs/observability.md` +- `contrib/openobserve/**` as optional, pinned deployment/dashboard/alert assets +- `scripts/openobserve-observability-smoke.sh`, `justfile` +- one structural rule under `rules/` preventing sensitive tracing fields, matching the ast-grep YAML + format of the three existing rules (`no-expose-in-loop.yml`, `no-inline-masking.yml`, + `no-raw-secret-replace.yml`) +- `CHANGELOG.md`, `plans/README.md` + +**Out of scope**: + +- any official runner protocol body/status/header change; +- exporting directly from guest runners; +- exporting workflow step stdout/stderr, annotations, environment, or secrets; +- changing protocol flow recording into general telemetry; +- installing or starting OpenObserve automatically from `preloop serve`; +- bundling or modifying the OpenObserve binary/image; +- OpenObserve HA provisioning, multi-server Preloop, or a distributed state bus; +- host eBPF, automatic kernel/container instrumentation, or a required OTel Collector; +- periodic commands inside guests or changes to the official runner for telemetry; +- guest-kernel memory/process/OOM/filesystem/network metrics until SmolVM provides a stable + out-of-band source; +- per-machine metric labels, or fabricated network/block-I/O zeroes where no stable source exists; +- retrying business operations because telemetry failed; +- a new web UI inside Preloop; the direct CLI/API and reference backend are the deliverables. + +## Git workflow + +- Branch: `advisor/002-observability-strategy` unless the operator supplies another branch. +- One commit/PR per implementation step below. Keep protocol instrumentation changes separate from + reference-backend assets. +- Match the repository's imperative commit-message style observed in current history. +- Do not push or open a PR unless instructed. + +## Implementation steps + +### Step 1: Freeze the signal/security contract and make existing logs safe + +Targets: + +- add `docs/observability.md` containing the architecture, cardinality rules, log classes, status + semantics, signal catalog, SLO definitions, deployment profiles, and runbook links from this plan; +- re-run the grep in "Logging is local, duplicated, and not export-ready" and fix **every** hit, not + just the anchors below — three of the original four moved between `84d92cfd` and `673bdfa0`, so a + fixed list rots. At `673bdfa0` the hits are `artifact_twirp.rs:94`, `results_twirp.rs:713`, + `blob_store.rs:63, 78, 95, 113, 121, 127, 131`, and `distributed_task.rs:327`; +- preserve useful fields: operation/kind, byte count, request ID, parsed terminal result, and duration; +- add a structural `sg` rule rejecting INFO/WARN/ERROR tracing fields named `token`, `authorization`, + `cookie`, `headers`, `body`, `payload`, or `signed_url` unless the source is the explicitly excluded + `recording.rs` conformance path; +- audit every non-DEBUG tracing macro in server/orchestrator code for raw URLs, query strings, + headers, bodies, secrets, and opaque Debug dumps; +- document flow capture as sensitive local data and verify its permissions at creation. + +Do not “redact” tokens by logging a prefix, suffix, length, or stable hash; those are still unnecessary +capability correlators. Log the operation and outcome instead. + +**Tests**: + +- behavioral logging test with sentinel token/body/header values captured by an in-memory subscriber; + assert no sentinel appears while safe operation fields remain; +- flow-capture test remains local and proves its file mode is 0600 on Unix; +- structural rule fixture accepts safe lifecycle fields and rejects a token/raw-body field. + +**Verify**: + +```sh +just sg-scan-strict +cargo test --locked -p preloop-runner-server observability_log_safety -- --nocapture +``` + +Expected: exit 0; sentinels absent; flow capture behavior unchanged. + +### Step 2: Add the observability crate and unify process initialization + +Targets: + +- add workspace-member `crates/preloop-observability` with the API described above; +- pin the current compatible release line: `opentelemetry`, `opentelemetry_sdk`, + `opentelemetry-otlp`, `opentelemetry-prometheus`, `opentelemetry-appender-tracing`, and + `opentelemetry-semantic-conventions` at `0.32`, `tracing-opentelemetry` at `0.33`, and + `prometheus` at `0.14`; +- disable default exporter features; enable OTLP `http-proto` plus `trace`, `metrics`, and `logs` + explicitly (the `http-proto` feature alone does not enable logs), using the existing + rustls/reqwest stack through an `opentelemetry_http::HttpClient` adapter rather than pulling the + tonic stack or a second reqwest major/minor line; +- provide no-op, Prometheus-only, and Prometheus+OTLP initialization paths; +- add the `TaskHeartbeat` registry and `LimitRegistry` described in "New crate"; both must be usable + from a no-op handle so `preloop-orchestrator` and `preloop-vm` can register without an SDK; +- install stderr fmt/JSON, OTel trace layer, and OTel log bridge once in `preloop` and standalone + `preloop-server`; initialize the guest Rust runner with stderr only; +- parse `PRELOOP_LOG_FORMAT`; preserve `RUST_LOG=info` fallback and give the standalone server the + same fallback the CLI has — `preloop-runner-server/src/main.rs:78-80` currently uses + `EnvFilter::from_default_env()`, so an unset `RUST_LOG` silently disables its logging; +- create OTel resources and per-process instance ID; +- add bounded queues/timeouts and a two-second explicit shutdown path around command completion; +- sanitize all init/export errors and expose telemetry health through the shared handle. + +Do not make OTel globals the test seam. Constructors accept an explicit handle; tests use no-op or +recording exporters with scoped subscribers. + +**Tests**: + +- no endpoint/env means no DNS/socket attempt and no exporter worker; +- malformed/down endpoint does not fail initialization or delay a synthetic request; +- queue capacity and shutdown timeout are bounded; +- pretty/JSON/auto formats preserve structured fields and omit ANSI when noninteractive; +- config `Debug` never reveals OTLP headers or credential-bearing endpoint components; +- one event inside a span reaches the recording log exporter with trace/span correlation. +- a registered heartbeat that stops beating becomes stale and a dropped `HeartbeatHandle` + deregisters; a registry with a critical stale entry reports not-ready; +- `LimitRegistry` counts drops and rejects separately and reports registered limits with zero counts. + +**Verify**: + +```sh +cargo test --locked -p preloop-observability +cargo check --locked -p preloop-cli -p preloop-runner-server -p preloop-runner +``` + +Expected: all tests pass and all three binaries compile. + +### Step 3: Add truthful liveness, readiness, aggregate status, and CLI output + +Targets: + +- add `status.rs` in the observability crate for neutral DTOs/handles and in server for state sampling; +- add a five-second sampler in `bootstrap.rs` with a heartbeat and bounded five-exemplar conditions; +- register a `TaskHeartbeat` for every long-lived task in the fifteen-task inventory, without changing + any task's cadence or behavior; mark the state sampler, reaper, scheduler scan, and (Postgres only) + the store connection task critical; +- wire the consolidated `PoolStatus` handle through `ServerConfig`, `RunnerPoolConfig`, and CLI + construction, **removing** the four ad-hoc handles (`pending_jobs`, `preparing_signal`, + `next_job_runs_on`, `pending_registrations`) rather than adding a fifth beside them; +- register every cap in the limits table with `LimitRegistry` and report them in status; +- add `/readyz` and authenticated `/api/v1/status`; make authenticated `/metrics` available from the + provider registry; +- keep `/healthz` lock/dependency-free and return 503 during shutdown; +- update OpenAPI for all operator routes and native bearer requirements; +- replace `Command::Status` with `Status(StatusArgs)`, implement exact JSON plus sectional human output, + and preserve recent runs; copy the `#[arg(long)] json: bool` shape from `PlanArgs` + (`preloop-cli/src/main.rs:706-707`); +- change `wait_for_engine_socket` (`preloop-cli/src/main.rs:1458-1475`) to probe `/readyz` and to + surface the last reason code on timeout. + +The sampler computes claimability using existing runner label matching. It distinguishes: + +- claimable now; +- temporarily unclaimable because pool is preparing/provisioning; +- no registered runner; +- registered runners exist but labels do not match. + +It must not call the external backend and must remain responsive when OTLP is down. + +**Tests** (follow the existing router/auth test pattern in `lib_tests.rs`, which builds state with +`AppState::new(temp.path().to_path_buf()).await.unwrap()` and the `app(...)` helper): + +- health is public/shallow; ready returns 503 with the stale task's registry name as reason, and a + stale **non-critical** task does not affect readiness; +- status and metrics reject missing/invalid native bearer and accept valid bearer; +- every queue class and pool mode appears correctly; +- concurrency, scheduler, storage, limits, and tasks blocks render with correct values, and `limits` + includes registered caps whose counters are zero; +- a `queue: max` overflow at `QUEUE_MAX_PENDING` produces the `concurrency_queue_overflow` condition; +- claimability distinguishes absence, label mismatch, preparing, provisioning, and compatible runner; +- VM fleet totals, source capabilities, sample age, and the deterministic five-entry top-consumer + bound render in status/CLI without creating metric labels; +- snapshot fallback remains available when a deliberately held state lock prevents a fresh sample; +- JSON schema version is fixed; CLI JSON is byte-for-byte valid endpoint JSON; +- human output names an actionable cause for a stuck job. + +**Verify**: + +```sh +cargo test --locked -p preloop-runner-server status_observability -- --nocapture +cargo test --locked -p preloop-cli status -- --nocapture +``` + +Expected: all matching tests pass; unauthorized cases return 401; readiness cases return 200/503 as specified. + +### Step 4: Instrument HTTP, runs/jobs, dispatch, runners, and the store + +Targets: + +- replace default `TraceLayer` with custom matched-route/surface instrumentation; +- classify routes into finite surfaces covering all thirteen path families actually served: + `native` (`/api/v1`), `runner` (`/_apis`, `/runner`, `/runner/server`), `broker` (`/broker`), + `results` (`/twirp`, `/twirp-blob`), `webhook`, `git` (`/snapshots`, `/repos`), `oidc` + (`/oidc`, `/.well-known`), `live_logs` (`/ws/live-logs`), `public` (`/healthz`, `/readyz`), + `test` (`/internal/test`), and `unknown`; `unknown` is a constant label, never a raw path; +- exclude the `live_logs` surface from `http.server.request.duration` and instrument it with + `preloop.livelog.connections` plus a close-reason counter instead; +- create exact HTTP metrics and safe spans without headers/body/query; +- instrument run/job terminal transitions exactly once using central transition/event boundaries; +- record queue wait at claim or terminal-unclaimed failure; +- instrument poll outcomes, session transitions, renew errors, lease expiry, no-matching-runner, + deaf-runner reaping, and restart-orphan reconciliation; +- wrap the private `Store` trait in `store.rs` with `InstrumentedStore`, including all methods and + backend/outcome; preserve every return/error and best-effort persistence rule. Wrap the + `Arc` the factory at `store.rs:261-267` returns; because the trait is `#[async_trait]` + and already consumed as `Arc` (`state.rs:360`), no call site changes; +- instrument the Postgres connection task at `store_pg.rs:95-103` with a heartbeat plus + `preloop.store.connection.up`, so connection death is visible before the first failed write; +- instrument concurrency-group decisions at `concurrency.rs::apply_queue_mode` — every + `park`/`cancel_pending`/`cancel_arrival`/`admit`, with `cancel_arrival` also recording a + `LimitRegistry` reject against `QUEUE_MAX_PENDING`; +- instrument the scheduler scan: fire, late fire, overlap skip, and registered-schedule count; +- feed gauges only from cached sampler values. + +Do not count an emitted duplicate status event as a second completion. Add transition guards or record +at the state mutation that proves old-state to terminal-state movement. + +**Tests**: + +- route template is `/api/v1/runs/:run_id`, never a concrete ID/query; +- 1,000 unique HTTP/run/job/runner IDs do not increase metric series beyond the fixed bound; +- a claim emits one wait observation and one claim outcome; +- successful completion, timeout, no runner, lease expiry, deaf runner, and startup orphan each emit + exactly one bounded terminal reason; +- each `Store` method records one duration/outcome while preserving success/error values; +- all seven `Store` methods are covered — a new trait method without instrumentation must fail a test, + not pass silently; +- killing the Postgres connection task flips `store.connection.up` and emits `store.connection.lost`; +- `queue: single`, `queue: max` under the cap, and `queue: max` at the cap each emit exactly one + bounded decision outcome, and the overflow case increments the limit reject counter; +- slow/failing recording exporter does not measurably serialize handler completion; use paused time or + synchronization, not a flaky wall-clock threshold. + +**Verify**: + +```sh +cargo test --locked -p preloop-runner-server observability -- --nocapture +just conform-server-light +``` + +Expected: tests pass and official runner flow comparison has no diff. + +### Step 5: Instrument pool, VM resources, GitHub, webhook, cache/artifacts, and debug sessions + +Targets: + +- move pool idle/building/provisioning/busy/paused/preparing counters behind the consolidated + `PoolStatus` handle from step 3 and delete the four ad-hoc `Option>` fields; +- update state through RAII guards so cancellation/error cannot leak a count; +- instrument golden/artifact preparation, provision phases, registration, assignment, pause/resume, + delete/replacement, and supervisor exit with stable outcomes/reasons; +- replace the substring matching in `SmolVmProvider::status` (`preloop-vm/src/lib.rs:1018-1043`) with + typed inspection: `machine status --json` for one machine on a lifecycle path, and extend the + existing `machine ls --json` parser in `SmolVmProvider::list` + (`crates/preloop-vm/src/lib.rs:1045-1060`) into the + fleet-wide `VmRuntimeInfo` source used by the slow sampler. Resolve `machine data-dir` only when a + disk path is needed; +- add `MachineState::Unreachable` and map SmolVM's vsock-probed `Unreachable` state to it end to end + (metric attribute, status condition `vm_unreachable`, `vm.unreachable` event). Audit every existing + `MachineState` match arm — adding a variant is a breaking change for the pool's decision logic, and + treating `Unreachable` as `Unknown` would silently keep forking from a wedged golden; +- add workspace dependency + `sysinfo = { version = "0.39.6", default-features = false, features = ["system"] }` to + `preloop-vm`; retain one process snapshot and refresh only registered VMM PIDs rather than scanning + every process or enabling unrelated component, disk, network, and user collectors; +- add the shared VM registry plus one five-second host resource sampler and one 60-second disk sampler; + register runner/golden role and assignment on create/fork/adopt, preserve paused debug VMs, and + unregister only after confirmed deletion; +- on Linux, read the validated `vm-` cgroup-v2 CPU/memory/PID files under the root + `init_vm_cgroup_delegation` (`preloop-vm/src/lib.rs:1360` ff.) already delegates. The leaf is created + by SmolVM's supervisor, not by Preloop, so a missing leaf degrades to the process fallback and + records `capability=false`; it is never an error. On macOS/cgroup-unavailable hosts, read validated + process CPU/RSS; on every platform, sample known sparse-file allocated blocks and + state-filesystem free bytes; +- compute counter deltas and CPU-core rate without attributing a reused PID; carry unsupported/stale + fields as unavailable rather than zero; +- aggregate VM metrics by bounded role/activity/source and publish five authenticated top consumers; +- emit rate-limited pressure/recovery events for host memory, CPU throttling, host OOM, sparse disk, + and sampler health; +- instrument GitHub App token/check/API calls at centralized request boundaries, never token values; +- parse `x-ratelimit-limit`/`x-ratelimit-remaining`/`x-ratelimit-reset` from every GitHub response and + publish the budget gauges; track installation-token expiry and the `dispatch_token_cache` + /`dispatch_actor_cache`/`action_sha_cache` hit rates (`dispatch_auth.rs:46, 49`); +- track pending terminal check updates and propagation delay for status/SLOs; +- instrument webhook processing/dedup with event class and outcome, delivery ID only in logs/traces; +- instrument cache/artifact operations and bytes without key/name/token labels; +- instrument snapshot/git object serving (`snapshots.rs`), including `ObjectCache` hit/miss/evict and + the GC pass at `snapshots.rs:1896`; +- add the storage sampler: per-component bytes for database, cache, artifacts, run logs, snapshots, + and VM images, plus `statvfs` free bytes for the state-dir and SmolVM-data mounts. Run it on the + 60-second cadence; a recursive directory walk must never run on the fast path or under a state lock; +- instrument debug session create/pause/resume/detach/close/expire/crash and age; +- register the four `debug_sessions.rs` ring caps with `LimitRegistry` and emit `debug.audit.evicted` + when `MAX_SESSION_AUDIT` evicts; +- register the `live_logs.rs` per-job cap and count both tail-drops and oversized-batch rejects; +- add short synchronous spans around these operations; no workflow-lifetime span. + +Pool status is authoritative from transitions, while the periodic server sampler merges it with queue +and runner state. Avoid double-maintaining separate pool counters solely for metrics. + +**Tests**: + +- each pool state transition balances under success, error, cancellation, and paused-debug paths; +- SmolVM JSON parsing covers every state **including `Unreachable`**, optional PID/resource field, + malformed output, missing machine, and a fixture captured from the supported floor `v1.8.1`; +- `machine ls --json` fixture yields one `VmRuntimeInfo` per machine and the fast sampler spawns zero + subprocesses; +- an `Unreachable` golden is not used as a fork source; +- cgroup parser fixtures cover CPU units/deltas, throttle counters, `max` limits, memory events, PID + events, missing controller files, permission errors, counter reset, process exit, and PID reuse; +- a missing `vm-` leaf falls back to the process source and reports the capability false rather + than erroring or emitting zero; +- process fallback tests use injected process snapshots and prove unsupported cgroup-only values are + absent, not zero; +- sparse-file tests compare allocated blocks rather than logical length, count shared goldens once, + enforce the slow cadence, and treat filesystem free space as authoritative; +- cancellation, pause, adoption, replacement, and deletion keep the VM registry balanced and never + sample a deleted/reused process; +- 1,000 ephemeral machine names produce a constant metric series set while status retains at most five + deterministic top consumers; +- repeated provision failure produces status condition and metrics but does not create a log storm; +- GitHub/check failure uses bounded outcome and retains safe correlation IDs; +- rate-limit headers populate the budget gauges, and a response missing them leaves the previous + values with an explicit `observed_at`, never zero; +- the storage sampler reports per-component bytes and free space from a temp-dir fixture and does not + block the fast sampler; +- live-log tail-drop and debug-session ring eviction each increment `preloop.limit.dropped` with the + correct constant name; +- cache/artifact unique keys and webhook delivery IDs never become metric labels; +- debug session expiry/crash produces the correct terminal reason. + +**Verify**: + +```sh +cargo test --locked -p preloop-vm observability -- --nocapture +cargo test --locked -p preloop-orchestrator observability -- --nocapture +cargo test --locked -p preloop-runner-server observability -- --nocapture +``` + +Expected: all matching tests pass with balanced gauges and bounded labels. + +### Step 6: Add the optional OpenObserve reference profile and assets + +Targets: + +- add `contrib/openobserve/compose.yml` pinned to a reviewed immutable OpenObserve version/digest; +- bind the UI/API to loopback by default, use a persistent data volume, healthcheck, explicit CPU/memory + limits, short retention, and placeholder-only credentials sourced outside version control; +- do not vendor the binary/image; include upstream license/source notices and a legal-review note; +- add six importable dashboards: overview, scheduling/queue (including cron schedules and + concurrency-group contention), runners/pool, VM host resources, dependencies (GitHub budget, store, + storage capacity) and telemetry, and a "limits and background tasks" board showing every registered + cap and heartbeat; +- add alert definitions matching the catalog above, with baseline placeholders where thresholds require + measured data; +- add `docs/observability.md` setup for native binary, container, direct OTLP/HTTP, existing backend, + optional Collector, backup/retention, private access, and troubleshooting; +- add a smoke script/just recipe that starts the pinned profile, starts Preloop with sentinel-safe OTLP + config, performs health/status and a small workflow, queries OpenObserve for at least one log, + metric, and trace sharing expected resource/correlation fields, exercises CPU/memory/disk activity + in one VM, imports assets, and tears down. + +Before choosing resource limits, measure idle and ingest/query use on the target host. OpenObserve's +querier caching can consume substantial memory; do not guess a cap that starves the VM pool. Record the +measured default and explain how to change it. + +**Verify**: + +```sh +just observability-openobserve-smoke +``` + +Expected: the pinned single-node service becomes healthy; Preloop continues working; one safe log, +one domain metric, VM CPU/memory/disk series, and one trace are queryable; dashboard/alert assets +import; neither logs nor query results contain sentinel secrets. + +### Step 7: Baseline, ratify alerts, document the incident workflow, and close the loop + +Targets: + +- run local/self-hosted telemetry for two representative weeks or an agreed workload-equivalent soak; +- record series count, logs/day, sampled spans/day, exporter drops, VM sampler overhead, VM CPU/RSS + correlation, sparse/CoW disk-accounting behavior, OpenObserve CPU/RAM/disk growth, and query latency; +- tune histogram buckets, sampling, retention, and alert thresholds based on evidence without changing + signal names casually; +- document runbooks for queue stall, pool deficit, VM host memory/throttling/OOM/disk pressure, stale + VM sampling, deaf runner, store failure, GitHub check lag, telemetry failure, and OpenObserve + disk/metadata recovery; +- integrate direct diagnosis: every alert links first to `preloop status --json`, then dashboard/log/trace + queries, then existing `preloop debug` for a failed job; +- update architecture, self-hosting, CLI docs and changelog; +- run the complete compatibility and dogfood gates. + +**Verify**: + +```sh +just test-ci +just conform-server-light +just dogfood +``` + +Expected: all gates pass; dogfood completes with official-runner protocol behavior unchanged; the +status snapshot and reference dashboards explain every observed transition. + +## Test plan + +Permanent tests must defend observable contracts: + +1. **Disabled path**: no OTel endpoint creates no network work; status/metrics/logging remain usable. +2. **Failure isolation**: malformed endpoint, unavailable backend, slow exporter, full queue, and flush + timeout never alter API result or workflow state. +3. **Security**: sentinel tokens, headers, bodies, signed URLs, workflow output, and flow-capture bytes + never enter ordinary logs/traces/metrics. +4. **Cardinality**: 1,000 unique user/domain identifiers produce a constant bounded metric series set. +5. **HTTP semantics**: matched templates and finite surfaces only; query values absent. +6. **Status**: all queue/capacity/store/GitHub/debug/telemetry states and conditions, with auth and stale + snapshot behavior. +7. **Lifecycle exactness**: accepted/ready/claimed/completed/requeued/reaped/reconciled counters and + durations record once under success, error, retry, cancellation, and restart paths. +8. **Pool balance**: no gauge leak under every early return/cancellation. +9. **VM sampling**: minimum-version JSON fixtures, cgroup/process source fallback, PID reuse/counter + reset, missing capabilities, sparse allocation, filesystem pressure, cadence, registry lifecycle, + top-consumer bound, and sampler overhead. +10. **VM semantics**: tests and docs never present host RSS/PIDs/OOM as guest memory/processes/OOM and + never emit unsupported network/block-I/O metrics as zero. +11. **Shutdown**: exporters flush within the bound and process exits if backend is hung. +12. **Backend E2E**: pinned OpenObserve ingests/query-correlates all three signals, VM resource series, + and imports assets. +13. **Protocol fidelity**: committed conformance flows and real official runner remain byte/behavior + compatible. +14. **Limit honesty**: every constant registered with `LimitRegistry` has a test that drives it past + its cap and asserts the drop/reject counter moved. A cap with no such test is an unproven claim. +15. **Task liveness**: a task that panics, returns early, or stops beating is reported; a dropped + handle deregisters; a stale non-critical task does not fail readiness. +16. **Scheduler**: a schedule that fires, one skipped for overlap, and one whose scan task is dead + produce distinct, correct signals. +17. **GitHub budget**: rate-limit headers, a response lacking them, and an expiring installation token + each produce the right gauge/condition without exporting credential material. + +Avoid source-text unit tests, sleeps, and exact wall-clock assertions. Use in-memory exporters, +paused Tokio time, barriers, real router requests, and Prometheus registry gathering. + +## Done criteria + +All must hold: + +- [ ] `preloop status` explains queue, capacity, runner freshness, store, GitHub, debug, and telemetry state. +- [ ] `preloop status --json` returns the versioned authenticated snapshot with no prose. +- [ ] `/healthz`, `/readyz`, `/api/v1/status`, and `/metrics` have the specified auth/semantics. +- [ ] No backend is contacted when OTLP endpoint configuration is absent. +- [ ] OTLP failure cannot block or fail a request/workflow; queues and shutdown are bounded. +- [ ] Existing known capability-token/raw-body logs are removed and the structural guard passes. +- [ ] Workflow output and flow recordings are absent from ordinary telemetry. +- [ ] Metric labels pass the 1,000-identifier cardinality/sentinel test. +- [ ] Pool, queue, runner, store, GitHub, cache/artifact, and debug lifecycle signals exist and are exact. +- [ ] Every cap in the limits table is registered, reported in `/api/v1/status` even at zero, and + counted when exceeded; no code path discards data without a counter. +- [ ] Every long-lived task in the fifteen-task inventory registers a heartbeat; the four critical + tasks gate `/readyz` and the rest are visible in status. +- [ ] Scheduled workflows report fire, late fire, overlap skip, and scan-task liveness. +- [ ] Concurrency-group contention and `queue: max` overflow cancellation are visible with a hashed — + never raw — group key. +- [ ] GitHub rate-limit budget, installation-token expiry, and auth-cache hit rate are reported. +- [ ] Per-component persistent storage bytes and state-dir free space are reported; Plan 001 emits + through `preloop.storage.*` rather than a parallel family. +- [ ] The four ad-hoc `RunnerPoolConfig` shared handles are gone, replaced by one `PoolStatus`. +- [ ] `MachineState::Unreachable` exists, is honored by pool fork decisions, and surfaces as a condition. +- [ ] Active Preloop VMs expose bounded aggregate configured capacity, host CPU, host memory, + Linux throttling/events/PIDs when supported, sparse allocated disk, filesystem free space, and + sampler freshness without running a guest command. +- [ ] `/api/v1/status` reports VM capability gaps and at most five correlated top consumers; unsupported + metrics are absent rather than zero. +- [ ] OpenObserve remains optional, pinned, private-by-default, resource-capped, and separately licensed. +- [ ] Six dashboards, including VM host resources and limits/tasks, and initial alerts import and query + real emitted fields. +- [ ] `cargo fmt --all --check`, `just sg-scan-strict`, `just test-ci`, `just conform-server-light`, + `just dogfood`, and `just observability-openobserve-smoke` pass. +- [ ] Docs include signal definitions, deployment profiles, retention/backup/security, SLOs, alerts, and + incident runbooks. +- [ ] `plans/README.md` status row is updated. + +## STOP conditions + +Stop and report; do not improvise if: + +- instrumentation requires changing an official runner wire response or guest runner behavior; +- OTel Rust crate versions cannot provide multiple metric readers (Prometheus plus OTLP) without a + second independently updated instrument set; +- the selected OTel logs bridge cannot preserve trace/span correlation or bounded nonblocking export; +- an exporter can perform network I/O on the handler thread/runtime path rather than a batch worker; +- OpenObserve asset format/API is unstable across the pinned version and cannot be smoke-tested; +- legal review rejects distributing the compose/dashboard/alert assets under the repository license; +- status sampling must hold `InnerState` while awaiting I/O or copies unbounded payload/log data; +- VM sampling requires one SmolVM subprocess per VM per scrape, periodic guest execution, or an + unsupported private SmolVM storage/database layout rather than the public CLI/cgroup/process + boundaries; +- the supported SmolVM floor (`versions.toml` `smolvm_min_version`, currently `1.8.1`) lacks the + `machine status --json` / `machine ls --json` / `machine data-dir` contract, stops sharing one + `machine_status_json` shape between the two JSON outputs, or changes its field semantics; +- `opentelemetry-prometheus` has fallen behind the `opentelemetry` release line again, forcing either + a downgrade of the whole OTel stack or a second independently versioned instrument set; +- adding `MachineState::Unreachable` cannot be done without changing pool scheduling behavior — that + is a correctness fix, not an observability change, and belongs in its own reviewed PR; +- PID start identity cannot be validated on a supported host, making cross-process attribution unsafe; +- a metric requires a user-controlled/unbounded label to answer the intended question; +- registering a cap or heartbeat would require changing the behavior of the code it observes; +- the current code no longer has the named centralized lifecycle/store/pool boundaries; +- conformance or dogfood changes after instrumentation, even if unit tests pass. + +## Maintenance notes + +- Signal names, units, label sets, status schema, condition codes, and termination reasons are public + operational APIs. Review changes like wire changes; additive first, documented migration when not. +- Every new queue/state/runner failure path must update status, metrics, structured lifecycle logs, and + tests together. Do not add a dashboard-only derivation for state the server already knows. +- Every new metric label needs a bounded-cardinality review and the unique-ID test. +- Every new cap, ring buffer, or truncation point must be registered with `LimitRegistry` in the same + PR that introduces it. Every new long-lived `tokio::spawn` must register a `TaskHeartbeat`. Both are + review checklist items, not follow-ups — this plan exists because fifteen tasks and nine caps + accumulated without either. +- Every new tracing field needs a secret/PII review. `Debug` on request/config/domain structs is unsafe + unless the type has an explicit redacted implementation. +- Anchors in this plan rot fast: the churn from `84d92cfd` to `673bdfa0` moved three of four named + logging sites, the `TraceLayer` line, the `preloop status` implementation, the CLI subscriber, and + every `broker.rs` range. Prefer the greps and symbol names over the line numbers, and re-run the + drift check before each step. +- VM metric names/documentation must retain the `host` distinction: cgroup/process RSS, PID, OOM, and + throttling are observations of the VMM on the host, not guest-kernel counters. +- A new SmolVM release must run the JSON inspection and cgroup/process telemetry fixtures in the + existing runtime verification workflow before the verified version moves. +- Network/block-I/O or guest OS telemetry requires a separate source/semantics review; absence is more + correct than a portable-looking zero from one platform. +- Keep OpenObserve versions/digests and assets tested together. Preloop's OTLP contract must continue to + work with other vendors. +- If Preloop later becomes multi-server, re-design gauges and status around a shared authoritative bus; + `service.instance.id` is not a substitute for distributed coordination. +- If workflow-log export is ever proposed, treat it as a separate security/retention product decision: + workflow output is high-volume, user-controlled, and may contain masked or unrecognized secrets. diff --git a/plans/README.md b/plans/README.md new file mode 100644 index 00000000..a4445bfd --- /dev/null +++ b/plans/README.md @@ -0,0 +1,51 @@ +# Implementation Plans + +Generated and reconciled by the improve skill on 2026-08-17. Plan 002 was re-verified and revised +against live code at commit `673bdfa0` on 2026-08-20. Execute plans in the order selected by the +maintainer; the two current plans are independent. Each executor must read its plan fully, honor its +STOP conditions, run every verification gate, and update the corresponding status row. + +## Execution order and status + +| Plan | Title | Priority | Effort | Depends on | Status | +|---|---|---:|---:|---|---| +| [001](001-caching-performance-strategy.md) | Caching strategy for local and self-hosted CI | P1 | L | — | TODO | +| [002](002-observability-strategy.md) | Make Preloop observable without making a backend mandatory | P1 | L | — | TODO (revised at `673bdfa0`) | + +Status values: `TODO` | `IN PROGRESS` | `DONE` | `BLOCKED: ` | `REJECTED: `. + +## Dependency notes + +- Plans 001 and 002 can execute independently. +- When both touch cache/artifact operations, Plan 002 owns the stable observability contract and Plan + 001 must emit through that contract rather than introduce separate metric/log conventions. +- Plan 002 also owns the `preloop.storage.*` measurement contract. Plan 001 adds cache quotas and + eviction; it must report them through `preloop.storage.bytes` / `preloop.storage.gc` and register + any new cap with `LimitRegistry`, not invent a parallel family. +- Plan 002 is intentionally split into seven PR-sized steps. Its security/log-safety step must land + before OTLP log export is enabled. + +## Findings considered and rejected + +- **Make OpenObserve a required or embedded Preloop component**: rejected. Direct status, local logs, + and Prometheus metrics must work with no backend; OTLP keeps backend choice interchangeable. +- **Require an OpenTelemetry Collector for the minimum profile**: rejected. It adds another process; + direct bounded OTLP/HTTP is enough for low-volume application telemetry. Preloop collects + host-observed CPU, memory, throttling, PID, and sparse-disk metrics for VMs it owns; a Collector + remains optional for whole-node and unrelated-process metrics, buffering, sampling, or fan-out. +- **Export workflow step logs by default**: rejected. They are high-volume, user-controlled, and may + contain secrets; they remain in Preloop's existing run-log store. +- **Represent each workflow as one hours-long trace**: rejected. Workflows cross asynchronous requests, + sessions, and processes; use short operation traces plus structured run/job correlation fields. +- **Treat OpenObserve HA as the self-hosted default**: rejected. Its Kubernetes, object storage, + PostgreSQL, NATS, and multi-role topology conflicts with the minimal-dependency requirement. +- **Use observability to imply multi-server Preloop support**: rejected. In-memory state remains + authoritative per instance; shared database storage is only a restart source, not a shared bus. +- **Keep the first draft's two ad-hoc readiness heartbeats**: rejected on revision. Preloop runs + fifteen long-lived tasks; wiring two by hand and leaving thirteen silent reproduces the problem the + plan is meant to solve. One `TaskHeartbeat` registry with a critical subset replaces it. +- **Add a fifth shared handle to `RunnerPoolConfig`**: rejected on revision. Four ad-hoc + `Option>` channels already exist; the plan consolidates them into one `PoolStatus` instead of + extending the pattern. +- **Emit a duration histogram for `/ws/live-logs`**: rejected. A long-lived WebSocket would dominate + p99 and corrupt the availability SLI denominator; use a connection gauge and close-reason counter. From 2e91532a43fefba8009fad8af313332e0dc0bc3c Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 12:10:04 -0400 Subject: [PATCH 02/22] docs(plans): add observability strategy HTML with architecture diagram and mockups HTML companion to 002-observability-strategy.md (revised at 673bdfa0). Renders the three-layer architecture as an inline SVG, all operator surfaces as interactive tabbed previews (preloop status human/JSON, readyz 200/503, /metrics Prometheus exposition, structured log stream), and the six importable dashboards as panel grids with sparklines. Matches the 001 HTML precedent (b7597769) in typography and layout. Entire-Checkpoint: 01M0FYZK2WG8WMY3E63DXQ37EX --- plans/002-observability-strategy.html | 1725 +++++++++++++++++++++++++ 1 file changed, 1725 insertions(+) create mode 100644 plans/002-observability-strategy.html diff --git a/plans/002-observability-strategy.html b/plans/002-observability-strategy.html new file mode 100644 index 00000000..1aa95ced --- /dev/null +++ b/plans/002-observability-strategy.html @@ -0,0 +1,1725 @@ + + + + + +Plan 002 — Make Preloop Observable Without Making a Backend Mandatory + + + + + + +
+ + + +
+ +
+
Plan 002 · Observability · Architecture & Operations
+

Make Preloop observable without making a backend mandatory

+

Every important control-plane question answerable from one authenticated endpoint, with no sidecar. OpenTelemetry and an optional single-node OpenObserve layer on top — fail-open, by design.

+
+ P1 — In Review + Effort L · 7 PRs + Risk MED + Category direction · architecture · operations · security · DX +
+
+ Planned 84d92cfd · 2026-08-17 + Revised at 673bdfa0 · 2026-08-20 +
+
+ +
+ ⚠ Drift check — run first. Every file:line anchor and version pin below was re-verified at 673bdfa0. If the repository has moved, compare the excerpts with live code — a semantic mismatch is a STOP condition. +
git diff --stat 673bdfa0..HEAD -- Cargo.toml Cargo.lock crates/preloop-observability crates/preloop-cli crates/preloop-runner-server crates/preloop-orchestrator crates/preloop-vm crates/preloop-runner docs contrib/openobserve scripts rules justfile versions.toml CHANGELOG.md +
+ + +

Status

+ + + + + + + + +
FieldValue
PriorityP1
EffortL — seven independently reviewable PRs
RiskMED — HTTP/runner lifecycle are protocol-critical; telemetry must be fail-open
Depends onnone — independent of Plan 001
+
+
Dependency note
+ Plan 002 owns the stable observability contract. When Plan 001 touches cache/artifact operations it must emit through preloop.storage.* and register caps with LimitRegistry rather than inventing a parallel family. +
+ + +

Executive Decision

+

Build observability in three layers, in this order:

+
    +
  1. Zero-dependency operator diagnostics — truthful liveness/readiness, an authenticated aggregate status endpoint, preloop status --json, structured stderr/journald logs, and authenticated Prometheus text at /metrics, including host-observed resource use for Preloop-owned microVMs. No sidecar. Answers “why is this job not moving?” when no telemetry backend exists.
    2. +
  2. Vendor-neutral telemetry — OpenTelemetry metrics, logs, and short-lived traces exported through bounded OTLP/HTTP batches only when standard OTEL_* configuration is present. Export failure never rejects, delays, or cancels a workflow.
    3. +
  3. OpenObserve as an optional reference backend — a pinned, private single-node deployment plus importable dashboards and alerts. Preloop neither embeds nor requires OpenObserve. Any OTLP backend remains interchangeable.
    4. +
+
+
Product boundary
+ OpenObserve single-node is one binary/container, SQLite + local disk, OTLP ingest, dashboards and alerts. Its HA mode needs Kubernetes, object storage, Postgres, NATS, and five roles — not minimal. Keep the boundary at OTLP and Prometheus. The highest-priority deliverable is not a dashboard. It is preloop status. +
+ + +

Why This Matters

+

Today Preloop can accept work while the pool repeatedly fails to provision, can leave an operator unable to distinguish “no compatible runner” from “runner has stopped polling,” and can report a healthy process while critical background behavior is degraded. The only built-in aggregate view is a recent-runs table.

+
+ + + + + + + + + + +
Operator questionAnswered today?After this plan
Is the critical event loop making progress?No — process can be wedged while /healthz returns 200/readyz + TaskHeartbeat registry
Why is this job not claimed?No/api/v1/status jobs + concurrency + pool blocks
Are runners polling / renewing leases?No visibilityrunner poll/lease gauges + reaper heartbeat
Is VM CPU/memory/disk limiting capacity?No surfaceVM fleet sample w/ host cgroup/process
Are store / GitHub writes succeeding?Silent divergent DBstore + GitHub budget gauges
Is telemetry itself healthy?No self-healthpreloop.telemetry.export + status
Is data being silently dropped?9 caps, zero counterspreloop.limit.* + status limits array
+ + +

Current State

+ +

Logging is local, duplicated, and not export-ready

+

crates/preloop-cli/src/main.rs:743-745 initializes a plain formatting subscriber; the standalone server (preloop-runner-server/src/main.rs:78-80, no info fallback) and the Rust runner (preloop-runner/src/main.rs:17-21) each do the same. Three binaries, three slightly different filter defaults, no JSON selection, no OTLP pipeline.

+
+
Security gate — four unsafe sites at 673bdfa0
+
    +
  • artifact_twirp.rs:94 — info!(token, …) capability token
    • +
  • results_twirp.rs:713 — info!(token, …) capability token
    • +
  • blob_store.rs:63,78,95,113,121,127,131 — seven warn!/info!(kind, token) sites
    • +
  • distributed_task.rs:327 — info!(?body, …) full PATCH JSON body
    • +
  • recording.rs:1-90 — deliberate full-header/body capture (excluded from export, mode 0600)
    • +
+

Three of these moved between 84d92cfd and 673bdfa0. Re-run the scan: grep -rnE '(info|warn|error)!\(' crates/preloop-runner-server/src | grep -E '\b(token|authorization|cookie|headers|body|payload|signed_url)\b'

+
+ +

Health and status do not diagnose the control plane

+

crates/preloop-runner-server/src/runs.rs:4-9 always returns ok: true. The router exposes only GET /healthz (routes.rs:256) and installs TraceLayer::new_for_http() at routes.rs:805 — whose default span leaks raw URIs. No /readyz, no /metrics, no Prometheus dependency.

+

preloop status at crates/preloop-cli/src/main.rs:2831 calls only GET /api/v1/runs?limit=20. wait_for_engine_socket (main.rs:1458-1475) probes http://localhost/healthz over 30 s, treating “accepts connections” as “usable”.

+ +

The state needed for useful diagnostics already exists

+

AppState (state.rs:358-503) and InnerState (state.rs:1067-1153) already contain ready/dependency/concurrency/expansion queues, runners/sessions/claims/leases, pool reservations, debug sessions, plus the GitHub App surface added since the first draft. AppState::emit (state.rs:819, lock released at 827-832) releases the mutex before persistence — in-memory is authoritative.

+ +

Six blind spots the first draft missed

+

Re-auditing at 673bdfa0 found six classes of state with no surface at all. Each can independently make Preloop behave incorrectly with no visible signal.

+ +

1 · Bounded buffers and hard limits drop data silently

+
+ + + + + + + + + + + + +
LimitLocationValueObservable?
per-job live-log byteslive_logs.rs:2564 MiBNo — tail-drop
oversized live-log batchlive_logs.rs:250> capWARN only
queue: max pending holdersconcurrency.rs:255100No — cancels a job
archived debug sessionsdebug_sessions.rs:6464Ring eviction
debug session eventsdebug_sessions.rs:68512Ring eviction
debug session auditdebug_sessions.rs:71512Silent audit loss
completed debug opsdebug_sessions.rs:74256Ring eviction
git request bodysnapshots.rs:2016 MiBRejected, not counted
reusable workflow depthremote_workflows.rs:64Rejected, not counted
+
+
Why this matters
+ A cap that silently discards data is worse than an outage. queue: max overflow cancels a user's job with no distinction from any other cancellation. Silent audit eviction is a compliance defect. Fix: one bounded preloop.limit.* family whose limit attribute is the constant's name — a compile-time-finite set. +
+ +

2 · Scheduled workflows have a history endpoint and no telemetry

+

scheduler.rs drives cron workflows from bootstrap.rs:479/485 and exposes GET /api/v1/scheduler/history. Nothing reports whether a schedule fired, fired late, was skipped for overlap, or stopped because the scan task died.

+ +

3 · GitHub dependency budget is untracked

+

Caches at dispatch_auth.rs:46,49 (60 s TTL), oidc.rs:25 (300 s), actions.rs:32 (6 h). No x-ratelimit-* parsing, no token-expiry tracking, no hit-rate reporting. Rate-limit exhaustion is indistinguishable from “GitHub is slow”.

+ +

4 · Persistent storage growth is unbounded and unreported

+

State dir, cache, artifacts, run logs, snapshots, VM images all grow. Only the VM volume gets a free-space signal in the first draft. A full state-dir filesystem is a hard outage; the first symptom today is a store write failure with no capacity context.

+ +

5 · Fifteen background tasks, two proposed heartbeats

+
+ + + + + + + + + + + + + + + + +
TaskLocationCadence
reaperbootstrap.rs:396-41010 s
scheduler scanbootstrap.rs:479, 485timer
GitHub App event loopbootstrap.rs:497, 517event-driven
shutdown supervisorbootstrap.rs:568event-driven
listener accept loopsbootstrap.rs:601,615,660,679per-conn
snapshot GCsnapshots.rs:1896periodic
replay prunedistributed_task.rs:895periodic
Postgres connectionstore_pg.rs:95lifetime
GitHub check dispatchgithub.rs:1064event-driven
auto-PR gategithub_pr.rs:397event-driven
pool supervisororchestrator/lib.rs:1880-2100loop
guest pause watchersorchestrator/lib.rs:3027per-VM
key rotationorchestrator/keys.rs:74periodic
+

Hand-wiring two heartbeats and leaving thirteen silent reproduces the failure mode. Fix: one TaskHeartbeat registry — register at spawn, beat each iteration, deregister on drop; readiness reads only the critical subset.

+ +

6 · HTTP surfaces are wider than classified

+

routes.rs serves thirteen path families, not eight: /_apis, /broker, /runner, /twirp, /api/v1, /ws/live-logs, /snapshots, /repos, /oidc, etc. Two need different treatment from a duration histogram: the WebSocket and the git-object path.

+ + +

Architecture & Invariants

+ +
+
Three layers — zero-dependency diagnostics first, vendor-neutral telemetry second, optional backend third
+ + + + + + + + + LAYER 1 — ZERO-DEPENDENCY + LAYER 2 — VENDOR-NEUTRAL TELEMETRY + LAYER 3 — OPTIONAL BACKEND + + + + + + preloop status + CLI · native bearer + + + + + systemd / probes + health checks + + + + + Prometheus + scraper · bearer auth + + + + + + /api/v1/status + + + + /healthz /readyz + public · shallow + + + + /metrics + + + + + + + + + + + Control plane + broker · store · scheduler + + + + Runner pool + provision · slots · goldens + + + + Host VM sampler + cgroup · process · sparse disk + + + + + + preloop-observability + metrics · logs · traces · status snapshot + + + + + + + + + + stderr / journald + always retained + + + + /metrics + + + + OTLP/HTTP + bounded · optional + + + + + + + + + OpenObserve + or any OTLP backend + + + opt-in + + + + + OTel Collector — optional + host telemetry · fan-out + + + + + SIDE CHANNELS — NOT TELEMETRY + + + run-log store + + + + flow capture · 0600 + + + + + +
+ Layer 1 — always on + In-process + Optional + Side channel + — solid = always · - - dashed = opt-in +
+
+ +

New crate — preloop-observability

+

Small, explicit API with no dependency on server or orchestrator internals. Both preloop and standalone preloop-server construct one handle/runtime before building ServerConfig; the same handle is cloned into AppState and RunnerPoolConfig.

+
+ + + + + + + + + +
SymbolRole
ObservabilityConfig::from_env()Parses PRELOOP_LOG_FORMAT + standard OTEL_* without exposing header values in Debug
Observability::noop()Allocation-light handle for tests — zero network I/O
ObservabilityCloneable handle: instruments, cached snapshot, pool/VM handles, heartbeats, export health
TaskHeartbeatregister(name, critical) → HeartbeatHandle · beat() · Drop deregisters
LimitRegistryrecord_drop(limit, n) / record_reject(limit) — limit is a &'static str constant name
ObservabilityRuntimeOwns provider guards; bounded 2 s flush on shutdown
+ +

Non-negotiable invariants

+
+
1
Fail open

Telemetry failure produces a warning + condition, never a failed request.

+
2
No request-path export

Bounded non-blocking batches. Queue overflow drops telemetry, increments health.

+
3
Bounded shutdown

Flush ≤ 2 s after pool shutdown. Then exit.

+
4
No backend by default

Absent OTEL endpoint = disabled. Documented deviation from spec default localhost:4318.

+
5
Always retain stderr

OTLP augments, never replaces journald.

+
6
No wire changes

No field/status/body changes on runner protocol routes.

+
7
No high-cardinality labels

IDs and user strings are logs/traces only.

+
8
No workflow output export

Workflow stdout stays in the run-log store.

+
9
No raw HTTP capture

Allowlisted trace attributes, not denylisted.

+
10
No state-lock callbacks

Sampler updates a cached snapshot; exporters read it.

+
11
No hot-path allocation

Prebuilt instruments; poll/renew is metrics-only.

+
12
No guest polling

Host cgroup/process/filesystem only. Never free inside the VM.

+
13
No fake zeroes

Unsupported = absent + capability false.

+
14
No silent drop

Any cap that discards data must record it via LimitRegistry.

+
15
No unregistered task

Every long-lived spawn registers a TaskHeartbeat.

+
+ + +

Operator Surfaces

+ +

/healthz — liveness only

+

Unauthenticated, shallow. 200 while serving, 503 during shutdown. Must not touch DB, GitHub, SmolVM, or InnerState.

+
GET /healthz — 200public
// always lock-free +{
  "schema_version": 1,
  "ok": true,
  "protocol_version": "0.1.0",
  "shutdown_requested": false
}
+ +

/readyz — critical-loop readiness

+

Unauthenticated, boolean + stable reason codes. 200 when the state sampler and every critical heartbeat are fresh. 503 for starting, task_stale (with the stale task's registry name), or shutting_down. Non-critical staleness never gates readiness.

+
Critical set
State sampler · reaper (bootstrap.rs:396) · scheduler scan (bootstrap.rs:479) · store connection task when Postgres (store_pg.rs:95). Everything else degrades to /api/v1/status.
+

wait_for_engine_socket (preloop-cli/src/main.rs:1458-1475) now probes /readyz; on 30 s timeout it surfaces the last reason code instead of a generic timeout.

+ +

/api/v1/status — authenticated operational diagnosis

+

Versioned snapshot, 5-second sampler, never blocks on InnerState. Reports snapshot_age_seconds. Bearer auth required.

+ + +

UI Mockups — Live Preview

+

Interactive previews of every operator surface. Switch tabs to compare the human CLI, the machine JSON, and the metric wire format.

+ +
+
+ + + + + +
+ + +
+
+
preloop status — human output (sections 1–10)
+
preloop v0.29.0 · instance 9f3a…c1 · up 2h 14m · snapshot 0.4s ago ● ok + +Queue — oldest ready 14.2s + ready 2 jobs · 1 claimable · 1 unclaimable (label mismatch) + dependency_blocked 1 · concurrency_blocked 1 · expanding 0 + +Concurrency — groups 3 · contended 1 · queue max 100 + deepest pending 4 · overflow cancellations 0 + +Scheduler — 4 schedules · last scan 3.1s ago + fired 12 · skipped (overlap) 1 · late fires 0 · max delay 2.1s + +Pool — mode warm · desired 2 · preparing false + idle 1 · busy 1 · building 0 · provisioning 0 · paused 0 · consecutive failures 0 + +Runners — registered 2 · sessions 2 + idle 1 · busy 1 · stale 0 · max poll age 3.1s · max lease age 8.0s + +VMs — source cgroup_v2 · sample 1.2s ago · capabilities: cpu ✓ mem ✓ throttle ✓ oom ✓ pids ✓ sparse ✓ + runner 2 · golden 1 · unreachable 0 + configured 10 vCPUs · 20 GiB mem · 80 GiB storage + host usage 2.4 cores · 7.0 GiB · sparse 12 GiB + top consumers (by memory pressure) + preloop-runner-0 busy 1.8 cores · 4.0 GiB / 7.0 GiB 57% + preloop-runner-1 idle 0.2 cores · 0.9 GiB / 7.0 GiB 13% + +Store — sqlite · consecutive failures 0 · conn up +Storage — state dir 41 GiB free (42%) · largest: vm_images 64 GiB · cache 2.0 GiB +GitHub — configured · rate 4812/5000 (reset in 18m) · token 47m · cache 91/4 hits +Debug — active 1 · oldest 3m · audit cap 512 (evicted 0) +Telemetry — OTLP disabled · dropped 0 + +▸ condition: queue_label_mismatch — job abc123 requires runs-on: gpu but no runner advertises it (1 exemplar) + action: add a runner with that label or change runs-on + +Recent runs — last 5 + ✓ #42 push main abc123 1.2m queued → in_progress → completed + ✓ #41 pull_request def456 58s + ● #43 schedule nightly — queued 14s
+
+

Human output omits healthy limits/tasks (all zero) and clean conditions. --json includes everything. One-line actions on every condition, bounded to five exemplars.

+
+ + +
+
+
GET /api/v1/status — 200 · bearer requiredschema v1 · snapshot_age 0.4s
+
{ +  "schema_version": 1, +  "observed_at": "2026-08-20T11:30:42Z", +  "snapshot_age_seconds": 0.4, +  "overall": "degraded", +  "service": { "version": "0.29.0", "instance_id": "9f3a…c1", "uptime_seconds": 8040 }, +  "jobs": { "ready": 2, "claimable": 1, "unclaimable": 1, "oldest_ready_seconds": 14.2 }, +  "concurrency": { "groups_active": 3, "deepest_group_pending": 4, "overflow_cancellations": 0 }, +  "scheduler": { "enabled": true, "schedules": 4, "late_fires": 0, "max_fire_delay_seconds": 2.1 }, +  "pool": { "mode": "warm", "desired": 2, "idle": 1, "busy": 1 }, +  "vms": { +    "source": "cgroup_v2", "count": { "runner": 2, "golden": 1 }, +    "host_usage": { "cpu_cores": 2.4, "memory_bytes": 7516192768 }, +    "capabilities": { "cpu_throttling": true, "host_oom_events": true }, … 5 top consumers +  }, +  "storage": { "state_fs_free_bytes": 41231234560, "components": [… 6 entries] }, +  "limits": [{ "limit": "LIVE_LOG_MAX_BYTES", "value": 67108864, "dropped": 0, "rejected": 0 }, … 8 more — all caps even at zero], +  "tasks": [{ "name": "reaper", "critical": true, "heartbeat_age_seconds": 3.2, "state": "running" }, … 14 more], +  "github": { "rate_limit": { "remaining": 4812, "limit": 5000 }, "installation_token_expires_in_seconds": 2841 }, +  "conditions": [{ "code": "queue_label_mismatch", "severity": "warn", "exemplars": [… ≤5, with run/job IDs] }] +}
+
+
Contract
Field names are frozen at schema v1. New fields are additive. limits and tasks are arrays of constant-named entries so the contract survives new caps/tasks. preloop status --json prints this body byte-for-byte — pipe-friendly for jq.
+
+ + +
+
+
GET /readyz — 200ready
{
  "ready": true,
  "checks": {
    "state_sampler": "ok",
    "reaper": "ok",
    "scheduler_scan": "ok",
    "store_connection": "ok"
  }
}
+
GET /readyz — 503not ready
{
  "ready": false,
  "reason": "task_stale",
  "task": "scheduler_scan",
  "heartbeat_age_seconds": 38.4,
  "hint": "schedule fires delayed — see /api/v1/status"
}
+
+

Only the four critical heartbeats gate readiness. A stale non-critical task (e.g. snapshot GC) surfaces in /api/v1/status but still returns 200.

+
+ + +
+
# HELP preloop_job_queue_depth Jobs by queue kind +# TYPE preloop_job_queue_depth gauge +preloop_job_queue_depth{queue="ready"} 2 +preloop_job_queue_depth{queue="concurrency_blocked"} 1 +preloop_concurrency_pending_depth 4 +preloop_scheduler_fire{outcome="fired"} 12 +preloop_scheduler_fire{outcome="skipped_overlapping"} 1 +preloop_limit_dropped{limit="LIVE_LOG_MAX_BYTES"} 0 +preloop_limit_rejected{limit="QUEUE_MAX_PENDING"} 0 +preloop_task_heartbeat_age{task="reaper"} 3.2 +preloop_vm_host_memory_usage{role="runner",source="cgroup_v2"} 7516192768 +preloop_github_rate_limit_remaining{resource="core"} 4812 +preloop_storage_fs_available{mount="state_dir"} 41231234560 +# HELP http_server_request_duration API latency — matched route, never raw URI +http_server_request_duration_bucket{method="POST",route="/api/v1/runs",surface="native",le="0.25"} 42
+
Cardinality guard
A bounded set of metric names × finite label values. 1,000 distinct run/job/runner IDs must not increase series count — tested by driving 1,000 sentinels and asserting the Prometheus exposition is constant.
+
+ + +
+
+
journalctl -u preloop -o json | jq — structured logs (OTLP + stderr)
+
2026-08-20T11:30:42Z INFO job.claimed run_id=abc123 job_id=job_7 runner_id=42 duration_ms=12 +2026-08-20T11:30:43Z WARN vm.host.memory.pressure machine=preloop-runner-0 ratio=0.92 current=6.6GiB limit=7.0GiB source=cgroup_v2 +2026-08-20T11:30:44Z WARN limit.exceeded limit=QUEUE_MAX_PENDING value=100 rejected=1 # concurrency group overflow +2026-08-20T11:30:44Z WARN github.rate_limit.low resource=core remaining=412 limit=5000 reset_in=847s +2026-08-20T11:30:45Z INFO schedule.fired schedule=nightly@04:00 delay_s=1.8 # cron +2026-08-20T11:30:46Z ERROR store.connection.lost backend=postgres # connection task exited +2026-08-20T11:30:47Z WARN vm.unreachable machine=preloop-golden-0 pid=18412 age=67s # vsock probe — stop forking from it
+
+

event.name + stable fields on every line. Trace/span IDs provide correlation — no second UUID. Workflow step output never appears here.

+
+
+ + +

Signal Contract

+ +

Resource attributes on every signal

+
+ + + + + + + +
AttributeValue
service.namepreloop (overridable via OTEL_SERVICE_NAME)
service.versioncrate version
service.instance.idUUID per process start — not a substitute for distributed coordination
deployment.environment.nameonly when supplied via OTEL_RESOURCE_ATTRIBUTES
+ +

Metric attribute policy

+

Allowed values are bounded enums or finite route templates. Forbidden: IDs, repo/workflow/ref/SHA, runs-on values, cache keys, artifact names, raw URLs, error text, tokens.

+

Three narrow exceptions — all compile-time-finite: limit (cap constant names), task (heartbeat registry names), store (storage component names). A cardinality test drives 1,000 distinct IDs through instrumentation and asserts a fixed series bound and no sentinel leakage.

+ +

Metrics catalog — highlights

+

Full catalog is 50+ instruments. Key additions since the first draft:

+
+ + + + + + + + + + + + + + + + + + +
InstrumentTypeKey attributesPurpose
preloop.limit.droppedcounterlimittail-drops, ring evictions
preloop.limit.rejectedcounterlimitqueue-max overflows, body rejections
preloop.limit.valuegaugelimitconfigured ceiling — alerts compare against it
preloop.task.heartbeat.agegaugetaskdead-loop detector (generic)
preloop.task.exitedcountertask, outcomebackground task termination
preloop.concurrency.decisioncounterqueue mode, actionwhy a job was parked/cancelled
preloop.scheduler.firecounteroutcomedid the cron actually run
preloop.scheduler.fire.delayhistogram—schedule slot → dispatch (buckets 1s–3600s)
preloop.github.rate_limit.remaininggaugeresourcex-ratelimit-remaining
preloop.github.token.expires_ingaugekindcredential countdown
preloop.storage.bytesgaugestoreper-component footprint
preloop.storage.fs.availablegaugemountauthoritative free space
preloop.store.connection.upgaugebackendPostgres connection liveness
preloop.snapshot.cachecounteroutcomegit object serving
preloop.livelog.connectionsupdown—WebSocket connections (not a duration)
+
New bucket view
Scheduler fire delay: 1, 5, 15, 30, 60, 300, 900, 3600 s. A cron job is not late at 10 ms resolution — reusing HTTP buckets would waste eight of eleven boundaries. /ws/live-logs is excluded from http.server.request.duration entirely.
+ +

VM telemetry contract

+

VM metrics are first-class Preloop telemetry — the embedded pool owns these processes. General node telemetry remains optional via an OTel Collector.

+
+ + + + + + + + + + +
SignalLinux sourcemacOS / fallbackMeaning
configured CPU/mem/storagecached SmolVM --jsonsamerequested capacity, not use
CPU time / corescpu.stat usage_usecVMM process CPUhost CPU for the VM runtime
CPU throttlingcpu.stat throttle fieldsunavailablecgroup quota pressure
host memorymemory.currentVMM RSShost memory charged to the VMM
host PIDspids.currentunavailableVMM/helper processes, not guest
sparse diskst_blocks × 512samephysical blocks for VM files
free diskstatvfssameauthoritative pressure — per-VM blocks are diagnostic only
+
+
SmolVM floor v1.8.1 — stronger than first draft assumed
+

machine status --json and machine ls --json emit the same object from one helper (machine_status_json). state is vsock-probed and includes Unreachable — “VMM PID alive, guest agent not answering.” Fleet sampling needs one machine ls --json per slow pass, not one subprocess per VM (and SmolVmProvider::list already parses it).

+

A missing vm-<pid> cgroup leaf (SmolVM's supervisor creates it, not Preloop) degrades to the process fallback — never an error, capability reported false.

+
+
+ vm_unreachable + vm_host_memory_pressure + vm_host_cpu_throttled + vm_host_oom_kill + vm_sparse_disk_pressure + vm_sampler_stale + vm_sampler_unavailable +
+ +

Structured log catalog

+
+ + + + + + + + + + + + +
EventLevelKey fields
job.concurrency.cancelledWARNrun/job ID, queue mode, hash of group key (never raw)
schedule.fired / skippedINFO / WARNschedule ID, delay, reason
task.exitedWARN / ERRORregistry name, outcome; ERROR if critical
limit.exceededWARNlimit name, value, drop/reject delta (rate-limited)
vm.unreachable / reachableWARN / INFOmachine, PID, age since reachable
store.connection.lostERRORbackend — Postgres connection task died
storage.pressureWARNmount, free bytes/ratio, largest component
github.rate_limit.lowWARNresource, remaining, reset_in
debug.audit.evictedWARNsession ID, evicted count, cap 512
+
Hash, don't leak
job.concurrency.cancelled logs a hash of the group key, never the raw key — the expression interpolates branch names and PR titles. The hash still correlates all jobs in one group without exporting user data.
+ +

Trace policy

+

Short synchronous spans, not one hours-long workflow trace. Correlate via run.id / job.id / runner.id / machine.name fields. Suppress successful health probes, renewals, and empty long-polls.

+ + +

SLOs & Alerts

+
+ + + + + + + + + + + + +
SLIInitial objectiveDenominator
Control API / broker availability99.9%exclude 4xx + empty long-polls
Dispatch latency99% within 30 sonly claimable jobs; preparation reported separately
Terminal propagation99% within 30 sjob terminal → run final + check ack
Durable-state writesno failure > 5 mall store ops
Runner livenessno deaf session beyond boundactive sessions
VM telemetry freshnessno stale fast sample > 3 intervalsowned VMs
Scheduled-trigger fidelity99% within 60 sexcludes deliberate overlap skips
Critical-task livenessno critical heartbeat stale > 3 intervalsregistry critical set
Data retention honestyzero unreported dropsevery drop must have a visible condition
+

Instrument first, baseline two weeks, then ratify. Do not page on workflow failure rate — user code fails legitimately.

+ +
+
!
Telemetry absent
no preloop.service.uptime for 5 m
+
!
Dispatch stalled
claimable job exceeds baseline while idle capacity exists
+
⚠
Unclaimable queue
label mismatch vs no runner — distinguished
+
!
Pool deficit
desired > idle+busy+building+provisioning
+
!
Runner deaf
max poll/lease age near timeout
+
!
Store failing
consecutive failures or connection down
+
⚠
Schedule did not fire
slot passed, no scheduler.fire — invisible to queue alerts
+
⚠
Concurrency overflow
user job cancelled by queue: max policy
+
⚠
Data being dropped
any limit.dropped/rejected — page on audit eviction
+
!
GitHub budget low
remaining/limit < 10% or token near expiry
+
!
VM unreachable
wedged golden — stop forking from it
+
⚠
VM pressure
memory / CPU throttle / disk / OOM
+
!
Background task dead
task.heartbeat.age > 3 intervals or task.exited
+
⚠
Storage capacity
free space low + component growing past GC
+
+ + +

Dashboard Gallery — Six Importable Boards

+

Every panel maps 1:1 to a metric in the catalog. No dashboard-only derivations for state the server already knows. Sparklines below are illustrative — real boards query the same Prometheus/OTLP source.

+ +
+ + +
+

1 · Overview

always on
+
+
+
Service uptime
2h 14m
no-data heartbeat
+
API p99 latency
42 ms
native + broker
+
Runs active
3
queued 1 · in_progress 2
+
Conditions
1 warn
queue_label_mismatch
+
+
+
+ + +
+

2 · Scheduling & Queue

new
+
+
+
Queue depth by kind
2 ready
blocked 1 · expanding 0
+
Oldest ready
14.2 s
claimability: label mismatch
+
Concurrency — deepest pending
4
groups active 3 · contended 1
+
Scheduler fires
12 · 1 skipped
max delay 2.1 s
+
+
+
+ + +
+

3 · Runners & Pool

capacity
+
+
+
Runners
2 reg
idle 1 · busy 1 · stale 0
+
Pool desired vs actual
2 / 2
building 0 · provisioning 0
+
Max poll age
3.1 s
max lease 8.0 s
+
Session transitions
—
create / delete / reap
+
+
+
+ + +
+

4 · VM Host Resources

host-observed
+
+
+
Fleet
3 VMs
runner 2 · golden 1 · unreachable 0
+
Host CPU
2.4 cores
of 10 vCPUs configured
+
Host memory
7.0 GiB
pressure events 0 · OOM 0
+
Sparse disk
12 GiB
free 41 GiB (42%) · capability ✓
+
+
Top consumers (authenticated /api/v1/status only — never metric labels)
+
+
+ + +
+

5 · Dependencies & Telemetry

budget
+
+
+
GitHub rate limit
4812 / 5000
reset in 18 m · token 47 m
+
Token cache hit rate
96%
91 hits · 4 misses · TTL 60s
+
Store
sqlite · 0 fail
conn up · WAL ckpt 128
+
Telemetry export
disabled
dropped 0 · OTLP opt-in
+
+
+
+ + +
+

6 · Limits & Background Tasks

new
+
+
+
Caps registered
9
all at 0 drops — clean
+
Largest drop source
— none —
last drop: —
+
Critical heartbeats
4 / 4 ok
sampler · reaper · scheduler · store
+
All tasks
15
11 non-critical · all running
+
+
Invariants 14 & 15 — a new cap or spawn without a counter/heartbeat is a review failure.
+
+
+ +
+ + +

OpenObserve Evaluation

+

Verdict: use OpenObserve as the documented, optional “batteries available” backend. Do not make it a runtime dependency, embed it, or shape signals around its fields.

+
+ + + + + + + + + + +
CriterionAssessmentDecision
Minimal deploymentGood — single binary/containerPinned compose, never auto-start
Unified signalsOTLP/HTTP + gRPC for logs/metrics/tracesPreloop does OTLP/HTTP first (no tonic)
Dashboards / alertsGoodSix importable boards + 14 alerts
StorageAnother stateful volumeShort retention, persistent volume, backups
HAKubernetes + object store + PG + NATS + 5 rolesNot shipped; link upstream docs
SecuritySSO/RBAC/audit are enterprise-onlyLoopback by default, operator auth if shared
LicensingAGPL-3.0Separate process; legal review before bundling
+

Losing SQLite metadata makes the installation inoperable — persist and back up both metadata and stream data. Do not put the UI on the public webhook origin.

+ + +

Configuration Contract

+

One Preloop-specific setting: PRELOOP_LOG_FORMAT=auto|pretty|json (auto = TTY-aware, no ANSI in journald).

+

Honor standard variables — do not invent backend-specific ones:

+
+ + + + + + + + +
VariablePurpose
RUST_LOGfilter (standalone server now gets the same info fallback as the CLI)
OTEL_SERVICE_NAMEoverrides service.name
OTEL_EXPORTER_OTLP_ENDPOINTenables export — absent means disabled (deviation from spec default localhost:4318)
OTEL_EXPORTER_OTLP_PROTOCOLhttp/protobuf — only supported value in this plan
OTEL_TRACES/METRICS/LOGS_EXPORTERnone disables the signal
+
Spec deviation — document it
OTel's default endpoint is http://localhost:4318. Preloop treats an absent variable as disabled so a CI control plane does not emit background connection attempts. State this next to the variable list in docs/observability.md.
+ + +

Scope

+
+
+

In scope

+
    +
  • Cargo.toml, versions.toml (floor only if needed)
    • +
  • new crates/preloop-observability
    • +
  • CLI + server + orchestrator + VM crates
    • +
  • docs/{architecture,self-hosting,cli_reference,observability}.md
    • +
  • contrib/openobserve/ — pinned compose + 6 boards
    • +
  • one rules/ ast-grep guard
    • +
+
+
+

Out of scope

+
    +
  • Runner protocol changes · guest export
    • +
  • Workflow step output export
    • +
  • OpenObserve HA · multi-server Preloop
    • +
  • eBPF / required Collector
    • +
  • Guest-kernel metrics until SmolVM exposes them
    • +
  • Per-machine metric labels · fake zeroes
    • +
  • A new web UI inside Preloop
    • +
+
+
+ + +

Implementation Steps — Seven PRs

+

One commit per step. Keep protocol instrumentation separate from reference-backend assets.

+ +
+
+
1
+

Freeze the signal/security contract and make existing logs safe

Add docs/observability.md. Re-run the token/body grep and fix every hit (four moved between 84d92cfd and 673bdfa0). Add an ast-grep rule rejecting token / authorization / cookie / headers / body / payload / signed_url at INFO/WARN/ERROR. Audit all non-DEBUG macros.

securitydocs

just sg-scan-strict + sentinel logging test — no sentinel appears, flow capture stays 0600.

+
+
+
2
+

Add the observability crate and unify process initialization

Workspace member preloop-observability at opentelemetry 0.32 / tracing-opentelemetry 0.33 / prometheus 0.14 / sysinfo 0.39.6. No-op / Prometheus-only / Prometheus+OTLP paths. TaskHeartbeat + LimitRegistry usable from the no-op handle. Fix standalone server's missing info fallback.

crate

No endpoint = no socket. Malformed endpoint never delays a request. Bounded queues, 2 s flush, sanitized Debug.

+
+
+
3
+

Truthful liveness, readiness, aggregate status, and CLI output

5 s sampler, /readyz gated on the four critical heartbeats, /api/v1/status + /metrics behind native bearer, consolidated PoolStatus replacing four ad-hoc handles. preloop status --json (copy PlanArgs shape) + 10-section human output.

apicrate

Health public/shallow; status/metrics 401 without bearer; claimability distinguishes absence / mismatch / preparing / provisioning.

+
+
+
4
+

Instrument HTTP, runs/jobs, dispatch, runners, and the store

Matched-route instrumentation, thirteen surfaces (WebSocket excluded from duration histogram), one InstrumentedStore wrapping Arc<dyn Store>, concurrency decisions + scheduler scan, Postgres connection liveness.

instrumentation

just conform-server-light — no protocol diff. 1,000 IDs do not increase series.

+
+
+
5
+

Instrument pool, VM resources, GitHub, webhook, cache/artifacts, and debug sessions

Typed SmolVM inspection (status --json + ls --json fleet call), MachineState::Unreachable, 5 s / 60 s samplers, storage sampler, GitHub budget gauges, snapshot serving, ring-cap registrations.

pool · vm · github

Cgroup fixtures, PID-reuse safety, missing-leaf fallback, 1,000 ephemeral machines = constant series.

+
+
+
6
+

Add the optional OpenObserve reference profile and assets

Pinned compose (loopback, persistent volume, healthcheck, resource caps), six dashboards, 14 alerts. Measure resource caps — do not guess.

compose · dashboards

just observability-openobserve-smoke — one log, one metric, VM series, one trace queryable.

+
+
+
7
+

Baseline, ratify alerts, document the incident workflow, and close the loop

Two-week soak, tune buckets/thresholds, runbooks for every alert, architecture + self-hosting docs, just test-ci && just conform-server-light && just dogfood.

docs · runbooks
+
+
+ + +

Test Plan

+

Permanent tests must defend observable contracts, not source text:

+
+ + + + + + + + + + + + + + +
#ContractHow
1Disabled pathNo OTEL endpoint → no DNS/socket, status/metrics still usable
2Failure isolationMalformed/down/slow/full/timeout never alters API result
3SecuritySentinel tokens/headers/bodies never enter logs/traces/metrics
4Cardinality1,000 IDs → constant bounded series, no sentinel in exposition
5HTTP semanticsMatched templates + finite surfaces; query values absent
6StatusAll queue/capacity/GitHub/debug states + auth + stale snapshot
7Lifecycle exactnessEach terminal reason fires exactly once
8Pool balanceNo gauge leak on early return/cancellation
9VM samplingv1.8.1 JSON, cgroup fallback, PID reuse, sparse allocation
14Limit honestyEvery cap driven past its ceiling → counter moves
15Task livenessPanicked/early-returned task reported; non-critical does not gate readiness
+ + +

Done Criteria

+
All must hold before closing
+
    +
  • preloop status explains queue, capacity, runner freshness, and every condition names an action.
    • +
  • preloop status --json returns schema v1 exactly — no prose.
    • +
  • /healthz / /readyz / /api/v1/status / /metrics have specified auth/semantics.
    • +
  • No backend contacted when OTLP endpoint absent.
    • +
  • OTLP failure cannot block a request; shutdown flush ≤ 2 s.
    • +
  • Every cap registered — zero drops still reported — and every long-lived task has a heartbeat.
    • +
  • Scheduled workflows, concurrency overflow, GitHub budget, and storage capacity are visible.
    • +
  • Four ad-hoc RunnerPoolConfig handles gone, replaced by one PoolStatus.
    • +
  • VM Unreachable state honored end-to-end.
    • +
  • Six dashboards + 14 alerts import and query real fields.
    • +
  • cargo fmt --all --check, just sg-scan-strict, just test-ci, just conform-server-light, just dogfood, just observability-openobserve-smoke pass.
    • +
+
+ + +

Stop Conditions

+
Stop and report — do not improvise
+
    +
  • Instrumentation would change a runner wire response.
    • +
  • OTel crate versions cannot provide Prometheus + OTLP without a second instrument set.
    • +
  • An exporter performs I/O on the handler thread.
    • +
  • SmolVM floor lacks status --json / ls --json / data-dir or changes field semantics — including dropping the shared machine_status_json shape.
    • +
  • Adding MachineState::Unreachable would require changing pool scheduling behavior in the same PR.
    • +
  • PID start identity cannot be validated on a supported host.
    • +
  • A metric needs a user-controlled label.
    • +
  • Current code no longer has the named centralized boundaries.
    • +
  • Conformance or dogfood changes after instrumentation.
    • +
+
+ + +

Maintenance Notes

+
    +
  • Signal names, units, label sets, status schema, condition codes, and termination reasons are public operational APIs — additive first, documented migration when not.
    • +
  • Every new cap/ring/truncation must register with LimitRegistry in the same PR. Every new tokio::spawn that outlives a request must register a TaskHeartbeat. Both are review checklist items.
    • +
  • Anchors rot fast — prefer the greps and symbol names over line numbers; re-run the drift check before each step.
    • +
  • VM names/docs must retain the host distinction: cgroup RSS/PIDs/OOM are VMM observations, not guest-kernel counters.
    • +
  • Keep OpenObserve digests and assets tested together. Preloop's OTLP contract must work with other vendors.
    • +
+ +
+

+ Plan 002 · 673bdfa0 · 2026-08-20 · Markdown source · Plans index +

+ +
+
+ + + + From a30f6910f90c370fa81b90994eaf23a1c196e4cf Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 15:15:11 -0400 Subject: [PATCH 03/22] fix(server): scrub capability tokens from INFO/WARN logs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Before any OTLP export is wired, remove bearer material that would leak into journald and the telemetry pipeline. At 673bdfa0 four sites logged capability tokens or raw bodies; three of them had already moved between 84d92cfd and 673bdfa0, so the fix is verified by grep rather than fixed anchors. - artifact_twirp.rs:94 info!(token, name) -> info!(name, workflow_run_backend_id, workflow_job_run_backend_id) — token is the signed upload URL capability, keep registry coordinates - results_twirp.rs:713 info!(token, "cache v2 create") -> info!(key, version) — storage identity, not capability; clone key/version for the pending insert so the log can still borrow them - blob_store.rs:63,78,95,113,121,127,131 warn!/info!(kind, token) -> kind + block/size/blocks — blob operation, not token; debug! block log drops token entirely - distributed_task.rs:327 info!(?body, "agent_request_patch") -> info!(pool_id, request_id, result, has_result) — bounded enum, not raw PATCH JSON Add rules/no-sensitive-log-fields.yml (ast-grep, error level, ignores recording.rs conformance capture) rejecting INFO/WARN/ERROR fields token, authorization, cookie, headers, body, payload, signed_url. Mark docs/internal/observability.md as the internal contract (docs/internal/ is gitignored; public docs/observability.md will be a redacted subset later) and update target in plans/002-observability-strategy.md accordingly. cargo fmt --all, cargo check -p preloop-runner-server, just sg-scan-strict all pass. Entire-Checkpoint: 01M0G9JJ3FE3Q591DR1YH8EJDH --- .../src/artifact_twirp.rs | 7 ++- .../preloop-runner-server/src/blob_store.rs | 24 ++++++---- .../src/distributed_task.rs | 12 ++++- .../src/results_twirp.rs | 25 +++++++--- crates/preloop-runner-server/src/store.rs | 2 +- plans/002-observability-strategy.md | 4 +- rules/no-sensitive-log-fields.yml | 48 +++++++++++++++++++ 7 files changed, 103 insertions(+), 19 deletions(-) create mode 100644 rules/no-sensitive-log-fields.yml diff --git a/crates/preloop-runner-server/src/artifact_twirp.rs b/crates/preloop-runner-server/src/artifact_twirp.rs index c2ebfb49..e8ba7fa6 100644 --- a/crates/preloop-runner-server/src/artifact_twirp.rs +++ b/crates/preloop-runner-server/src/artifact_twirp.rs @@ -91,7 +91,12 @@ pub(crate) async fn twirp_artifact_v2_create( } } let upload_url = format!("{}/twirp-blob/artifact/{token}", runner_base_url()); - info!(token, name = request.name, "artifact v2 create"); + info!( + name = request.name, + workflow_run_backend_id = request.workflow_run_backend_id, + workflow_job_run_backend_id = request.workflow_job_run_backend_id, + "artifact v2 create" + ); Ok(Json(json!({ "ok": true, "signed_upload_url": upload_url }))) } diff --git a/crates/preloop-runner-server/src/blob_store.rs b/crates/preloop-runner-server/src/blob_store.rs index 41fa0cd2..0d308f40 100644 --- a/crates/preloop-runner-server/src/blob_store.rs +++ b/crates/preloop-runner-server/src/blob_store.rs @@ -60,14 +60,13 @@ pub(crate) async fn blob_put( let safe_id = blockid_to_filename(&block_id); let blocks_dir = blob_root.join("blocks"); if let Err(e) = tokio::fs::create_dir_all(&blocks_dir).await { - warn!(kind, token, "failed to create blocks dir: {e}"); + warn!(kind, "failed to create blocks dir: {e}"); return StatusCode::INTERNAL_SERVER_ERROR; } match tokio::fs::write(blocks_dir.join(&safe_id), &body).await { Ok(()) => { debug!( kind, - token, block = safe_id, bytes = body.len(), "blob block staged" @@ -75,7 +74,7 @@ pub(crate) async fn blob_put( StatusCode::CREATED } Err(e) => { - warn!(kind, token, "failed to write block {safe_id}: {e}"); + warn!(kind, block = %safe_id, "failed to write block: {e}"); StatusCode::INTERNAL_SERVER_ERROR } } @@ -92,7 +91,7 @@ pub(crate) async fn blob_put( match tokio::fs::read(blocks_dir.join(&safe_id)).await { Ok(bytes) => assembled.extend_from_slice(&bytes), Err(e) => { - warn!(kind, token, "failed to read block {safe_id}: {e}"); + warn!(kind, block = %safe_id, "failed to read block: {e}"); return StatusCode::INTERNAL_SERVER_ERROR; } } @@ -102,7 +101,6 @@ pub(crate) async fn blob_put( let _ = tokio::fs::remove_dir_all(&blocks_dir).await; info!( kind, - token, size = assembled.len(), blocks = block_ids.len(), "blob assembled from blocks" @@ -110,7 +108,11 @@ pub(crate) async fn blob_put( StatusCode::CREATED } Err(e) => { - warn!(kind, token, "failed to write assembled blob: {e}"); + warn!( + kind, + blocks = block_ids.len(), + "failed to write assembled blob: {e}" + ); StatusCode::INTERNAL_SERVER_ERROR } } @@ -118,17 +120,21 @@ pub(crate) async fn blob_put( _ => { // Single-shot upload. if let Err(e) = tokio::fs::create_dir_all(&blob_root).await { - warn!(kind, token, "failed to create blob dir: {e}"); + warn!(kind, "failed to create blob dir: {e}"); return StatusCode::INTERNAL_SERVER_ERROR; } let data_path = blob_root.join("data"); match tokio::fs::write(&data_path, &body).await { Ok(()) => { - info!(kind, token, size = body.len(), "blob single-shot upload"); + info!(kind, size = body.len(), "blob single-shot upload"); StatusCode::CREATED } Err(e) => { - warn!(kind, token, "failed to write single-shot blob: {e}"); + warn!( + kind, + size = body.len(), + "failed to write single-shot blob: {e}" + ); StatusCode::INTERNAL_SERVER_ERROR } } diff --git a/crates/preloop-runner-server/src/distributed_task.rs b/crates/preloop-runner-server/src/distributed_task.rs index 6bfde3f2..55fd1eb6 100644 --- a/crates/preloop-runner-server/src/distributed_task.rs +++ b/crates/preloop-runner-server/src/distributed_task.rs @@ -324,7 +324,17 @@ pub(crate) async fn agent_request_patch( Path((pool_id, request_id)): Path<(i64, i64)>, Json(body): Json, ) -> Json { - info!(?body, "agent_request_patch received"); + let result_hint = body + .get("result") + .and_then(|v| v.as_str()) + .unwrap_or("renew"); + info!( + pool_id, + request_id, + result = %result_hint, + has_result = body.get("result").is_some(), + "agent_request_patch received" + ); // If this is a completion (has result), delegate to complete_job_inner // so summarize_run, promote_ready_jobs, and notify_waiters all fire. // The result field is only present on the final PATCH; renewals have no result. diff --git a/crates/preloop-runner-server/src/results_twirp.rs b/crates/preloop-runner-server/src/results_twirp.rs index dbcfd84c..ca0732f5 100644 --- a/crates/preloop-runner-server/src/results_twirp.rs +++ b/crates/preloop-runner-server/src/results_twirp.rs @@ -683,8 +683,8 @@ pub(crate) async fn twirp_cache_v2_create( inner.cache_v2_pending.insert( token.clone(), CacheV2Pending { - key: storage_key, - version, + key: storage_key.clone(), + version: version.clone(), }, ); let meta = crate::store::build_meta_snapshot(&inner); @@ -710,7 +710,11 @@ pub(crate) async fn twirp_cache_v2_create( )); } let upload_url = format!("{}/twirp-blob/cache/{token}", runner_base_url()); - info!(token, "cache v2 create entry"); + info!( + key = %storage_key, + version = %version, + "cache v2 create entry" + ); Ok(pb_or_json( &headers, PbCreateCacheEntryResponse { @@ -996,8 +1000,14 @@ mod cache_pb_tests { metadata: Some(PbCacheMetadata { repository_id: 42, scope: vec![ - PbCacheScope { scope: "refs/heads/main".to_string(), permission: 1 }, - PbCacheScope { scope: "refs/heads/feature".to_string(), permission: 2 }, + PbCacheScope { + scope: "refs/heads/main".to_string(), + permission: 1, + }, + PbCacheScope { + scope: "refs/heads/feature".to_string(), + permission: 2, + }, ], }), key: "k".to_string(), @@ -1015,7 +1025,10 @@ mod cache_pb_tests { let fixture = include_bytes!("../../../fixtures/wire/cache-multi-scope.pb"); let (_, _, _, fixture_scopes, _) = pb_cache_request(fixture, CacheRequestKind::GetDownloadUrl).unwrap(); - assert_eq!(fixture_scopes, vec!["refs/heads/main", "refs/heads/feature"]); + assert_eq!( + fixture_scopes, + vec!["refs/heads/main", "refs/heads/feature"] + ); } #[test] diff --git a/crates/preloop-runner-server/src/store.rs b/crates/preloop-runner-server/src/store.rs index 56c86110..87b5ff50 100644 --- a/crates/preloop-runner-server/src/store.rs +++ b/crates/preloop-runner-server/src/store.rs @@ -18,8 +18,8 @@ use async_trait::async_trait; use preloop_gha_protocol::SessionId; use rusqlite::{params, Connection, OptionalExtension, Transaction}; use sha2::Digest; -use std::sync::Mutex as StdMutex; use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Mutex as StdMutex; const DATABASE_FILE: &str = "preloop.db"; pub(crate) const SNAPSHOT_FORMAT: u8 = 2; diff --git a/plans/002-observability-strategy.md b/plans/002-observability-strategy.md index 29057fe3..1493a93f 100644 --- a/plans/002-observability-strategy.md +++ b/plans/002-observability-strategy.md @@ -1346,7 +1346,9 @@ Toolchain and existing dependencies verified at `673bdfa0`: Targets: -- add `docs/observability.md` containing the architecture, cardinality rules, log classes, status +- add `docs/internal/observability.md` (internal contract, kept in `docs/internal/` which is + `.gitignore`'d; the public `docs/observability.md` will be a redacted subset later) containing the + architecture, cardinality rules, log classes, status semantics, signal catalog, SLO definitions, deployment profiles, and runbook links from this plan; - re-run the grep in "Logging is local, duplicated, and not export-ready" and fix **every** hit, not just the anchors below — three of the original four moved between `84d92cfd` and `673bdfa0`, so a diff --git a/rules/no-sensitive-log-fields.yml b/rules/no-sensitive-log-fields.yml new file mode 100644 index 00000000..8dacbdff --- /dev/null +++ b/rules/no-sensitive-log-fields.yml @@ -0,0 +1,48 @@ +id: no-sensitive-log-fields +message: | + INFO/WARN/ERROR tracing fields must not carry capability material: token, + authorization, cookie, headers, body, payload, or signed_url. These leak + bearer material into journald and OTLP. Use operation/kind/size/result + fields instead. The conformance flow recorder (recording.rs) is exempt. +severity: error +language: rust +# recording.rs deliberately captures every header and body for conformance. +ignores: + - "**/recording.rs" +rule: + any: + # info!(token, …) / warn!(authorization = …, …) + - pattern: info!($$$ARGS, token, $$$REST) + - pattern: warn!($$$ARGS, token, $$$REST) + - pattern: error!($$$ARGS, token, $$$REST) + - pattern: info!(token, $$$REST) + - pattern: warn!(token, $$$REST) + - pattern: error!(token, $$$REST) + - pattern: info!($$$ARGS, authorization, $$$REST) + - pattern: warn!($$$ARGS, authorization, $$$REST) + - pattern: error!($$$ARGS, authorization, $$$REST) + - pattern: info!($$$ARGS, cookie, $$$REST) + - pattern: warn!($$$ARGS, cookie, $$$REST) + - pattern: error!($$$ARGS, cookie, $$$REST) + - pattern: info!($$$ARGS, headers, $$$REST) + - pattern: warn!($$$ARGS, headers, $$$REST) + - pattern: error!($$$ARGS, headers, $$$REST) + - pattern: info!($$$ARGS, signed_url, $$$REST) + - pattern: warn!($$$ARGS, signed_url, $$$REST) + - pattern: error!($$$ARGS, signed_url, $$$REST) + # ?body / body = … inside the macro + - pattern: info!(?body, $$$REST) + - pattern: warn!(?body, $$$REST) + - pattern: error!(?body, $$$REST) + - pattern: info!(body, $$$REST) + - pattern: warn!(body, $$$REST) + - pattern: error!(body, $$$REST) + - pattern: info!(%body, $$$REST) + - pattern: warn!(%body, $$$REST) + - pattern: error!(%body, $$$REST) + - pattern: info!(payload, $$$REST) + - pattern: warn!(payload, $$$REST) + - pattern: error!(payload, $$$REST) + - pattern: info!(?payload, $$$REST) + - pattern: warn!(?payload, $$$REST) + - pattern: error!(?payload, $$$REST) From 19ca6facc0ac06c398e18c41000a6cca2cdfeb41 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 15:33:25 -0400 Subject: [PATCH 04/22] docs(internal): track observability signal/security contract The public docs/observability.md will be a redacted subset later. This internal contract (docs/internal/ is gitignored) lands before any OTLP export is wired and is the source for cardinality rules, log classes, status semantics, VM contract, and deployment profiles. Tracked via -f like plans/ (both are /-ignored internal notes kept local per .gitignore:132). See plans/002-observability-strategy.md and docs/internal/observability.md for the full internal spec. Entire-Checkpoint: 01M0GAKYM6G0NKH9HMWMFWGJCS --- docs/internal/observability.md | 87 ++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 docs/internal/observability.md diff --git a/docs/internal/observability.md b/docs/internal/observability.md new file mode 100644 index 00000000..b2285a5c --- /dev/null +++ b/docs/internal/observability.md @@ -0,0 +1,87 @@ +# Observability — Internal Signal & Security Contract + +> **Status:** Step 1 of Plan 002 (`plans/002-observability-strategy.md` revised at `673bdfa0`). This is the **internal** contract that must land before any OTLP export is wired. The public `docs/observability.md` will be a redacted subset later. Do not publish this file. + +## Why + +Preloop can report healthy while the pool repeatedly fails to provision, can't distinguish "no runner" from "stale runner," and diverges two servers sharing the same SQLite file. The only aggregate view is `preloop logs`. This contract makes every control-plane question answerable from one endpoint. + +## Architecture — Three Layers + +1. **Zero-dependency diagnostics** — `/healthz`, `/readyz`, `GET /api/v1/status`, `preloop status --json`, structured `stderr`/`journald`, `GET /metrics` (Prometheus text). No sidecar. Answers "why is this job not moving?" with no backend. +2. **Vendor-neutral telemetry** — OpenTelemetry metrics/logs/short traces via bounded `OTLP/HTTP` batches only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. Fail-open: export never rejects/delays a workflow. +3. **Optional reference backend** — pinned single-node OpenObserve (SQLite + local disk, loopback) + importable dashboards. Product boundary is `OTLP + Prometheus`; any backend is interchangeable. + +``` +CLI --bearer--> /api/v1/status +probes --------> /healthz + /readyz +prom ----------> /metrics (bearer) +control plane + pool + host VM sampler --> preloop-observability --> stderr/journald + `--> /metrics + -. OTLP/HTTP .-> OpenObserve (opt-in) +``` + +## Security — What Must Not Be Exported + +**Scrubbed at `673bdfa0`:** + +- `crates/preloop-runner-server/src/artifact_twirp.rs:94` — `info!(token, name)` → `info!(name, workflow_run_backend_id, workflow_job_run_backend_id)` (token removed, keep registry coordinates) +- `crates/preloop-runner-server/src/results_twirp.rs:713` — `info!(token, "cache v2 create")` → `info!(key, version)` (storage identity, not capability) +- `crates/preloop-runner-server/src/blob_store.rs:63,78,95,113,121,127,131` — all `warn!/info!(kind, token)` → `kind` + `block`/`size`/`blocks` (blob operation, not token) +- `crates/preloop-runner-server/src/distributed_task.rs:327` — `info!(?body, …)` full PATCH JSON → `info!(pool_id, request_id, result, has_result)` (bounded enum, not raw body) +- `crates/preloop-runner-server/src/recording.rs:1-90` — **exempt** conformance capture (headers + bodies, mode `0600`, never through OTLP). Rule explicitly ignores this file. + +**Guard:** `rules/no-sensitive-log-fields.yml` rejects `INFO/WARN/ERROR` fields `token`, `authorization`, `cookie`, `headers`, `body`, `payload`, `signed_url` outside `recording.rs`. Run `just sg-scan-strict` — must be `0`. + +**Never log:** raw URLs/query strings, error text into metrics, workflow `stdout/stderr` (stays in `live_logs.rs:25` run-log store, 64 MiB cap), environment values, secrets. + +## Cardinality Rules + +- Allowed metric attributes: bounded enums / finite route templates — HTTP method + matched `route` (never raw URI), queue kind, pool mode/state, backend, `limit` constant name (finite set), `task` registry name (finite set). +- Forbidden: `run_id`, `job_id`, `runner_id`, `machine_name`, repo/workflow/ref/SHA, `runs-on` values, cache keys, artifact names, raw URL/query, `x-delivery-id`, tokens. +- IDs belong in **logs/traces only** (`event.name` + `run.id`/`job.id`/`machine.name`), correlated via `trace_id`/`span_id`. +- Test: drive 1,000 distinct IDs through instrumentation, gather Prometheus registry, assert fixed series count, no sentinel in exposition. + +## Status Semantics + +- **`GET /healthz`** — public, shallow, lock-free. `200` while process serves, `503` during shutdown. Fields: `schema_version`, `ok`, `protocol_version`, `shutdown_requested`. Touches nothing (no DB, no SmolVM, no `InnerState`). +- **`GET /readyz`** — public, reason codes. `200` when durable state restored + every **critical** `TaskHeartbeat` fresh (state sampler, reaper `bootstrap.rs:396`, scheduler scan `bootstrap.rs:479`, PG `store_pg.rs:95`). `503` → `starting | task_stale{task} | shutting_down`. Non-critical staleness does not gate readiness. +- **`GET /api/v1/status`** — `bearer` required, `schema_version: 1`, sampled every 5s without holding `InnerState`, reports `snapshot_age_seconds`. Full shape in `plans/002-observability-strategy.md` § Operator surfaces — includes `jobs`, `concurrency`, `scheduler`, `pool`, `vms` (with `capabilities` + 5 top consumers), `store`, `storage`, `limits[]`, `tasks[]`, `github` (with `rate_limit`, `token_cache`), `conditions[]` (bounded stable codes, ≤5 exemplars, safe messages). +- **`preloop status`** — unit variant → `Status(StatusArgs)` with `--json` (shape of `PlanArgs` at `main.rs:706`). Human output ordered 1..10 (service → queue → concurrency/scheduler → pool/runners → VM fleet → store/storage/GitHub/debug/telemetry → non-zero limits/tasks → conditions → recent runs). `--json` prints the endpoint body exactly. +- **`GET /metrics`** — `bearer` required, Prometheus text from the same snapshot, never per-machine labels. + +`wait_for_engine_socket` (`main.rs:1458`) now probes `/readyz` and surfaces the last `reason` on 30s timeout. + +## Log Classes + +- `INFO` — low-frequency control-plane/pool/scheduler transitions +- `WARN` — recoverable degradation (`vm.host.memory.pressure`, `limit.exceeded{limit="QUEUE_MAX_PENDING"}`, `github.rate_limit.low`, `vm.unreachable`) +- `ERROR` — supervisor death, invariant break, `store.connection.lost` (PG), `task.exited{critical=true}` +- `DEBUG` — successful long-poll/renew (never `INFO` per poll) + +Catalog (excerpt): `job.concurrency.cancelled` (hash of group key, not raw — branch names are PII), `schedule.fired/skipped`, `task.exited`, `limit.exceeded` (rate-limited), `vm.unreachable/reachable`, `store.connection.lost`, `storage.pressure`, `github.rate_limit.low`, `debug.audit.evicted`. + +## VM Telemetry Contract + +- `VmProvider::status` at `preloop-vm/src/lib.rs:1018` today substring-matches human text — replaced with typed `machine status --json` (one machine on lifecycle path) + `machine ls --json` (whole fleet, one subprocess per 60s slow pass, already parsed in `list` at `1045`) + `machine data-dir` for disk paths, at `smolvm 1.8.1` floor (`versions.toml:8`, `machine_status_json` at `src/cli/vm_common.rs:2360`). +- `state` includes `Unreachable` (vsock probe, `src/agent/state_probe.rs:41`) → `MachineState::Unreachable` metric + `vm_unreachable` condition. A missing `vm-` cgroup leaf (SmolVM creates it, `src/process.rs:375`) degrades to process fallback, `capability=false`, never zero. +- Host-only: `cpu.stat usage_usec`, `memory.current`, `pids.current`, `st_blocks` + `statvfs`. Never guest `free/df/ps`. + +## Deployment Profiles + +- **Default local** — no backend, pretty on TTY / JSON when piped, `PRELOOP_LOG_FORMAT=auto`. +- **Local enhanced** — single OpenObserve container, loopback, persistent volume, short retention, resource caps. +- **Self-hosted single node** — Preloop + OpenObserve may share a host only with separate volumes + caps. +- **Existing estate** — point same OTLP at your backend. +- **Optional Collector** — whole-node host telemetry + fan-out (VM `preloop.vm.*` does not need it). +- **Quickwit+S3 alternative** — `Vector → Quickwit → S3` for cheap searchable logs/traces; metrics stay on `Mimir`/`VictoriaMetrics` until Quickwit metrics matures. + +## Flow Capture + +`recording.rs` is an explicit local conformance facility, stored `0600`, never through the normal logging/OTLP pipeline. Verified at creation. + +## References + +- Plan: `plans/002-observability-strategy.md` (and HTML companion with mockups) +- Code anchors re-verified at `673bdfa0`; re-run `git diff --stat 673bdfa0..HEAD -- …` before each step +- Rules mirror `rules/no-expose-in-loop.yml` / `no-raw-secret-replace.yml` / `no-inline-masking.yml` From da68f73e716fd0a94633a30e69a46ff14e456c25 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 15:59:14 -0400 Subject: [PATCH 05/22] feat(observability): add observability crate and unify process init MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Create crates/preloop-observability with the small explicit API from Plan 002: ObservabilityConfig::from_env (PRELOOP_LOG_FORMAT auto/pretty/json, RUST_LOG default info, OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_* with redacted Debug), Observability::noop (no socket), Observability::from_config + ObservabilityRuntime (2s bounded shutdown), TaskHeartbeat registry (critical vs non-critical, Drop deregisters, any_critical_stale) and LimitRegistry (register/record_drop/record_reject with &'static str keys). No SDK, no OTLP http-proto, no Prometheus reader yet — host-only features are deferred; the handle is cloneable into AppState and RunnerPoolConfig and makes tests perform no network I/O. Wire the same init into all three binaries: - preloop-cli/src/main.rs: replace fmt::init with Observability::from_config + install_fmt_subscriber, hold handle for later pool wiring - preloop-runner-server/src/main.rs: same, fixing the missing info fallback (from_default_env with no default hid pool faults) - preloop-runner/src/main.rs: structured local logger only, never export PRELOOP_LOG_FORMAT auto now means pretty on TTY and JSON when piped (no ANSI in journald), via IsTerminal. Verify: cargo test -p preloop-observability 9 passed (noop, absent endpoint disabled not localhost, none disables, heartbeat deregister, critical staleness, limit counts, Debug redaction, shutdown bounded); cargo check --locked --workspace and cargo check -p preloop-cli -p preloop-runner-server -p preloop-runner all pass; just sg-scan-strict and cargo fmt --all --check pass. Entire-Checkpoint: 01M0GC36ZYTA6J8Q1E0HY264R1 --- Cargo.lock | 15 + Cargo.toml | 1 + crates/preloop-cli/Cargo.toml | 1 + crates/preloop-cli/src/main.rs | 25 +- crates/preloop-observability/Cargo.toml | 22 + crates/preloop-observability/src/lib.rs | 696 +++++++++++++++++++++++ crates/preloop-runner-server/Cargo.toml | 1 + crates/preloop-runner-server/src/main.rs | 13 +- crates/preloop-runner/Cargo.toml | 1 + crates/preloop-runner/src/main.rs | 14 +- 10 files changed, 769 insertions(+), 20 deletions(-) create mode 100644 crates/preloop-observability/Cargo.toml create mode 100644 crates/preloop-observability/src/lib.rs diff --git a/Cargo.lock b/Cargo.lock index 63a2e8ba..e2f30bb8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1875,6 +1875,7 @@ dependencies = [ "libc", "preloop-gha-parser", "preloop-gha-protocol", + "preloop-observability", "preloop-orchestrator", "preloop-runner-server", "preloop-vm", @@ -2015,6 +2016,18 @@ dependencies = [ "uuid", ] +[[package]] +name = "preloop-observability" +version = "0.1.0" +dependencies = [ + "anyhow", + "parking_lot", + "tokio", + "tracing", + "tracing-subscriber", + "uuid", +] + [[package]] name = "preloop-orchestrator" version = "0.21.0" @@ -2061,6 +2074,7 @@ dependencies = [ "preloop-gha-expressions", "preloop-gha-parser", "preloop-gha-protocol", + "preloop-observability", "proptest", "rand 0.8.6", "regex", @@ -2124,6 +2138,7 @@ dependencies = [ "preloop-gha-expressions", "preloop-gha-parser", "preloop-gha-protocol", + "preloop-observability", "preloop-runner", "preloop-socket-activation", "proptest", diff --git a/Cargo.toml b/Cargo.toml index e886d32f..96a1fb03 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -15,6 +15,7 @@ members = [ "crates/preloop-orchestrator", "crates/preloop-socket-activation", "crates/preloop-cli", + "crates/preloop-observability", "benchmarks/preloop-perf", ] resolver = "2" diff --git a/crates/preloop-cli/Cargo.toml b/crates/preloop-cli/Cargo.toml index ca3c150b..234d484a 100644 --- a/crates/preloop-cli/Cargo.toml +++ b/crates/preloop-cli/Cargo.toml @@ -15,6 +15,7 @@ name = "preloop" path = "src/main.rs" [dependencies] +preloop-observability = { path = "../preloop-observability" } preloop-orchestrator = { path = "../preloop-orchestrator" } preloop-vm = { path = "../preloop-vm" } preloop-runner-server = { path = "../preloop-runner-server" } diff --git a/crates/preloop-cli/src/main.rs b/crates/preloop-cli/src/main.rs index 3d00e702..d5b0816b 100644 --- a/crates/preloop-cli/src/main.rs +++ b/crates/preloop-cli/src/main.rs @@ -736,17 +736,20 @@ struct ShellArgs { #[tokio::main] async fn main() -> anyhow::Result<()> { - // `fmt::init()` alone filters to ERROR when `RUST_LOG` is unset, which hid - // a runner pool that failed to provision 77 times in a row: every - // provisioning fault logs at `warn` or `info`, so the operator saw a server - // that accepted webhooks and silently never ran anything. Default to `info` - // and let `RUST_LOG` override as usual. - tracing_subscriber::fmt() - .with_env_filter( - tracing_subscriber::EnvFilter::try_from_default_env() - .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")), - ) - .init(); + // Unified observability init (Step 2): one handle for the process, shared + // with `AppState` and `RunnerPoolConfig` later. `PRELOOP_LOG_FORMAT` now + // controls pretty/json/auto (auto = pretty on TTY, JSON when piped), and + // `RUST_LOG` defaults to `info` (the old `fmt::init()` default of ERROR hid + // pool provisioning faults). The runtime is held for the life of `main` + // and flushed with a bounded 2s shutdown on exit. + let obs_config = preloop_observability::ObservabilityConfig::from_env(); + let (observability, observability_runtime) = + preloop_observability::Observability::from_config(obs_config); + preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); + // Keep the handle alive; the pool/server will clone it later. Suppress + // unused warning until the wiring lands in Step 3. + let _observability = observability; + let _observability_runtime = observability_runtime; let cli = Cli::parse(); // One config path for the whole process. `setup`/`doctor`/`secret` return diff --git a/crates/preloop-observability/Cargo.toml b/crates/preloop-observability/Cargo.toml new file mode 100644 index 00000000..dea5e730 --- /dev/null +++ b/crates/preloop-observability/Cargo.toml @@ -0,0 +1,22 @@ +[package] +name = "preloop-observability" +version = "0.1.0" +edition = "2021" +license = "MIT" +description = "Observability handle for Preloop — metrics, logs, traces, status snapshot (Step 2)" +repository = "https://github.com/preloopdev/preloop" + +[dependencies] +anyhow = { workspace = true } +parking_lot = { workspace = true } +tracing = { workspace = true } +tracing-subscriber = { workspace = true } +uuid = { workspace = true } +tokio = { workspace = true } + +[dev-dependencies] +tokio = { workspace = true } +tracing = { workspace = true } + +[lints] +workspace = true diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs new file mode 100644 index 00000000..1d926147 --- /dev/null +++ b/crates/preloop-observability/src/lib.rs @@ -0,0 +1,696 @@ +//! `preloop-observability` — Step 2 of Plan 002. +//! +//! Small, explicit API with no dependency on server/orchestrator internals. Both +//! `preloop` and `preloop-server` construct one handle/runtime before building +//! `ServerConfig`; the handle is cloned into `AppState` and `RunnerPoolConfig`. +//! Tests use `Observability::noop()` which performs no network I/O. +//! +//! Invariants from the plan: +//! - Fail open: export failure never rejects a workflow. +//! - Bounded queues, 2s flush, no backend by default (absent `OTEL_EXPORTER_OTLP_*` = disabled, not `localhost:4318`). +//! - Always retain `stderr`/`journald` even when OTLP is configured. +//! - `Debug` on config never reveals headers or credential-bearing endpoint parts. + +use std::collections::HashMap; +use std::fmt; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use parking_lot::RwLock; +use uuid::Uuid; + +// --------------------------------------------------------------------------- +// Log format +// --------------------------------------------------------------------------- + +/// How `tracing_subscriber::fmt` should render. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LogFormat { + /// Pretty on TTY, JSON when piped / in journald. + Auto, + Pretty, + Json, +} + +impl LogFormat { + fn parse(s: &str) -> Option { + match s.to_ascii_lowercase().as_str() { + "auto" => Some(Self::Auto), + "pretty" => Some(Self::Pretty), + "json" => Some(Self::Json), + _ => None, + } + } + + /// Resolve `Auto` to a concrete format for the current stderr. + pub fn resolve(self) -> Self { + if self != Self::Auto { + return self; + } + if std::io::IsTerminal::is_terminal(&std::io::stderr()) { + Self::Pretty + } else { + Self::Json + } + } +} + +// --------------------------------------------------------------------------- +// ObservabilityConfig +// --------------------------------------------------------------------------- + +/// Parsed logging + OTel configuration. `Debug` is redacted. +pub struct ObservabilityConfig { + /// `PRELOOP_LOG_FORMAT` resolved to concrete `LogFormat` (but `Auto` is kept for display). + pub log_format: LogFormat, + /// Effective `RUST_LOG` filter string (default `info` when unset, matching CLI behaviour). + pub rust_log: String, + /// `service.name` — `preloop` or `OTEL_SERVICE_NAME`. + pub service_name: String, + /// Per-process instance ID (UUID v4). + pub instance_id: String, + /// `OTEL_EXPORTER_OTLP_ENDPOINT` or signal-specific variant, if any. Kept as + /// given for transport, but `Debug` redacts userinfo/query. + otel_endpoint: Option, + /// `OTEL_EXPORTER_OTLP_HEADERS` or signal-specific variant, if any. Never shown in `Debug` or errors. + otel_headers: Option, + /// Whether any OTLP endpoint is present (i.e. export enabled). + pub otlp_enabled: bool, +} + +impl ObservabilityConfig { + /// Read `PRELOOP_LOG_FORMAT`, `RUST_LOG`, and standard `OTEL_*` vars. + /// + /// `Debug` and error paths never expose `OTEL_EXPORTER_OTLP_HEADERS` values + /// or credential-bearing endpoint components (userinfo/query). + pub fn from_env() -> Self { + let log_format = std::env::var("PRELOOP_LOG_FORMAT") + .ok() + .and_then(|v| LogFormat::parse(&v)) + .unwrap_or(LogFormat::Auto); + + // CLI defaults to `info` when unset; the standalone server historically + // used `EnvFilter::from_default_env()` with no fallback (silent when + // unset). We unify on `info` per Step 2. + let rust_log = std::env::var("RUST_LOG").unwrap_or_else(|_| "info".to_string()); + + let service_name = + std::env::var("OTEL_SERVICE_NAME").unwrap_or_else(|_| "preloop".to_string()); + + // Endpoint: generic or signal-specific. Presence — not value — enables export. + // This is a deliberate deviation from the OTel spec default `http://localhost:4318`. + let otel_endpoint = std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT") + .or_else(|_| std::env::var("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT")) + .or_else(|_| std::env::var("OTEL_EXPORTER_OTLP_METRICS_ENDPOINT")) + .or_else(|_| std::env::var("OTEL_EXPORTER_OTLP_LOGS_ENDPOINT")) + .ok() + .filter(|v| !v.trim().is_empty() && v.trim() != "none"); + + let otel_headers = std::env::var("OTEL_EXPORTER_OTLP_HEADERS") + .or_else(|_| std::env::var("OTEL_EXPORTER_OTLP_TRACES_HEADERS")) + .or_else(|_| std::env::var("OTEL_EXPORTER_OTLP_METRICS_HEADERS")) + .or_else(|_| std::env::var("OTEL_EXPORTER_OTLP_LOGS_HEADERS")) + .ok() + .filter(|v| !v.trim().is_empty()); + + let otlp_enabled = otel_endpoint.is_some(); + let instance_id = Uuid::new_v4().to_string(); + + Self { + log_format, + rust_log, + service_name, + instance_id, + otel_endpoint, + otel_headers, + otlp_enabled, + } + } + + /// Raw endpoint if export is enabled, for transport construction. + pub fn otel_endpoint_raw(&self) -> Option<&str> { + self.otel_endpoint.as_deref() + } + + /// Whether any `OTEL_EXPORTER_OTLP_HEADERS` was supplied (for health reporting). + pub fn has_otel_headers(&self) -> bool { + self.otel_headers.is_some() + } + + /// Sanitized endpoint for `Debug`/errors: strips userinfo and query. + fn sanitized_endpoint(&self) -> Option { + self.otel_endpoint.as_ref().map(|raw| { + // Best-effort: hide `user:pass@` and `?...` without a URL parser dep. + let without_query = raw.split('?').next().unwrap_or(raw); + if let Some(at) = without_query.rfind('@') { + // Keep scheme + host/path, hide userinfo. + if let Some(scheme_end) = without_query.find("://") { + let scheme = &without_query[..scheme_end + 3]; + return format!("{scheme}***@{}", &without_query[at + 1..]); + } + return format!("***@{}", &without_query[at + 1..]); + } + without_query.to_string() + }) + } +} + +impl fmt::Debug for ObservabilityConfig { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ObservabilityConfig") + .field("log_format", &self.log_format) + .field("rust_log", &self.rust_log) + .field("service_name", &self.service_name) + .field("instance_id", &self.instance_id) + .field( + "otel_endpoint", + &self.sanitized_endpoint().map(|_| ""), + ) + .field( + "otel_headers", + &self.otel_headers.as_ref().map(|_| ""), + ) + .field("otlp_enabled", &self.otlp_enabled) + .finish() + } +} + +// --------------------------------------------------------------------------- +// TaskHeartbeat registry +// --------------------------------------------------------------------------- + +/// How a task gates `/readyz`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Criticality { + /// Staleness returns 503 from `/readyz`. + Critical, + /// Staleness surfaces only in `/api/v1/status` and metrics. + NonCritical, +} + +/// Registry of long-lived background tasks. Generic replacement for per-task +/// `AtomicU64` timestamps; every `tokio::spawn` that outlives a request +/// registers here per invariant 15. +#[derive(Debug, Clone, Default)] +pub struct TaskHeartbeat { + inner: Arc>>, +} + +#[derive(Debug, Clone)] +struct HeartbeatEntry { + critical: Criticality, + last_beat: Instant, + /// Whether the task has exited cleanly (Drop without panic). + exited: bool, +} + +impl TaskHeartbeat { + /// Register a task. Returns a guard — `Drop` deregisters. + pub fn register(&self, name: &'static str, critical: Criticality) -> HeartbeatHandle { + self.inner.write().insert( + name, + HeartbeatEntry { + critical, + last_beat: Instant::now(), + exited: false, + }, + ); + HeartbeatHandle { + registry: self.clone(), + name, + } + } + + /// Record a beat for `name`. No-op if not registered (so tests can `noop()` without registering). + pub fn beat(&self, name: &'static str) { + if let Some(entry) = self.inner.write().get_mut(name) { + entry.last_beat = Instant::now(); + } + } + + pub(crate) fn deregister(&self, name: &'static str) { + self.inner.write().remove(name); + } + + pub(crate) fn mark_exited(&self, name: &'static str) { + if let Some(entry) = self.inner.write().get_mut(name) { + entry.exited = true; + } + } + + /// Snapshot for `/readyz` and `/api/v1/status`. + pub fn snapshot(&self) -> Vec { + self.inner + .read() + .iter() + .map(|(name, e)| TaskSnapshot { + name, + critical: e.critical, + heartbeat_age: e.last_beat.elapsed(), + exited: e.exited, + }) + .collect() + } + + /// Whether any critical task is stale beyond `threshold`. + pub fn any_critical_stale(&self, threshold: Duration) -> Option<&'static str> { + // Hold read lock across iteration to avoid TOCTOU. + let guard = self.inner.read(); + for (name, e) in guard.iter() { + if e.critical == Criticality::Critical && !e.exited && e.last_beat.elapsed() > threshold + { + return Some(*name); + } + } + None + } + + /// Number of registered tasks (for tests). + #[cfg(test)] + pub fn len(&self) -> usize { + self.inner.read().len() + } +} + +/// Guard — `beat()` updates, `Drop` deregisters. +pub struct HeartbeatHandle { + registry: TaskHeartbeat, + name: &'static str, +} + +impl HeartbeatHandle { + pub fn beat(&self) { + self.registry.beat(self.name); + } +} + +impl Drop for HeartbeatHandle { + fn drop(&mut self) { + self.registry.deregister(self.name); + } +} + +impl fmt::Debug for HeartbeatHandle { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("HeartbeatHandle") + .field("name", &self.name) + .finish() + } +} + +#[derive(Debug, Clone)] +pub struct TaskSnapshot { + pub name: &'static str, + pub critical: Criticality, + pub heartbeat_age: Duration, + pub exited: bool, +} + +// --------------------------------------------------------------------------- +// LimitRegistry +// --------------------------------------------------------------------------- + +/// Bounded-cap registry. `limit` is a `&'static str` constant name (finite set), never a value. +#[derive(Debug, Clone, Default)] +pub struct LimitRegistry { + inner: Arc>>, +} + +#[derive(Debug, Clone, Default)] +struct LimitEntry { + value: usize, + dropped: u64, + rejected: u64, +} + +impl LimitRegistry { + /// Register a cap with its configured ceiling. Idempotent. + pub fn register(&self, limit: &'static str, value: usize) { + self.inner.write().entry(limit).or_insert(LimitEntry { + value, + ..Default::default() + }); + // Update value if re-registered with different ceiling (for tests). + if let Some(entry) = self.inner.write().get_mut(limit) { + entry.value = value; + } + } + + pub fn record_drop(&self, limit: &'static str, n: u64) { + if let Some(entry) = self.inner.write().get_mut(limit) { + entry.dropped = entry.dropped.saturating_add(n); + } + } + + pub fn record_reject(&self, limit: &'static str) { + if let Some(entry) = self.inner.write().get_mut(limit) { + entry.rejected = entry.rejected.saturating_add(1); + } + } + + pub fn snapshot(&self) -> Vec { + self.inner + .read() + .iter() + .map(|(limit, e)| LimitSnapshot { + limit, + value: e.value, + dropped: e.dropped, + rejected: e.rejected, + }) + .collect() + } +} + +#[derive(Debug, Clone)] +pub struct LimitSnapshot { + pub limit: &'static str, + pub value: usize, + pub dropped: u64, + pub rejected: u64, +} + +// --------------------------------------------------------------------------- +// Observability handle + Runtime +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone)] +struct Inner { + config: Arc, + heartbeat: TaskHeartbeat, + limits: LimitRegistry, + is_noop: bool, +} + +/// Cloneable handle — cheap to clone into `AppState` and `RunnerPoolConfig`. +#[derive(Debug, Clone)] +pub struct Observability { + inner: Arc, +} + +impl Observability { + /// Allocation-light handle for tests and library-only consumers. Performs no I/O, no socket. + pub fn noop() -> Self { + let config = ObservabilityConfig { + log_format: LogFormat::Auto, + rust_log: "info".to_string(), + service_name: "preloop".to_string(), + instance_id: Uuid::new_v4().to_string(), + otel_endpoint: None, + otel_headers: None, + otlp_enabled: false, + }; + Self { + inner: Arc::new(Inner { + config: Arc::new(config), + heartbeat: TaskHeartbeat::default(), + limits: LimitRegistry::default(), + is_noop: true, + }), + } + } + + /// Real handle from `ObservabilityConfig`. Does not install the global subscriber — pair with `ObservabilityRuntime`. + pub fn from_config(config: ObservabilityConfig) -> (Self, ObservabilityRuntime) { + let is_noop = !config.otlp_enabled; + let handle = Self { + inner: Arc::new(Inner { + config: Arc::new(config), + heartbeat: TaskHeartbeat::default(), + limits: LimitRegistry::default(), + is_noop, + }), + }; + let runtime = ObservabilityRuntime::new(handle.clone()); + (handle, runtime) + } + + pub fn is_noop(&self) -> bool { + self.inner.is_noop + } + + pub fn otlp_enabled(&self) -> bool { + self.inner.config.otlp_enabled + } + + pub fn instance_id(&self) -> &str { + &self.inner.config.instance_id + } + + pub fn service_name(&self) -> &str { + &self.inner.config.service_name + } + + pub fn heartbeat(&self) -> &TaskHeartbeat { + &self.inner.heartbeat + } + + pub fn limits(&self) -> &LimitRegistry { + &self.inner.limits + } + + pub fn config(&self) -> &ObservabilityConfig { + &self.inner.config + } +} + +/// Owns subscriber/provider guards and performs bounded shutdown/flush. +/// +/// On `Drop`, attempts to flush for at most 2s per invariant 3, then exits. +/// Tests use scoped subscribers and never install the global one twice. +pub struct ObservabilityRuntime { + _handle: Observability, + // Hold the tracing guard so it isn't dropped early when we use a + // non-global dispatcher in tests. For the global install, this is `None` + // and the global dispatcher owns the guard. + _guard: Option< + tracing_subscriber::reload::Handle< + tracing_subscriber::EnvFilter, + tracing_subscriber::Registry, + >, + >, +} + +impl ObservabilityRuntime { + fn new(handle: Observability) -> Self { + // Step 2 does not install the global subscriber here — the binaries do + // that via `install_fmt_subscriber`. This runtime is the place for the + // future OTel provider guards and the 2s flush on `Drop`. + Self { + _handle: handle, + _guard: None, + } + } + + /// Install the global `tracing_subscriber::fmt` layer once, respecting + /// `PRELOOP_LOG_FORMAT` and `RUST_LOG` from `config`. Safe to call at most + /// once per process; tests use `install_fmt_subscriber_for_test` instead. + pub fn install_fmt_subscriber(config: &ObservabilityConfig) { + let filter = tracing_subscriber::EnvFilter::try_new(&config.rust_log) + .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")); + let fmt = config.log_format.resolve(); + match fmt { + LogFormat::Json => { + let subscriber = tracing_subscriber::fmt() + .with_env_filter(filter) + .json() + .with_current_span(true) + .with_span_list(true) + .finish(); + let _ = tracing::subscriber::set_global_default(subscriber); + } + LogFormat::Pretty | LogFormat::Auto => { + let subscriber = tracing_subscriber::fmt() + .with_env_filter(filter) + .with_ansi(std::io::IsTerminal::is_terminal(&std::io::stderr())) + .finish(); + let _ = tracing::subscriber::set_global_default(subscriber); + } + } + } + + /// Attempt to flush exporters for at most 2s. Export failure is logged, never propagated. + pub async fn shutdown(self) { + // Step 2 has no exporter worker yet; this is the bounded-flush seam for + // the future OTel BatchSpanProcessor / metrics reader. + tokio::time::timeout(Duration::from_secs(2), async { + // No-op until OTel providers are wired in Step 3+. + tokio::task::yield_now().await; + }) + .await + .ok(); + } +} + +impl fmt::Debug for ObservabilityRuntime { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ObservabilityRuntime").finish() + } +} + +// --------------------------------------------------------------------------- +// Tests — Step 2 gates +// --------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn noop_performs_no_network_io() { + let obs = Observability::noop(); + assert!(obs.is_noop()); + assert!(!obs.otlp_enabled()); + assert!(!obs.instance_id().is_empty()); + } + + #[test] + fn absent_endpoint_means_disabled_not_localhost() { + // Ensure no ambient OTEL vars leak into the test. + for k in [ + "OTEL_EXPORTER_OTLP_ENDPOINT", + "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", + "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT", + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT", + ] { + std::env::remove_var(k); + } + // Also headers, so `has_otel_headers` is false. + for k in [ + "OTEL_EXPORTER_OTLP_HEADERS", + "OTEL_EXPORTER_OTLP_TRACES_HEADERS", + "OTEL_EXPORTER_OTLP_METRICS_HEADERS", + "OTEL_EXPORTER_OTLP_LOGS_HEADERS", + ] { + std::env::remove_var(k); + } + let cfg = ObservabilityConfig::from_env(); + assert!( + !cfg.otlp_enabled, + "absent endpoint must be disabled, not localhost:4318" + ); + assert!(cfg.otel_endpoint_raw().is_none()); + let (obs, _rt) = Observability::from_config(cfg); + assert!(!obs.otlp_enabled()); + assert!(obs.is_noop()); + } + + #[test] + fn none_disables_signal() { + std::env::set_var("OTEL_EXPORTER_OTLP_ENDPOINT", "none"); + let cfg = ObservabilityConfig::from_env(); + assert!(!cfg.otlp_enabled, "`none` must disable export"); + std::env::remove_var("OTEL_EXPORTER_OTLP_ENDPOINT"); + } + + #[test] + fn heartbeat_register_beat_deregister() { + let obs = Observability::noop(); + assert_eq!(obs.heartbeat().len(), 0); + { + let h = obs.heartbeat().register("reaper", Criticality::Critical); + assert_eq!(obs.heartbeat().len(), 1); + h.beat(); + assert_eq!(obs.heartbeat().len(), 1); + // staleness: threshold 50ms, just-beat handle is fresh. + assert!(obs + .heartbeat() + .any_critical_stale(Duration::from_millis(50)) + .is_none()); + } + assert_eq!(obs.heartbeat().len(), 0, "Drop must deregister"); + } + + #[test] + fn critical_stale_detection() { + let obs = Observability::noop(); + let _h = obs + .heartbeat() + .register("scheduler_scan", Criticality::Critical); + // Sleep past threshold — stale. + std::thread::sleep(Duration::from_millis(20)); + assert_eq!( + obs.heartbeat().any_critical_stale(Duration::from_millis(5)), + Some("scheduler_scan") + ); + // Non-critical with same age must not gate readiness. + let obs2 = Observability::noop(); + let _h2 = obs2 + .heartbeat() + .register("snapshot_gc", Criticality::NonCritical); + std::thread::sleep(Duration::from_millis(20)); + assert_eq!( + obs2.heartbeat() + .any_critical_stale(Duration::from_millis(5)), + None + ); + } + + #[test] + fn limit_registry_counts() { + let obs = Observability::noop(); + obs.limits().register("QUEUE_MAX_PENDING", 100); + obs.limits() + .register("LIVE_LOG_MAX_BYTES", 64 * 1024 * 1024); + obs.limits().record_reject("QUEUE_MAX_PENDING"); + obs.limits().record_drop("LIVE_LOG_MAX_BYTES", 3); + let snap = obs.limits().snapshot(); + let q = snap + .iter() + .find(|s| s.limit == "QUEUE_MAX_PENDING") + .unwrap(); + assert_eq!(q.value, 100); + assert_eq!(q.rejected, 1); + let l = snap + .iter() + .find(|s| s.limit == "LIVE_LOG_MAX_BYTES") + .unwrap(); + assert_eq!(l.dropped, 3); + } + + #[test] + fn debug_redacts_headers_and_endpoint_userinfo() { + std::env::set_var( + "OTEL_EXPORTER_OTLP_ENDPOINT", + "https://user:secret@example.com:4318/v1/traces?token=abc", + ); + std::env::set_var( + "OTEL_EXPORTER_OTLP_HEADERS", + "Authorization=Bearer secret123", + ); + let cfg = ObservabilityConfig::from_env(); + let dbg = format!("{cfg:?}"); + assert!( + !dbg.contains("secret"), + "Debug must not contain secret material: {dbg}" + ); + assert!( + !dbg.contains("Authorization"), + "Debug must not contain header values: {dbg}" + ); + assert!( + !dbg.contains("user:"), + "Debug must not contain userinfo: {dbg}" + ); + std::env::remove_var("OTEL_EXPORTER_OTLP_ENDPOINT"); + std::env::remove_var("OTEL_EXPORTER_OTLP_HEADERS"); + } + + #[test] + fn log_format_auto_resolves() { + assert_eq!(LogFormat::parse("auto"), Some(LogFormat::Auto)); + assert_eq!(LogFormat::parse("PRETTY"), Some(LogFormat::Pretty)); + assert_eq!(LogFormat::parse("json"), Some(LogFormat::Json)); + assert_eq!(LogFormat::parse("bogus"), None); + } + + #[tokio::test] + async fn shutdown_is_bounded() { + let (obs, rt) = Observability::from_config(ObservabilityConfig::from_env()); + // Must not hang even though there's no exporter. + tokio::time::timeout(Duration::from_secs(3), rt.shutdown()) + .await + .expect("shutdown must be bounded to 2s"); + drop(obs); + } +} diff --git a/crates/preloop-runner-server/Cargo.toml b/crates/preloop-runner-server/Cargo.toml index 0e7707da..b84c32b5 100644 --- a/crates/preloop-runner-server/Cargo.toml +++ b/crates/preloop-runner-server/Cargo.toml @@ -22,6 +22,7 @@ clap.workspace = true hyper.workspace = true hyper-util.workspace = true tower.workspace = true +preloop-observability = { path = "../preloop-observability" } preloop-artifacts = { path = "../preloop-artifacts" } preloop-cache = { path = "../preloop-cache" } preloop-gha-parser = { path = "../preloop-gha-parser" } diff --git a/crates/preloop-runner-server/src/main.rs b/crates/preloop-runner-server/src/main.rs index 12a54066..aed62825 100644 --- a/crates/preloop-runner-server/src/main.rs +++ b/crates/preloop-runner-server/src/main.rs @@ -75,9 +75,16 @@ async fn main() -> anyhow::Result<()> { .install_default() .ok(); - tracing_subscriber::fmt() - .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) - .init(); + // Unified observability init (Step 2): `RUST_LOG` now defaults to `info` + // like the CLI, instead of falling silent when unset. `PRELOOP_LOG_FORMAT` + // controls pretty/json/auto. The `Observability` handle will be cloned + // into `AppState` in Step 3; for now it is held for the life of `main`. + let obs_config = preloop_observability::ObservabilityConfig::from_env(); + let (observability, observability_runtime) = + preloop_observability::Observability::from_config(obs_config); + preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); + let _observability = observability; + let _observability_runtime = observability_runtime; let cli = Cli::parse(); match cli.command { diff --git a/crates/preloop-runner/Cargo.toml b/crates/preloop-runner/Cargo.toml index 60003cf3..163a98fd 100644 --- a/crates/preloop-runner/Cargo.toml +++ b/crates/preloop-runner/Cargo.toml @@ -15,6 +15,7 @@ path = "src/main.rs" workspace = true [dependencies] +preloop-observability = { path = "../preloop-observability" } preloop-gha-protocol = { path = "../preloop-gha-protocol" } preloop-gha-expressions = { path = "../preloop-gha-expressions" } preloop-gha-parser = { path = "../preloop-gha-parser" } diff --git a/crates/preloop-runner/src/main.rs b/crates/preloop-runner/src/main.rs index d67e7bad..a7c5a687 100644 --- a/crates/preloop-runner/src/main.rs +++ b/crates/preloop-runner/src/main.rs @@ -14,12 +14,14 @@ const MAX_REUSABLE_WORKFLOW_DEPTH: usize = 4; async fn main() -> Result<()> { let cli = Cli::parse(); - tracing_subscriber::fmt() - .with_env_filter( - tracing_subscriber::EnvFilter::try_from_default_env() - .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")), - ) - .init(); + // Runner gets structured local logging only — never OTLP export by default. + // `PRELOOP_LOG_FORMAT` still controls pretty/json/auto for consistency. + let obs_config = preloop_observability::ObservabilityConfig::from_env(); + let (observability, observability_runtime) = + preloop_observability::Observability::from_config(obs_config); + preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); + let _observability = observability; + let _observability_runtime = observability_runtime; match cli.command { Commands::Configure(args) => { From cdab10d5b3094fb78e75039a7b4c5f2ac213ce83 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 16:43:37 -0400 Subject: [PATCH 06/22] feat(server,cli): add liveness, readiness, aggregate status and metrics, and preloop status MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the zero-dependency operator surfaces. healthz stays public and lock-free, now 503 during shutdown; add public readyz that gates on critical heartbeats (state sampler, reaper, scheduler, pg connection) and sampler age >15s staleness; add authenticated GET /api/v1/status (versioned OperationalSnapshot, 5s cached sampler, no InnerState lock, snapshot_age, bounded exemplars) and GET /metrics (Prometheus text with preloop_service_uptime and pool/queue gauges, bearer required). Sampler runs every 5s in bootstrap, clones queue/runner/pool state under lock then releases before building the snapshot, beats a TaskHeartbeat and falls back to the last good snapshot when the lock is contended. Reaper and scheduler scan also beat their heartbeats without changing cadence. Route /healthz and /readyz public, /api/v1/status and /metrics behind require_native_bearer, TraceLayer stays last. OpenAPI documents the three new routes. Consolidate pool state: new PoolStatus handle in preloop-observability replaces the four ad-hoc Option> channels; RunnerPoolConfig now carries pool_status Option> and CLI construction passes None for now (wired in next change). Server state holds observability, status_snapshot Arc> and pool_status. Bootstrap syncs the legacy Atomics/RwLocks for compatibility. CLI: replace Command::Status unit variant with Status(StatusArgs) { json: bool, limit: usize default 20 } (shape of PlanArgs), implement cmd_status that prints the status body byte-for-byte on --json and renders 10 human sections otherwise (service, queue, concurrency, scheduler, pool/runners, VM stub, store/storage/github/debug/telemetry, non-zero limits, stale tasks, conditions with actionable queue_label_mismatch → "add a runner with that label", recent runs via limit). wait_for_engine_socket now probes /readyz with 500ms/30s window and reports the last readyz reason on timeout. Status DTOs in preloop-observability/src/status.rs: Overall, Service, Runs, Jobs, Concurrency, Scheduler, Runners, Pool, VmFleet (stubbed), Store, Storage, Github, Debug, Telemetry, Limits, Tasks, Conditions with bounded enums and serde, plus PoolStatus shared handle. cargo check --locked -p preloop-observability -p preloop-runner-server -p preloop-orchestrator -p preloop-cli passes; fmt and sg-scan-strict pass. Entire-Checkpoint: 01M0GEMFJ1ZVSCEXFM2GM2EX52 --- Cargo.lock | 4 + crates/preloop-cli/src/main.rs | 593 ++++++++++++++++-- crates/preloop-observability/Cargo.toml | 4 + crates/preloop-observability/src/lib.rs | 2 + crates/preloop-observability/src/status.rs | 484 ++++++++++++++ crates/preloop-orchestrator/Cargo.toml | 1 + crates/preloop-orchestrator/src/lib.rs | 6 + crates/preloop-runner-server/src/bootstrap.rs | 268 +++++++- crates/preloop-runner-server/src/lib_tests.rs | 2 + crates/preloop-runner-server/src/main.rs | 2 + crates/preloop-runner-server/src/openapi.rs | 37 +- crates/preloop-runner-server/src/routes.rs | 10 + crates/preloop-runner-server/src/runs.rs | 135 +++- crates/preloop-runner-server/src/state.rs | 15 + 14 files changed, 1493 insertions(+), 70 deletions(-) create mode 100644 crates/preloop-observability/src/status.rs diff --git a/Cargo.lock b/Cargo.lock index e2f30bb8..3caa0791 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2021,7 +2021,10 @@ name = "preloop-observability" version = "0.1.0" dependencies = [ "anyhow", + "chrono", "parking_lot", + "serde", + "serde_json", "tokio", "tracing", "tracing-subscriber", @@ -2038,6 +2041,7 @@ dependencies = [ "futures", "preloop-gha-parser", "preloop-gha-protocol", + "preloop-observability", "preloop-runner-server", "preloop-vm", "reqwest", diff --git a/crates/preloop-cli/src/main.rs b/crates/preloop-cli/src/main.rs index d5b0816b..66463d71 100644 --- a/crates/preloop-cli/src/main.rs +++ b/crates/preloop-cli/src/main.rs @@ -478,12 +478,8 @@ enum Command { /// Show the expanded job DAG without executing. Plan(PlanArgs), - /// Show active and recent runs, or the live status of one run (prints a - /// single machine-readable status word for scripting). - Status { - #[arg(value_name = "RUN_ID")] - run_id: Option, - }, + /// Show operational status, queue health, and recent runs. + Status(StatusArgs), Logs(LogsArgs), @@ -708,6 +704,17 @@ struct PlanArgs { json: bool, } +#[derive(Debug, Parser)] +struct StatusArgs { + /// Print the raw status JSON (no prose) for jq/scripting. + #[arg(long)] + json: bool, + + /// Number of recent runs to show in the table. + #[arg(long, default_value = "20")] + limit: usize, +} + #[derive(Debug, Parser)] struct LogsArgs { /// Run ID. Defaults to the most recent run. @@ -789,7 +796,7 @@ async fn main() -> anyhow::Result<()> { match cli.command { Command::Run(args) => cmd_run(args).await, Command::Plan(_) => unreachable!("plan is handled before engine bootstrap"), - Command::Status { run_id } => cmd_status(run_id).await, + Command::Status(args) => cmd_status(args).await, Command::Logs(args) => cmd_logs(args).await, Command::Cancel(args) => cmd_cancel(args).await, Command::Shell(args) => cmd_shell(args).await, @@ -903,6 +910,7 @@ async fn cmd_build_golden(args: BuildGoldenArgs) -> anyhow::Result<()> { next_job_runs_on: None, pending_registrations: None, preparing_signal: None, + pool_status: None, }; let payload = artifact_payload(&output, &config.base_image); RunnerPool::new(std::sync::Arc::new(SmolVmProvider::default()), config)? @@ -1366,6 +1374,8 @@ async fn cmd_engine(args: ServeArgs) -> anyhow::Result<()> { next_job_runs_on: Some(next_job_runs_on.clone()), pool_preparing: Some(pool_preparing.clone()), pending_registrations: pool_available.then_some(pending_registrations), + pool_status: None, + observability: None, require_job_assignments: env_flag("PRELOOP_REQUIRE_JOB_ASSIGNMENTS", false), state_dir, store_url: args.store.clone(), @@ -1459,26 +1469,87 @@ async fn engine_shutdown_signal() { } async fn wait_for_engine_socket(socket: &std::path::Path) -> anyhow::Result<()> { + // readyz probe: http://localhost/readyz (500ms timeout, 30s window) #[cfg(unix)] let client = reqwest::Client::builder().unix_socket(socket).build()?; #[cfg(not(unix))] let client = reqwest::Client::new(); let start = std::time::Instant::now(); + let mut last_reason: Option = None; while start.elapsed() < Duration::from_secs(30) { - if client - .get("http://localhost/healthz") + match client + .get("http://localhost/readyz") .timeout(Duration::from_millis(500)) .send() .await - .is_ok() { - return Ok(()); + Ok(resp) => { + if resp.status().is_success() { + return Ok(()); + } + // Non-2xx readyz: capture reason for timeout reporting. + let status = resp.status(); + let body = resp.text().await.unwrap_or_default(); + let reason = extract_readyz_reason(&body) + .unwrap_or_else(|| format!("{status}: {}", truncate_reason(&body))); + last_reason = Some(reason); + } + Err(err) => { + last_reason = Some(err.to_string()); + } } tokio::time::sleep(Duration::from_millis(50)).await; } + if let Some(reason) = last_reason { + anyhow::bail!("local control plane did not become ready within 30 seconds: last readyz reason: {reason}") + } anyhow::bail!("local control plane did not become ready within 30 seconds") } +fn extract_readyz_reason(body: &str) -> Option { + if body.trim().is_empty() { + return None; + } + if let Ok(v) = serde_json::from_str::(body) { + for key in ["reason", "code", "message", "error", "status"] { + if let Some(s) = v.get(key).and_then(|x| x.as_str()) { + if !s.trim().is_empty() { + return Some(s.to_owned()); + } + } + } + // Nested { "ready": { "reason": ... } } or similar + if let Some(obj) = v.as_object() { + for (_, val) in obj { + if let Some(s) = val.as_str() { + if !s.trim().is_empty() && s.len() < 200 { + return Some(s.to_owned()); + } + } + if let Some(inner) = val.as_object() { + for k in ["reason", "code"] { + if let Some(s) = inner.get(k).and_then(|x| x.as_str()) { + return Some(s.to_owned()); + } + } + } + } + } + // Return truncated JSON if no specific field + return Some(truncate_reason(body)); + } + Some(truncate_reason(body)) +} + +fn truncate_reason(s: &str) -> String { + let t = s.trim(); + if t.len() > 300 { + format!("{}…", &t[..300]) + } else { + t.to_owned() + } +} + // Configuration assembly, not a public API: the parameter list mirrors the // inputs the pool genuinely needs. #[allow(clippy::too_many_arguments)] @@ -1653,6 +1724,7 @@ fn local_runner_pool_config( next_job_runs_on: (!custom_base).then_some(next_job_runs_on), pending_registrations: Some(pending_registrations), preparing_signal: Some(preparing_signal), + pool_status: None, }) } @@ -2801,63 +2873,418 @@ fn plan_json(plan: &preloop_gha_protocol::JobPlan) -> serde_json::Value { }) } -async fn cmd_status(run_id: Option) -> anyhow::Result<()> { +async fn cmd_status(args: StatusArgs) -> anyhow::Result<()> { let client = build_client(); let url = server_url(); - if let Some(run_id) = run_id { - // Single-run mode: one machine-readable status word - // (success/failure/cancelled/skipped/in_progress/queued/pending) for - // scripts like the pre-push hook. Connection failures carry the - // engine-unreachable marker so the hook can fail open. - let mut request = client.get(format!("{url}/api/v1/runs/{run_id}")); - if let Some(token) = api_token() { - request = request.bearer_auth(token); - } - let response = request - .send() - .await - .with_context(engine_unreachable_context)?; - if !response.status().is_success() { - let status = response.status(); - let body = response.text().await.unwrap_or_default(); - anyhow::bail!("server returned {status}: {body}"); + // Native bearer required for /api/v1/status (same as other native calls) + let mut status_req = client.get(format!("{url}/api/v1/status")); + if let Some(token) = api_token() { + status_req = status_req.bearer_auth(token); + } + let status_resp = status_req + .send() + .await + .with_context(engine_unreachable_context)?; + if !status_resp.status().is_success() { + let status = status_resp.status(); + let body = status_resp.text().await.unwrap_or_default(); + anyhow::bail!("server returned {status}: {body}"); + } + let status_text = status_resp.text().await?; + if args.json { + // Byte-for-byte, no prose: so jq works + print!("{}", status_text); + if !status_text.ends_with('\n') { + println!(); } - let run: serde_json::Value = response.json().await?; - println!( - "{}", - run.get("status") - .and_then(serde_json::Value::as_str) - .unwrap_or("unknown") - ); return Ok(()); } - let mut request = client.get(format!("{url}/api/v1/runs?limit=20")); + let status: serde_json::Value = + serde_json::from_str(&status_text).context("parse status json")?; + + // Fetch recent runs for table (preserve existing behavior) + let mut runs_req = client.get(format!("{url}/api/v1/runs?limit={}", args.limit)); if let Some(token) = api_token() { - request = request.bearer_auth(token); + runs_req = runs_req.bearer_auth(token); + } + let runs: Vec = match runs_req.send().await { + Ok(r) if r.status().is_success() => r.json().await.unwrap_or_default(), + Ok(r) => { + let st = r.status(); + let body = r.text().await.unwrap_or_default(); + eprintln!("[warn] runs table unavailable: {st}: {body}"); + Vec::new() + } + Err(e) => { + eprintln!("[warn] runs table unavailable: {e}"); + Vec::new() + } + }; + + // --- Human rendering: 10 sections in order --- + render_status_human(&status, &runs, args.limit); + Ok(()) +} + +fn render_status_human(status: &serde_json::Value, runs: &[serde_json::Value], limit: usize) { + // Helpers to extract safely + let get_str = |v: &serde_json::Value, k: &str| -> Option { + v.get(k).and_then(|x| x.as_str()).map(|s| s.to_owned()) + }; + let get_f64 = + |v: &serde_json::Value, k: &str| -> Option { v.get(k).and_then(|x| x.as_f64()) }; + let get_u64 = + |v: &serde_json::Value, k: &str| -> Option { v.get(k).and_then(|x| x.as_u64()) }; + let get_bool = + |v: &serde_json::Value, k: &str| -> Option { v.get(k).and_then(|x| x.as_bool()) }; + + // 1. service + snapshot age + println!("== service =="); + let service = status.get("service").unwrap_or(&serde_json::Value::Null); + let version = + get_str(service, "version").unwrap_or_else(|| env!("CARGO_PKG_VERSION").to_owned()); + let instance = get_str(service, "instance_id") + .or_else(|| get_str(status, "instance_id")) + .unwrap_or_else(|| "-".to_owned()); + let uptime = get_f64(service, "uptime_seconds") + .or_else(|| get_f64(status, "uptime_seconds")) + .unwrap_or(0.0); + let shutdown = get_bool(service, "shutdown_requested") + .or_else(|| get_bool(status, "shutdown_requested")) + .unwrap_or(false); + let snapshot_age = get_f64(status, "snapshot_age_seconds").unwrap_or(0.0); + let observed_at = get_str(status, "observed_at").unwrap_or_else(|| "-".to_owned()); + let overall = get_str(status, "overall").unwrap_or_else(|| "unknown".to_owned()); + println!( + " version: {version} instance: {instance} uptime: {uptime:.0}s overall: {overall}" + ); + println!( + " observed_at: {observed_at} snapshot_age: {snapshot_age:.1}s shutdown: {shutdown}" + ); + if status.get("schema_version").is_some() { + println!(" schema_version: {}", status["schema_version"]); + } + + // 2. queue (ready/blocked + oldest) + println!("\n== queue =="); + let jobs = status.get("jobs").unwrap_or(&serde_json::Value::Null); + let ready = get_u64(jobs, "ready").unwrap_or(0); + let dep_blocked = get_u64(jobs, "dependency_blocked").unwrap_or(0); + let conc_blocked = get_u64(jobs, "concurrency_blocked").unwrap_or(0); + let pending_exp = get_u64(jobs, "pending_expansion").unwrap_or(0); + let expanding = get_u64(jobs, "expanding").unwrap_or(0); + let claimable = get_u64(jobs, "claimable").unwrap_or(0); + let unclaimable = get_u64(jobs, "unclaimable").unwrap_or(0); + let oldest = get_f64(jobs, "oldest_ready_seconds"); + println!(" ready: {ready} claimable: {claimable} unclaimable: {unclaimable} dependency_blocked: {dep_blocked} concurrency_blocked: {conc_blocked} pending_expansion: {pending_exp} expanding: {expanding}"); + match oldest { + Some(v) => println!(" oldest_ready: {v:.1}s"), + None => println!(" oldest_ready: -"), + } + // Also show runs queued/in_progress if present + if let Some(runs_obj) = status.get("runs") { + let q = get_u64(runs_obj, "queued").unwrap_or(0); + let ip = get_u64(runs_obj, "in_progress").unwrap_or(0); + let completed = get_u64(runs_obj, "completed").unwrap_or(0); + println!(" runs queued: {q} in_progress: {ip} completed: {completed}"); + } + + // 3. concurrency + scheduler + println!("\n== concurrency & scheduler =="); + let conc = status + .get("concurrency") + .unwrap_or(&serde_json::Value::Null); + let groups_active = get_u64(conc, "groups_active").unwrap_or(0); + let groups_contended = get_u64(conc, "groups_contended").unwrap_or(0); + let pending_holders = get_u64(conc, "pending_holders").unwrap_or(0); + let deepest = get_u64(conc, "deepest_group_pending").unwrap_or(0); + let qmax = get_u64(conc, "queue_max_pending").unwrap_or(0); + let overflow = get_u64(conc, "overflow_cancellations").unwrap_or(0); + println!(" groups active: {groups_active} contended: {groups_contended} pending_holders: {pending_holders} deepest_pending: {deepest} queue_max: {qmax} overflow_cancellations: {overflow}"); + let sched = status.get("scheduler").unwrap_or(&serde_json::Value::Null); + let enabled = get_bool(sched, "enabled").unwrap_or(false); + let schedules = get_u64(sched, "schedules").unwrap_or(0); + let last_scan = get_str(sched, "last_scan_at").unwrap_or_else(|| "-".to_owned()); + let next_fire = get_str(sched, "next_fire_at").unwrap_or_else(|| "-".to_owned()); + let fired = get_u64(sched, "fired").unwrap_or(0); + let skipped = get_u64(sched, "skipped_overlapping").unwrap_or(0); + let late = get_u64(sched, "late_fires").unwrap_or(0); + let max_delay = get_f64(sched, "max_fire_delay_seconds"); + println!(" scheduler enabled: {enabled} schedules: {schedules} fired: {fired} skipped_overlapping: {skipped} late_fires: {late}"); + println!( + " last_scan: {last_scan} next_fire: {next_fire} max_delay: {}", + max_delay + .map(|v| format!("{v:.1}s")) + .unwrap_or_else(|| "-".to_owned()) + ); + + // 4. pool + runners + println!("\n== pool & runners =="); + let pool = status.get("pool").unwrap_or(&serde_json::Value::Null); + let mode = get_str(pool, "mode").unwrap_or_else(|| "-".to_owned()); + let desired = get_u64(pool, "desired").unwrap_or(0); + let preparing = pool + .get("preparing") + .and_then(|x| x.as_bool()) + .unwrap_or(false); + let building = get_u64(pool, "building").unwrap_or(0); + let provisioning = get_u64(pool, "provisioning").unwrap_or(0); + let pool_idle = get_u64(pool, "idle").unwrap_or(0); + let pool_busy = get_u64(pool, "busy").unwrap_or(0); + let paused = get_u64(pool, "paused").unwrap_or(0); + let failures = get_u64(pool, "consecutive_provision_failures").unwrap_or(0); + println!(" pool mode: {mode} desired: {desired} idle: {pool_idle} busy: {pool_busy} building: {building} provisioning: {provisioning} paused: {paused} preparing: {preparing} provision_failures: {failures}"); + let runners = status.get("runners").unwrap_or(&serde_json::Value::Null); + let reg = get_u64(runners, "registered").unwrap_or(0); + let sessions = get_u64(runners, "sessions").unwrap_or(0); + let idle = get_u64(runners, "idle").unwrap_or(0); + let busy = get_u64(runners, "busy").unwrap_or(0); + let stale = get_u64(runners, "stale").unwrap_or(0); + let max_poll = get_f64(runners, "max_poll_age_seconds"); + let max_lease = get_f64(runners, "max_lease_age_seconds"); + println!(" runners registered: {reg} sessions: {sessions} idle: {idle} busy: {busy} stale: {stale}"); + println!( + " max_poll_age: {} max_lease_age: {}", + max_poll + .map(|v| format!("{v:.1}s")) + .unwrap_or_else(|| "-".to_owned()), + max_lease + .map(|v| format!("{v:.1}s")) + .unwrap_or_else(|| "-".to_owned()) + ); + + // 5. VM fleet stub + println!("\n== vm fleet =="); + let vms = status + .get("vms") + .unwrap_or(status.get("vm").unwrap_or(&serde_json::Value::Null)); + if vms.is_null() || vms.as_object().map(|m| m.is_empty()).unwrap_or(false) { + println!(" source: unavailable (host sampler not yet reporting)"); + println!(" capabilities: cpu=false memory=false sparse_disk=false"); + println!(" host_usage: -"); + } else { + let source = get_str(vms, "source").unwrap_or_else(|| "unknown".to_owned()); + let sample_age = get_f64(vms, "sample_age_seconds") + .map(|v| format!("{v:.1}s")) + .unwrap_or_else(|| "-".to_owned()); + println!(" source: {source} sample_age: {sample_age}"); + if let Some(caps) = vms.get("capabilities") { + let cap_str = caps + .as_object() + .map(|m| { + m.iter() + .map(|(k, v)| format!("{k}={}", v.as_bool().unwrap_or(false))) + .collect::>() + .join(" ") + }) + .unwrap_or_else(|| "-".to_owned()); + println!(" capabilities: {cap_str}"); + } + if let Some(cnt) = vms.get("count") { + let runner = get_u64(cnt, "runner").unwrap_or(0); + let golden = get_u64(cnt, "golden").unwrap_or(0); + let unavailable = get_u64(cnt, "unavailable").unwrap_or(0); + println!(" count runner: {runner} golden: {golden} unavailable: {unavailable}"); + } + if let Some(conf) = vms.get("configured") { + let vcpus = get_u64(conf, "vcpus") + .map(|v| v.to_string()) + .unwrap_or_else(|| "-".to_owned()); + let mem = get_u64(conf, "memory_bytes") + .map(|v| format!("{v}")) + .unwrap_or_else(|| "-".to_owned()); + let storage = get_u64(conf, "storage_bytes") + .map(|v| format!("{v}")) + .unwrap_or_else(|| "-".to_owned()); + println!(" configured vcpus: {vcpus} memory: {mem} storage: {storage}"); + } + if let Some(usage) = vms.get("host_usage") { + let cores = get_f64(usage, "cpu_cores") + .map(|v| format!("{v:.1}")) + .unwrap_or_else(|| "-".to_owned()); + let mem = get_u64(usage, "memory_bytes") + .map(|v| format!("{v}")) + .unwrap_or_else(|| "-".to_owned()); + println!(" host_usage cpu_cores: {cores} memory: {mem}"); + } + if let Some(top) = vms.get("top_consumers").and_then(|x| x.as_array()) { + if !top.is_empty() { + println!(" top_consumers ({}):", top.len().min(5)); + for c in top.iter().take(5) { + let name = c + .get("machine_name") + .and_then(|x| x.as_str()) + .unwrap_or("?"); + let role = c.get("role").and_then(|x| x.as_str()).unwrap_or("?"); + let activity = c.get("activity").and_then(|x| x.as_str()).unwrap_or("?"); + println!(" - {name} ({role}/{activity})"); + } + } else { + println!(" top_consumers: -"); + } + } } - let response = request.send().await?; - if !response.status().is_success() { - let status = response.status(); - let body = response.text().await.unwrap_or_default(); - anyhow::bail!("server returned {status}: {body}"); + + // 6. store/storage/GitHub/debug/telemetry + println!("\n== store / storage / github / debug / telemetry =="); + let store = status.get("store").unwrap_or(&serde_json::Value::Null); + let backend = get_str(store, "backend").unwrap_or_else(|| "-".to_owned()); + let cfail = get_u64(store, "consecutive_failures").unwrap_or(0); + println!(" store backend: {backend} consecutive_failures: {cfail}"); + let storage = status.get("storage").unwrap_or(&serde_json::Value::Null); + if !storage.is_null() { + let free = get_u64(storage, "state_fs_free_bytes") + .map(|v| format!("{v}")) + .unwrap_or_else(|| "-".to_owned()); + let ratio = get_f64(storage, "state_fs_free_ratio") + .map(|v| format!("{v:.2}")) + .unwrap_or_else(|| "-".to_owned()); + println!(" storage free: {free} ratio: {ratio}"); + if let Some(comps) = storage.get("components").and_then(|x| x.as_array()) { + for c in comps { + let store_name = c.get("store").and_then(|x| x.as_str()).unwrap_or("?"); + let bytes = c.get("bytes").and_then(|x| x.as_u64()).unwrap_or(0); + println!(" {store_name}: {bytes} bytes"); + } + } + } else { + println!(" storage: -"); + } + let github = status.get("github").unwrap_or(&serde_json::Value::Null); + if !github.is_null() { + let configured = get_bool(github, "configured").unwrap_or(false); + let pending = get_u64(github, "pending_check_updates").unwrap_or(0); + println!(" github configured: {configured} pending_check_updates: {pending}"); + if let Some(rl) = github.get("rate_limit") { + let remaining = get_u64(rl, "remaining").unwrap_or(0); + let limit_rl = get_u64(rl, "limit").unwrap_or(0); + println!(" github rate_limit: {remaining}/{limit_rl} remaining"); + } + if let Some(exp) = github + .get("installation_token_expires_in_seconds") + .and_then(|x| x.as_u64()) + { + println!(" github token expires_in: {exp}s"); + } + } else { + println!(" github: -"); + } + let debug = status.get("debug").unwrap_or(&serde_json::Value::Null); + if !debug.is_null() { + let active = get_u64(debug, "active_sessions").unwrap_or(0); + let oldest = debug + .get("oldest_session_seconds") + .and_then(|x| x.as_f64()) + .map(|v| format!("{v:.0}s")) + .unwrap_or_else(|| "-".to_owned()); + println!(" debug active_sessions: {active} oldest: {oldest}"); + } + let tele = status.get("telemetry").unwrap_or(&serde_json::Value::Null); + if !tele.is_null() { + let enabled = get_bool(tele, "otlp_enabled").unwrap_or(false); + let dropped = get_u64(tele, "dropped_records").unwrap_or(0); + println!(" telemetry otlp_enabled: {enabled} dropped_records: {dropped}"); + } + + // 7. non-zero limits + println!("\n== limits (non-zero) =="); + let limits = status.get("limits").and_then(|x| x.as_array()); + let mut any_limit = false; + if let Some(arr) = limits { + for l in arr { + let dropped = l.get("dropped").and_then(|x| x.as_u64()).unwrap_or(0); + let rejected = l.get("rejected").and_then(|x| x.as_u64()).unwrap_or(0); + if dropped > 0 || rejected > 0 { + any_limit = true; + let name = l.get("limit").and_then(|x| x.as_str()).unwrap_or("?"); + let value = l + .get("value") + .and_then(|x| x.as_u64()) + .map(|v| v.to_string()) + .unwrap_or_else(|| "-".to_owned()); + println!(" {name}: value={value} dropped={dropped} rejected={rejected}"); + } + } } - let runs: Vec = response.json().await?; + if !any_limit { + println!(" (no limits with drops/rejects)"); + } + + // 8. stale tasks + println!("\n== tasks (stale/exited) =="); + let tasks = status.get("tasks").and_then(|x| x.as_array()); + let mut any_task = false; + if let Some(arr) = tasks { + for t in arr { + let state = t.get("state").and_then(|x| x.as_str()).unwrap_or("running"); + if state == "stale" || state == "exited" { + any_task = true; + let name = t.get("name").and_then(|x| x.as_str()).unwrap_or("?"); + let critical = t.get("critical").and_then(|x| x.as_bool()).unwrap_or(false); + let age = t + .get("heartbeat_age_seconds") + .and_then(|x| x.as_f64()) + .map(|v| format!("{v:.1}s")) + .unwrap_or_else(|| "-".to_owned()); + println!(" {name}: state={state} critical={critical} age={age}"); + } + } + } + if !any_task { + println!(" (all tasks healthy)"); + } + + // 9. conditions with one-line actions (≤5 exemplars) + println!("\n== conditions =="); + let conditions = status.get("conditions").and_then(|x| x.as_array()); + if let Some(arr) = conditions { + if arr.is_empty() { + println!(" (no conditions)"); + } else { + for c in arr { + let code = c.get("code").and_then(|x| x.as_str()).unwrap_or("?"); + let severity = c.get("severity").and_then(|x| x.as_str()).unwrap_or("info"); + let msg = c.get("message").and_then(|x| x.as_str()).unwrap_or(""); + let action = condition_action(code); + println!(" [{severity}] {code}: {msg} -> {action}"); + if let Some(exs) = c.get("exemplars").and_then(|x| x.as_array()) { + for ex in exs.iter().take(5) { + let ex_str = match ex { + serde_json::Value::String(s) => s.clone(), + other => other.to_string(), + }; + println!(" - {ex_str}"); + } + if exs.len() > 5 { + println!(" ... and {} more", exs.len() - 5); + } + } else if let Some(exs) = c.get("exemplar").and_then(|x| x.as_str()) { + println!(" - {exs}"); + } + } + } + } else { + println!(" (no conditions)"); + } + + // 10. recent runs table + println!("\n== recent runs (limit={}) ==", limit); if runs.is_empty() { println!("No runs found."); - return Ok(()); + return; } println!( "{:<38} {:<6} {:<12} {:<12} {:<10} WORKFLOW", "RUN ID", "#", "STATUS", "EVENT", "PUSH" ); println!("{}", "-".repeat(104)); - for run in &runs { + for run in runs { let run_id = run["run_id"].as_str().unwrap_or("?"); let run_number = run .get("run_number") .and_then(serde_json::Value::as_u64) .unwrap_or(0); - let status = run["status"].as_str().unwrap_or("?"); + let st = run["status"].as_str().unwrap_or("?"); let event = run .get("event") .and_then(serde_json::Value::as_str) @@ -2881,21 +3308,56 @@ async fn cmd_status(run_id: Option) -> anyhow::Result<()> { }) .unwrap_or("?"); let push = match run.get("push_state").and_then(|state| state.get("status")) { - Some(status) => { - let status = status.as_str().unwrap_or("?"); + Some(s) => { + let s = s.as_str().unwrap_or("?"); match run["push_state"]["pr_number"].as_u64() { - Some(number) => format!("{status} #{number}"), - None => status.to_owned(), + Some(n) => format!("{s} #{n}"), + None => s.to_owned(), } } None => "-".to_owned(), }; println!( "{:<38} {:<6} {:<12} {:<12} {:<10} {}", - run_id, run_number, status, event, push, workflow + run_id, run_number, st, event, push, workflow ); } - Ok(()) +} + +fn condition_action(code: &str) -> &'static str { + match code { + "queue_no_registered_runner" => "register a runner or enable the pool", + "queue_label_mismatch" => "add a runner with that label", + "concurrency_queue_overflow" => "raise concurrency queue max or reduce parallelism", + "concurrency_group_starved" => "check concurrency group that starves others", + "scheduler_scan_stale" => "check scheduler scan heartbeat (restart if stuck)", + "scheduler_fire_late" => "check scheduler clock / load", + "pool_preparing" => "wait for image preparation to finish", + "pool_provisioning_deficit" => "check pool capacity / provision failures", + "pool_repeated_provision_failure" => "inspect provision logs and VM host capacity", + "runner_poll_stale" => "check runner connectivity and heartbeat", + "runner_lease_stale" => "check runner lease renewal", + "vm_sampler_stale" | "vm_sample_unavailable" | "vm_unreachable" => { + "check VM host sampler and SmolVM health" + } + "vm_host_memory_pressure" => "free host memory or reduce pool size", + "vm_host_cpu_throttled" => "reduce host CPU load or raise CPU quota", + "vm_host_oom_kill" => "check host OOM kills and runner memory", + "vm_sparse_disk_pressure" => "free disk space on VM data volume", + "store_write_failure" | "store_connection_down" => "check store connectivity and disk", + "storage_capacity_pressure" => "free disk space or run GC/prune", + "limit_drop_active" => "raise that limit or reduce load", + "limit_reject_active" => "raise that limit or back off", + "github_check_update_failure" => "check GitHub App permissions and network", + "github_terminal_check_pending" => "retry check update or check GitHub status", + "github_rate_limit_low" => "back off GitHub API or wait for rate-limit reset", + "github_installation_token_expiring" => "refresh GitHub installation token", + "debug_session_stale" => "close stale debug session", + "debug_audit_evicted" => "increase audit retention or flush audits", + "telemetry_export_failure" => "check OTLP endpoint and credentials", + "state_sampler_stale" | "task_stale" | "task_exited" => "check background task health", + _ => "see runbook for this condition", + } } async fn cmd_logs(args: LogsArgs) -> anyhow::Result<()> { @@ -3760,14 +4222,23 @@ mod tests { #[test] fn status_parses() { let cli = parse(&["status"]).unwrap(); - assert!(matches!(cli.command, Command::Status { run_id: None })); - let cli = parse(&["status", "550e8400-e29b-41d4-a716-446655440000"]).unwrap(); - assert!(matches!( - cli.command, - Command::Status { - run_id: Some(ref id) - } if id == "550e8400-e29b-41d4-a716-446655440000" - )); + let Command::Status(args) = cli.command else { + panic!("expected Status"); + }; + assert!(!args.json); + assert_eq!(args.limit, 20); + let cli = parse(&["status", "--json", "--limit", "5"]).unwrap(); + let Command::Status(args) = cli.command else { + panic!("expected Status"); + }; + assert!(args.json); + assert_eq!(args.limit, 5); + // Default limit is 20 and --json defaults to false + let cli = parse(&["status", "--limit", "42"]).unwrap(); + let Command::Status(args) = cli.command else { + panic!("expected Status"); + }; + assert_eq!(args.limit, 42); } #[test] diff --git a/crates/preloop-observability/Cargo.toml b/crates/preloop-observability/Cargo.toml index dea5e730..6a8e37b0 100644 --- a/crates/preloop-observability/Cargo.toml +++ b/crates/preloop-observability/Cargo.toml @@ -8,13 +8,17 @@ repository = "https://github.com/preloopdev/preloop" [dependencies] anyhow = { workspace = true } +chrono = { workspace = true } parking_lot = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } tracing = { workspace = true } tracing-subscriber = { workspace = true } uuid = { workspace = true } tokio = { workspace = true } [dev-dependencies] +serde_json = { workspace = true } tokio = { workspace = true } tracing = { workspace = true } diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs index 1d926147..cf5662ed 100644 --- a/crates/preloop-observability/src/lib.rs +++ b/crates/preloop-observability/src/lib.rs @@ -11,6 +11,8 @@ //! - Always retain `stderr`/`journald` even when OTLP is configured. //! - `Debug` on config never reveals headers or credential-bearing endpoint parts. +pub mod status; + use std::collections::HashMap; use std::fmt; use std::sync::Arc; diff --git a/crates/preloop-observability/src/status.rs b/crates/preloop-observability/src/status.rs new file mode 100644 index 00000000..b4853362 --- /dev/null +++ b/crates/preloop-observability/src/status.rs @@ -0,0 +1,484 @@ +//! Step 3 DTOs — OperationalSnapshot and supporting types. +//! +//! Neutral, serializable DTOs shared by server and CLI. No dependency on +//! server/orchestrator internals. Metric labels use only bounded enums. + +use std::sync::Arc; + +use chrono::{DateTime, Utc}; +use parking_lot::RwLock; +use serde::{Deserialize, Serialize}; + +// --------------------------------------------------------------------------- +// Overall +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Overall { + Ok, + Degraded, + Blocked, + ShuttingDown, +} + +impl Default for Overall { + fn default() -> Self { + Self::Ok + } +} + +// --------------------------------------------------------------------------- +// Service +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ServiceSnapshot { + pub version: String, + pub instance_id: String, + pub uptime_seconds: u64, + pub shutdown_requested: bool, +} + +// --------------------------------------------------------------------------- +// Runs / Jobs / Concurrency / Scheduler / Runners +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct RunsSnapshot { + pub queued: u32, + pub in_progress: u32, + pub completed: u32, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct JobsSnapshot { + pub ready: u32, + pub dependency_blocked: u32, + pub concurrency_blocked: u32, + pub pending_expansion: u32, + pub expanding: u32, + pub claimable: u32, + pub unclaimable: u32, + pub oldest_ready_seconds: Option, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct ConcurrencySnapshot { + pub groups_active: u32, + pub groups_contended: u32, + pub pending_holders: u32, + pub deepest_group_pending: u32, + pub queue_max_pending: usize, + pub overflow_cancellations: u64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct SchedulerSnapshot { + pub enabled: bool, + pub schedules: u32, + pub last_scan_at: Option>, + pub next_fire_at: Option>, + pub fired: u64, + pub skipped_overlapping: u64, + pub late_fires: u64, + pub max_fire_delay_seconds: Option, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct RunnersSnapshot { + pub registered: u32, + pub sessions: u32, + pub idle: u32, + pub busy: u32, + pub stale: u32, + pub max_poll_age_seconds: Option, + pub max_lease_age_seconds: Option, +} + +// --------------------------------------------------------------------------- +// Pool +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum PoolMode { + Warm, + OnDemand, + External, + Disabled, +} + +impl Default for PoolMode { + fn default() -> Self { + Self::Warm + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct PoolSnapshot { + pub mode: PoolMode, + pub desired: u32, + pub preparing: bool, + pub building: u32, + pub provisioning: u32, + pub idle: u32, + pub busy: u32, + pub paused: u32, + pub consecutive_provision_failures: u32, + pub last_transition_at: Option>, + /// Consolidated queue depth (server -> pool signal, now via PoolStatus). + #[serde(default)] + pub queue_depth: u32, + /// Next job labels for golden selection (server -> pool). + #[serde(default)] + pub next_job_runs_on: Vec, + /// Pending provision token count (pool -> server). + #[serde(default)] + pub pending_registrations: u32, +} + +impl Default for PoolSnapshot { + fn default() -> Self { + Self { + mode: PoolMode::Warm, + desired: 0, + preparing: false, + building: 0, + provisioning: 0, + idle: 0, + busy: 0, + paused: 0, + consecutive_provision_failures: 0, + last_transition_at: None, + queue_depth: 0, + next_job_runs_on: Vec::new(), + pending_registrations: 0, + } + } +} + +/// Shared handle that the pool updates and the sampler reads. +/// +/// Consolidates the four ad-hoc `Option>` handles. Single writer (pool) +/// + multiple readers (sampler, status) — cheap `RwLock`. +#[derive(Debug, Clone, Default)] +pub struct PoolStatus { + inner: Arc>, + /// One-time provision tokens (separate from snapshot to avoid cloning large map on every snapshot). + pending_tokens: Arc>>, +} + +impl PoolStatus { + pub fn new(snapshot: PoolSnapshot) -> Self { + Self { + inner: Arc::new(RwLock::new(snapshot)), + pending_tokens: Arc::new(RwLock::new(std::collections::BTreeMap::new())), + } + } + + pub fn snapshot(&self) -> PoolSnapshot { + let mut snap = self.inner.read().clone(); + snap.pending_registrations = self.pending_tokens.read().len() as u32; + snap + } + + pub fn set_desired(&self, desired: u32) { + self.inner.write().desired = desired; + } + + pub fn set_preparing(&self, preparing: bool) { + self.inner.write().preparing = preparing; + } + + pub fn set_counts(&self, idle: u32, busy: u32, building: u32, provisioning: u32, paused: u32) { + let mut g = self.inner.write(); + g.idle = idle; + g.busy = busy; + g.building = building; + g.provisioning = provisioning; + g.paused = paused; + } + + pub fn record_provision_failure(&self) { + self.inner.write().consecutive_provision_failures += 1; + } + + pub fn clear_provision_failures(&self) { + self.inner.write().consecutive_provision_failures = 0; + } + + pub fn set_queue_depth(&self, depth: u32) { + self.inner.write().queue_depth = depth; + } + + pub fn set_next_job_runs_on(&self, labels: Vec) { + self.inner.write().next_job_runs_on = labels; + } + + pub fn insert_pending(&self, token: String, at: std::time::SystemTime) { + self.pending_tokens.write().insert(token, at); + self.inner.write().pending_registrations = self.pending_tokens.read().len() as u32; + } + + pub fn remove_pending(&self, token: &str) -> bool { + let removed = self.pending_tokens.write().remove(token).is_some(); + if removed { + self.inner.write().pending_registrations = self.pending_tokens.read().len() as u32; + } + removed + } + + pub fn pending_tokens_snapshot( + &self, + ) -> std::collections::BTreeMap { + self.pending_tokens.read().clone() + } +} + +// --------------------------------------------------------------------------- +// VMs (stubbed for Step 3, full host sampler is Step 5) +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum VmSource { + CgroupV2, + Process, + Mixed, + Unavailable, +} + +impl Default for VmSource { + fn default() -> Self { + Self::Unavailable + } +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct VmFleetSnapshot { + pub source: VmSource, + pub sample_age_seconds: Option, + pub capabilities: HashMap, + pub count: VmCount, + pub configured: VmConfigured, + pub host_usage: VmHostUsage, + pub top_consumers: Vec, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct VmCount { + pub runner: u32, + pub golden: u32, + pub unavailable: u32, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct VmConfigured { + pub vcpus: u32, + pub memory_bytes: u64, + pub storage_bytes: u64, + pub overlay_bytes: u64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct VmHostUsage { + pub cpu_cores: f64, + pub memory_bytes: u64, + pub sparse_disk_allocated_bytes: u64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct VmTopConsumer { + pub machine_name: String, + pub role: String, + pub activity: String, + pub cpu_cores: f64, + pub memory_bytes: u64, + pub sparse_disk_allocated_bytes: u64, +} + +use std::collections::HashMap; + +// --------------------------------------------------------------------------- +// Store / Storage / Github / Debug / Telemetry +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum StoreBackend { + Sqlite, + Postgres, +} + +impl Default for StoreBackend { + fn default() -> Self { + Self::Sqlite + } +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct StoreSnapshot { + pub backend: StoreBackend, + pub consecutive_failures: u32, + pub last_success_at: Option>, + pub last_failure_at: Option>, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct StorageComponent { + pub store: String, + pub bytes: u64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct StorageSnapshot { + pub state_dir: String, + pub state_fs_free_bytes: Option, + pub state_fs_free_ratio: Option, + pub components: Vec, + pub last_gc_at: Option>, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct GithubSnapshot { + pub configured: bool, + pub last_webhook_at: Option>, + pub pending_check_updates: u32, + pub last_check_success_at: Option>, + pub last_check_failure_at: Option>, + pub rate_limit: Option, + pub installation_token_expires_in_seconds: Option, + pub token_cache: Option, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct GithubRateLimit { + pub resource: String, + pub limit: u32, + pub remaining: u32, + pub reset_at: Option>, + pub observed_at: Option>, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct TokenCacheSnapshot { + pub hits: u64, + pub misses: u64, + pub ttl_seconds: u64, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct DebugSnapshot { + pub active_sessions: u32, + pub oldest_session_seconds: Option, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct TelemetrySnapshot { + pub otlp_enabled: bool, + pub last_export_success_at: Option>, + pub last_export_failure_at: Option>, + pub dropped_records: u64, +} + +// --------------------------------------------------------------------------- +// Limits / Tasks (from heartbeat/limit registries) +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct LimitEntry { + pub limit: String, + pub value: usize, + pub dropped: u64, + pub rejected: u64, + pub last_at: Option>, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct TaskEntry { + pub name: String, + pub critical: bool, + pub heartbeat_age_seconds: f64, + pub state: String, +} + +// --------------------------------------------------------------------------- +// Conditions +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Condition { + pub code: String, + pub severity: String, + pub message: String, + pub exemplars: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ConditionExemplar { + pub run_id: Option, + pub job_id: Option, + pub runner_id: Option, + pub machine_name: Option, +} + +// --------------------------------------------------------------------------- +// OperationalSnapshot — the versioned status body +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct OperationalSnapshot { + pub schema_version: u32, + pub observed_at: DateTime, + pub snapshot_age_seconds: f64, + pub overall: Overall, + pub service: ServiceSnapshot, + pub runs: RunsSnapshot, + pub jobs: JobsSnapshot, + pub concurrency: ConcurrencySnapshot, + pub scheduler: SchedulerSnapshot, + pub runners: RunnersSnapshot, + pub pool: PoolSnapshot, + pub vms: VmFleetSnapshot, + pub store: StoreSnapshot, + pub storage: StorageSnapshot, + pub limits: Vec, + pub tasks: Vec, + pub github: GithubSnapshot, + pub debug: DebugSnapshot, + pub telemetry: TelemetrySnapshot, + pub conditions: Vec, +} + +impl Default for OperationalSnapshot { + fn default() -> Self { + Self { + schema_version: 1, + observed_at: Utc::now(), + snapshot_age_seconds: 0.0, + overall: Overall::Ok, + service: ServiceSnapshot { + version: env!("CARGO_PKG_VERSION").to_string(), + instance_id: String::new(), + uptime_seconds: 0, + shutdown_requested: false, + }, + runs: RunsSnapshot::default(), + jobs: JobsSnapshot::default(), + concurrency: ConcurrencySnapshot::default(), + scheduler: SchedulerSnapshot::default(), + runners: RunnersSnapshot::default(), + pool: PoolSnapshot::default(), + vms: VmFleetSnapshot::default(), + store: StoreSnapshot::default(), + storage: StorageSnapshot::default(), + limits: Vec::new(), + tasks: Vec::new(), + github: GithubSnapshot::default(), + debug: DebugSnapshot::default(), + telemetry: TelemetrySnapshot::default(), + conditions: Vec::new(), + } + } +} diff --git a/crates/preloop-orchestrator/Cargo.toml b/crates/preloop-orchestrator/Cargo.toml index 2a5c4e29..0f2763fa 100644 --- a/crates/preloop-orchestrator/Cargo.toml +++ b/crates/preloop-orchestrator/Cargo.toml @@ -10,6 +10,7 @@ description = "Job scheduling and VM lifecycle orchestration for Preloop CI" [dependencies] preloop-vm = { path = "../preloop-vm" } preloop-gha-protocol = { path = "../preloop-gha-protocol" } +preloop-observability = { path = "../preloop-observability" } preloop-runner-server = { path = "../preloop-runner-server" } preloop-gha-parser = { path = "../preloop-gha-parser" } anyhow = { workspace = true } diff --git a/crates/preloop-orchestrator/src/lib.rs b/crates/preloop-orchestrator/src/lib.rs index 3c4cfd2f..960a711d 100644 --- a/crates/preloop-orchestrator/src/lib.rs +++ b/crates/preloop-orchestrator/src/lib.rs @@ -1675,6 +1675,11 @@ pub struct RunnerPoolConfig { /// queued-job starvation clock during the warm; it is cleared before /// the pool serves its first job. pub preparing_signal: Option>, + /// Consolidated pool handle (replaces the four ad-hoc Option> fields above). + /// When `Some`, the pool updates it and the server sampler reads it. + /// Retain the legacy fields for now for backwards compatibility; new code + /// should read/write `pool_status`. + pub pool_status: Option>, } /// Cache of environment-specific golden VMs. @@ -4919,6 +4924,7 @@ chmod +x "$destination/bin/node" next_job_runs_on: None, pending_registrations: None, preparing_signal: None, + pool_status: None, } } diff --git a/crates/preloop-runner-server/src/bootstrap.rs b/crates/preloop-runner-server/src/bootstrap.rs index 7deddf79..77f9ffd1 100644 --- a/crates/preloop-runner-server/src/bootstrap.rs +++ b/crates/preloop-runner-server/src/bootstrap.rs @@ -48,6 +48,12 @@ pub struct ServerConfig { /// presents the matching provisioning token. pub pending_registrations: Option>>>, + /// Consolidated pool handle (replaces the four ad-hoc Option> fields). + /// When `Some`, the pool updates it and the sampler reads it. + pub pool_status: Option>, + /// Observability handle to clone into AppState (heartbeat, limits). + /// `None` falls back to `Observability::noop()` (tests). + pub observability: Option, /// `PRELOOP_REQUIRE_JOB_ASSIGNMENTS`: refuse to dispatch any job without /// a recorded assignment, including to external runners. pub require_job_assignments: bool, @@ -77,6 +83,7 @@ impl std::fmt::Debug for ServerConfig { .field("oidc_issuer", &self.oidc_issuer) .field("enable_scheduler", &self.enable_scheduler) .field("pending_registrations", &self.pending_registrations) + .field("pool_status", &self.pool_status) .field("require_job_assignments", &self.require_job_assignments) .finish() } @@ -182,7 +189,8 @@ pub(crate) async fn reap_once(shared: &Arc) { .state .pool_preparing .as_ref() - .is_some_and(|flag| flag.load(std::sync::atomic::Ordering::Acquire)); + .is_some_and(|flag| flag.load(std::sync::atomic::Ordering::Acquire)) + || shared.state.pool_status.snapshot().preparing; let queued_jobs: Vec<_> = inner.queue.iter().cloned().collect(); let in_queue: std::collections::BTreeSet<(RunId, JobId)> = queued_jobs .iter() @@ -397,10 +405,15 @@ async fn run_background_reaper(shared: Arc) { let mut interval = tokio::time::interval(Duration::from_secs(10)); // Skip the first tick interval.tick().await; + //Heartbeat for reaper (critical) — beat each interval, no cadence change. + let heartbeat = shared.state.observability.heartbeat().clone(); + let _reaper_handle = heartbeat.register("reaper", preloop_observability::Criticality::Critical); + heartbeat.beat("reaper"); while !shared.shutdown.is_cancelled() { tokio::select! { _ = interval.tick() => { + heartbeat.beat("reaper"); reap_once(&shared).await; } _ = shared.shutdown.cancelled() => { @@ -410,6 +423,176 @@ async fn run_background_reaper(shared: Arc) { } } +fn build_operational_snapshot_sync( + queue_len: usize, + pending_jobs_len: usize, + pending_expansions_len: usize, + expanding_len: usize, + runs_queued: u32, + runs_in_progress: u32, + runs_completed: u32, + registered: u32, + sessions_len: u32, + queue_jobs: Vec, + runner_labels: Vec>, + pool_snapshot: preloop_observability::status::PoolSnapshot, + observability: &preloop_observability::Observability, + started_at: std::time::Instant, + shutdown_requested: bool, + scheduler_enabled: bool, +) -> preloop_observability::status::OperationalSnapshot { + use chrono::Utc; + use preloop_observability::status::*; + let now = Utc::now(); + let uptime = started_at.elapsed().as_secs(); + // Claimability: distinguish claimable vs unclaimable using existing runner label matching. + let (claimable, unclaimable) = if queue_jobs.is_empty() { + (0, 0) + } else if runner_labels.is_empty() { + (0, queue_jobs.len() as u32) + } else if pool_snapshot.preparing { + // Temporarily unclaimable while pool prepares. + (0, queue_jobs.len() as u32) + } else { + let mut claimable = 0u32; + for job in &queue_jobs { + let matches = runner_labels + .iter() + .any(|labels| crate::runtime_scheduling::job_matches_runner(&job.runs_on, labels)); + if matches { + claimable += 1; + } + } + (claimable, queue_jobs.len() as u32 - claimable) + }; + + let oldest_ready_seconds = None; // TODO: track queued_at + + OperationalSnapshot { + schema_version: 1, + observed_at: now, + snapshot_age_seconds: 0.0, + overall: if shutdown_requested { + Overall::ShuttingDown + } else { + Overall::Ok + }, + service: ServiceSnapshot { + version: env!("CARGO_PKG_VERSION").to_string(), + instance_id: observability.instance_id().to_string(), + uptime_seconds: uptime, + shutdown_requested, + }, + runs: RunsSnapshot { + queued: runs_queued, + in_progress: runs_in_progress, + completed: runs_completed, + }, + jobs: JobsSnapshot { + ready: queue_len as u32, + dependency_blocked: pending_jobs_len as u32, + concurrency_blocked: 0, + pending_expansion: pending_expansions_len as u32, + expanding: expanding_len as u32, + claimable, + unclaimable, + oldest_ready_seconds, + }, + concurrency: ConcurrencySnapshot::default(), + scheduler: SchedulerSnapshot { + enabled: scheduler_enabled, + ..Default::default() + }, + runners: RunnersSnapshot { + registered, + sessions: sessions_len, + idle: 0, + busy: 0, + stale: 0, + max_poll_age_seconds: None, + max_lease_age_seconds: None, + }, + pool: pool_snapshot, + vms: VmFleetSnapshot { + source: VmSource::Unavailable, + sample_age_seconds: None, + ..Default::default() + }, + store: StoreSnapshot::default(), + storage: StorageSnapshot::default(), + limits: Vec::new(), + tasks: Vec::new(), + github: GithubSnapshot::default(), + debug: DebugSnapshot::default(), + telemetry: TelemetrySnapshot::default(), + conditions: Vec::new(), + } +} + +async fn run_state_sampler(shared: Arc) { + let heartbeat = shared.state.observability.heartbeat().clone(); + let _handle = heartbeat.register( + "state_sampler", + preloop_observability::Criticality::Critical, + ); + heartbeat.beat("state_sampler"); + let mut interval = tokio::time::interval(Duration::from_secs(5)); + // Immediate sample then every 5s. + interval.tick().await; + loop { + tokio::select! { + _ = interval.tick() => { + heartbeat.beat("state_sampler"); + // Clone needed state under lock, release, then build. + let (queue_len, pending_jobs_len, pending_expansions_len, expanding_len, runs_queued, runs_in_progress, runs_completed, registered, sessions_len, queue_jobs, runner_labels, scheduler_enabled) = { + let inner = shared.state.inner.lock().await; + let queue_len = inner.queue.len(); + let pending_jobs_len = inner.pending_jobs.len(); + let pending_expansions_len = inner.pending_expansions.len(); + let expanding_len = inner.expanding.len(); + let mut q = 0u32; let mut ip = 0u32; let mut c = 0u32; + for run in inner.runs.values() { + match run.status { + ExecutionStatus::Queued => q += 1, + ExecutionStatus::InProgress => ip += 1, + s if s.is_terminal() => c += 1, + _ => {} + } + } + let registered = inner.runners.len() as u32; + let sessions_len = inner.sessions.len() as u32; + let queue_jobs = inner.queue.iter().cloned().collect::>(); + let runner_labels = inner.runners.values().map(|r| r.labels.clone()).collect::>(); + let scheduler_enabled = shared.state.scheduler.is_some(); + // Clone pool snapshot outside inner lock? It's cheap, do after. + (queue_len, pending_jobs_len, pending_expansions_len, expanding_len, q, ip, c, registered, sessions_len, queue_jobs, runner_labels, scheduler_enabled) + }; + let pool_snapshot = shared.state.pool_status.snapshot(); + let snap = build_operational_snapshot_sync( + queue_len, + pending_jobs_len, + pending_expansions_len, + expanding_len, + runs_queued, + runs_in_progress, + runs_completed, + registered, + sessions_len, + queue_jobs, + runner_labels, + pool_snapshot, + &shared.state.observability, + shared.state.started_at, + shared.shutdown.is_cancelled(), + scheduler_enabled, + ); + *shared.state.status_snapshot.write() = snap; + } + _ = shared.shutdown.cancelled() => break, + } + } +} + fn is_routine_unix_disconnect(error: &(dyn std::error::Error + 'static)) -> bool { error .downcast_ref::() @@ -424,6 +607,37 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { config.store_url.as_deref(), ) .await?; + // Wire observability if supplied (CLI/server will pass its handle). + if let Some(obs) = config.observability.clone() { + state.observability = obs; + } + if let Some(ps) = config.pool_status.clone() { + state.pool_status = ps; + } + // Ensure uptime base is now (AppState::new set it, but re-arm after store load). + state.started_at = std::time::Instant::now(); + // Seed initial snapshot so /readyz and /status have data before first 5s tick. + { + let init = build_operational_snapshot_sync( + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + Vec::new(), + Vec::new(), + state.pool_status.snapshot(), + &state.observability, + state.started_at, + false, + false, + ); + *state.status_snapshot.write() = init; + } if let Some(queue_depth) = config.queue_depth.clone() { state.queue_depth = queue_depth; // The pool shares this same atomic and only forks a runner while it @@ -435,23 +649,46 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { state.inner.lock().await.queue.len(), std::sync::atomic::Ordering::Release, ); + state + .pool_status + .set_queue_depth(state.queue_depth.load(std::sync::atomic::Ordering::Acquire) as u32); } if let Some(next_job_runs_on) = config.next_job_runs_on.clone() { state.next_job_runs_on = next_job_runs_on; + if let Ok(v) = state.next_job_runs_on.read() { + state.pool_status.set_next_job_runs_on(v.clone()); + } } { let inner = state.inner.lock().await; crate::runtime_scheduling::sync_next_job_labels(&inner, &state.next_job_runs_on); + if state.pool_status.snapshot().next_job_runs_on.is_empty() { + if let Ok(v) = state.next_job_runs_on.read() { + state.pool_status.set_next_job_runs_on(v.clone()); + } + } } { let pool_managed = config.pending_registrations.is_some(); if let Some(pending_registrations) = config.pending_registrations.clone() { state.pending_registrations = pending_registrations; + // Mirror into consolidated handle for sampler visibility + if let Ok(map) = state.pending_registrations.read() { + for (k, v) in map.iter() { + state.pool_status.insert_pending(k.clone(), *v); + } + } } let mut inner = state.inner.lock().await; inner.pool_assignments_enabled = pool_managed; inner.require_job_assignments = config.require_job_assignments; } + if let Some(pool_preparing) = config.pool_preparing.clone() { + state.pool_preparing = Some(pool_preparing.clone()); + if pool_preparing.load(std::sync::atomic::Ordering::Acquire) { + state.pool_status.set_preparing(true); + } + } if !config.listen.ip().is_loopback() && state.system_token == DEFAULT_PRELOOP_SYSTEM_TOKEN { anyhow::bail!( "PRELOOP_SYSTEM_TOKEN must be explicitly configured when listening beyond loopback" @@ -467,6 +704,8 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { inner.oidc_issuer = oidc_issuer; } let shutdown = CancellationToken::new(); + // Heartbeat for scheduler scan (critical) if enabled — beat periodically. + let scheduler_heartbeat = state.observability.heartbeat().clone(); if config.enable_scheduler { let scheduler = crate::scheduler::Scheduler::new(); state.scheduler = Some(scheduler.clone()); @@ -475,6 +714,24 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { shutdown: shutdown.clone(), }); let scheduler_clone = scheduler.clone(); + // Spawn a holder task for scheduler_scan heartbeat — keep handle alive for lifetime. + let hb_for_holder = scheduler_heartbeat.clone(); + let shutdown_for_holder = shutdown.clone(); + tokio::spawn(async move { + let _handle = hb_for_holder.register( + "scheduler_scan", + preloop_observability::Criticality::Critical, + ); + hb_for_holder.beat("scheduler_scan"); + let mut int = tokio::time::interval(Duration::from_secs(10)); + int.tick().await; + while !shutdown_for_holder.is_cancelled() { + tokio::select! { + _ = int.tick() => hb_for_holder.beat("scheduler_scan"), + _ = shutdown_for_holder.cancelled() => break, + } + } + }); if let Some(workspace) = state.local_workspace.clone() { tokio::spawn(async move { scheduler_clone @@ -558,12 +815,17 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { }; let router = build_app(state.clone(), shutdown.clone(), test_api_token); - state.pool_preparing = config.pool_preparing.clone(); let shared = Arc::new(SharedState { - state, + state: state.clone(), shutdown: shutdown.clone(), }); + // 5s sampler — clone needed state under lock, release, then publish. + let sampler_shared = shared.clone(); + tokio::spawn(async move { + run_state_sampler(sampler_shared).await; + }); + let checker_shared = shared.clone(); tokio::spawn(async move { run_background_reaper(checker_shared).await; diff --git a/crates/preloop-runner-server/src/lib_tests.rs b/crates/preloop-runner-server/src/lib_tests.rs index 91e5d4f8..1a4d802d 100644 --- a/crates/preloop-runner-server/src/lib_tests.rs +++ b/crates/preloop-runner-server/src/lib_tests.rs @@ -18536,6 +18536,8 @@ fn server_config_debug_redacts_store_url_password() { oidc_issuer: None, enable_scheduler: false, pending_registrations: None, + pool_status: None, + observability: None, require_job_assignments: false, }; let debug = format!("{config:?}"); diff --git a/crates/preloop-runner-server/src/main.rs b/crates/preloop-runner-server/src/main.rs index aed62825..5acd779d 100644 --- a/crates/preloop-runner-server/src/main.rs +++ b/crates/preloop-runner-server/src/main.rs @@ -130,6 +130,8 @@ async fn main() -> anyhow::Result<()> { next_job_runs_on: None, pool_preparing: None, listen, + pool_status: None, + observability: None, systemd_socket_activation: false, unix_socket, state_dir, diff --git a/crates/preloop-runner-server/src/openapi.rs b/crates/preloop-runner-server/src/openapi.rs index 068f60e9..2d65b6f0 100644 --- a/crates/preloop-runner-server/src/openapi.rs +++ b/crates/preloop-runner-server/src/openapi.rs @@ -179,7 +179,10 @@ pub(crate) struct RunResponse { list_dispatch_runs, github_register, github_callback, - list_runners + list_runners, + readyz, + status, + metrics ), components( schemas( @@ -282,6 +285,38 @@ type JsonValue = serde_json::Value; )] fn healthz() {} +/// Server readiness check (public, reason codes on 503). +#[utoipa::path( + get, path = "/readyz", tag = "Health", + responses( + (status = 200, description = "Ready", body = JsonValue), + (status = 503, description = "Not ready", body = JsonValue) + ) +)] +fn readyz() {} + +/// Operational status snapshot (native bearer required). +#[utoipa::path( + get, path = "/api/v1/status", tag = "Health", + responses( + (status = 200, description = "Operational snapshot", body = JsonValue), + (status = 401, description = "Unauthorized", body = ApiErrorResponse) + ), + security(("native_bearer" = [])) +)] +fn status() {} + +/// Prometheus metrics (native bearer required). +#[utoipa::path( + get, path = "/metrics", tag = "Health", + responses( + (status = 200, description = "Prometheus text", content_type = "text/plain", body = String), + (status = 401, description = "Unauthorized", body = ApiErrorResponse) + ), + security(("native_bearer" = [])) +)] +fn metrics() {} + // ── Runs ──────────────────────────────────────────────────────────────────── /// Submit a workflow run. diff --git a/crates/preloop-runner-server/src/routes.rs b/crates/preloop-runner-server/src/routes.rs index d513e3a6..3757938f 100644 --- a/crates/preloop-runner-server/src/routes.rs +++ b/crates/preloop-runner-server/src/routes.rs @@ -252,8 +252,18 @@ pub(crate) fn build_app( crate::dispatch_auth::require_dispatch_auth, )); + let observability_routes = Router::new() + .route("/api/v1/status", get(status)) + .route("/metrics", get(metrics)) + .route_layer(middleware::from_fn_with_state( + shared.clone(), + require_native_bearer, + )); + let router = Router::new() .route("/healthz", get(healthz)) + .route("/readyz", get(readyz)) + .merge(observability_routes) .route("/runs/:run_id", get(get_public_run)) .route( "/openapi.json", diff --git a/crates/preloop-runner-server/src/runs.rs b/crates/preloop-runner-server/src/runs.rs index 90801dbf..b32a62c5 100644 --- a/crates/preloop-runner-server/src/runs.rs +++ b/crates/preloop-runner-server/src/runs.rs @@ -1,12 +1,137 @@ use super::*; use std::collections::BTreeSet; -pub(crate) async fn healthz(State(shared): State>) -> Json { - Json(json!({ - "ok": true, +pub(crate) async fn healthz(State(shared): State>) -> impl IntoResponse { + let shutdown = shared.shutdown.is_cancelled(); + let body = json!({ + "ok": !shutdown, "protocol_version": PROTOCOL_VERSION, - "shutdown_requested": shared.shutdown.is_cancelled(), - })) + "shutdown_requested": shutdown, + }); + if shutdown { + (StatusCode::SERVICE_UNAVAILABLE, Json(body)).into_response() + } else { + (StatusCode::OK, Json(body)).into_response() + } +} + +pub(crate) async fn readyz(State(shared): State>) -> impl IntoResponse { + if shared.shutdown.is_cancelled() { + let body = json!({ "ready": false, "reason": "shutting_down" }); + return (StatusCode::SERVICE_UNAVAILABLE, Json(body)).into_response(); + } + // Check critical heartbeats freshness (>15s stale is 3 intervals of 5s sampler) + if let Some(stale) = shared + .state + .observability + .heartbeat() + .any_critical_stale(Duration::from_secs(15)) + { + let body = json!({ "ready": false, "reason": format!("task_stale:{}", stale) }); + return (StatusCode::SERVICE_UNAVAILABLE, Json(body)).into_response(); + } + // Also check snapshot age (>15s stale sampler) + let age_secs = { + let snap = shared.state.status_snapshot.read(); + let now = chrono::Utc::now(); + (now - snap.observed_at).num_milliseconds() as f64 / 1000.0 + }; + if age_secs > 15.0 { + let body = json!({ "ready": false, "reason": "state_sampler_stale" }); + return (StatusCode::SERVICE_UNAVAILABLE, Json(body)).into_response(); + } + let body = json!({ "ready": true, "reason": serde_json::Value::Null }); + (StatusCode::OK, Json(body)).into_response() +} + +pub(crate) async fn status(State(shared): State>) -> impl IntoResponse { + // Fail-open, no InnerState lock — clone cached snapshot and update age. + let mut snap = shared.state.status_snapshot.read().clone(); + let now = chrono::Utc::now(); + let age = (now - snap.observed_at).num_milliseconds() as f64 / 1000.0; + snap.snapshot_age_seconds = if age.is_finite() && age >= 0.0 { + age + } else { + 0.0 + }; + // Also surface current heartbeat tasks without holding InnerState + // (best-effort: caller sees last sampler's tasks plus live heartbeat snapshot) + // We keep sampler's tasks but also append live task snapshot if empty. + if snap.tasks.is_empty() { + snap.tasks = shared + .state + .observability + .heartbeat() + .snapshot() + .into_iter() + .map(|t| preloop_observability::status::TaskEntry { + name: t.name.to_string(), + critical: t.critical == preloop_observability::Criticality::Critical, + heartbeat_age_seconds: t.heartbeat_age.as_secs_f64(), + state: if t.exited { + "exited".to_string() + } else if t.heartbeat_age > Duration::from_secs(15) { + "stale".to_string() + } else { + "running".to_string() + }, + }) + .collect(); + } + Json(snap).into_response() +} + +pub(crate) async fn metrics(State(shared): State>) -> impl IntoResponse { + let snap = shared.state.status_snapshot.read().clone(); + let mut out = String::new(); + out.push_str("# HELP preloop_service_uptime_seconds Service uptime in seconds.\n"); + out.push_str("# TYPE preloop_service_uptime_seconds gauge\n"); + out.push_str(&format!( + "preloop_service_uptime_seconds {}\n", + snap.service.uptime_seconds + )); + out.push_str("# HELP preloop_pool_desired Desired pool size.\n"); + out.push_str("# TYPE preloop_pool_desired gauge\n"); + out.push_str(&format!("preloop_pool_desired {}\n", snap.pool.desired)); + out.push_str("# HELP preloop_pool_preparing Pool preparing signal.\n"); + out.push_str("# TYPE preloop_pool_preparing gauge\n"); + out.push_str(&format!( + "preloop_pool_preparing {}\n", + if snap.pool.preparing { 1 } else { 0 } + )); + out.push_str("# HELP preloop_pool_idle Idle runners in pool.\n"); + out.push_str("# TYPE preloop_pool_idle gauge\n"); + out.push_str(&format!("preloop_pool_idle {}\n", snap.pool.idle)); + out.push_str("# HELP preloop_pool_busy Busy runners in pool.\n"); + out.push_str("# TYPE preloop_pool_busy gauge\n"); + out.push_str(&format!("preloop_pool_busy {}\n", snap.pool.busy)); + out.push_str("# HELP preloop_job_queue_depth Number of jobs by queue.\n"); + out.push_str("# TYPE preloop_job_queue_depth gauge\n"); + out.push_str(&format!( + "preloop_job_queue_depth{{queue=\"ready\"}} {}\n", + snap.jobs.ready + )); + out.push_str(&format!( + "preloop_job_queue_depth{{queue=\"claimable\"}} {}\n", + snap.jobs.claimable + )); + out.push_str(&format!( + "preloop_job_queue_depth{{queue=\"unclaimable\"}} {}\n", + snap.jobs.unclaimable + )); + out.push_str(&format!( + "preloop_job_queue_depth{{queue=\"dependency_blocked\"}} {}\n", + snap.jobs.dependency_blocked + )); + let body = out; + ( + [( + header::CONTENT_TYPE, + "text/plain; version=0.0.4; charset=utf-8", + )], + body, + ) + .into_response() } /// GitHub's `system.orchestrationId`: `{planId}.{jobId}.{suffix}` where the diff --git a/crates/preloop-runner-server/src/state.rs b/crates/preloop-runner-server/src/state.rs index f06e70b3..40714d52 100644 --- a/crates/preloop-runner-server/src/state.rs +++ b/crates/preloop-runner-server/src/state.rs @@ -364,6 +364,15 @@ pub struct AppState { /// lock. Monotonically increases; the inner counter is no longer the /// source of truth once this is in use. pub(crate) next_request_id: Arc, + /// Observability handle (cloneable, holds heartbeat & limit registries). + pub(crate) observability: preloop_observability::Observability, + /// Cached operational snapshot, updated every 5s by the sampler without holding `inner`. + pub status_snapshot: + Arc>, + /// Consolidated pool handle replacing the four ad-hoc Option> fields. + pub pool_status: Arc, + /// When this AppState was created (for uptime). + pub(crate) started_at: std::time::Instant, /// Jobs accepted and still waiting for a runner, refreshed whenever one /// is claimed. A supervising runner pool reads it to decide whether the /// work already queued outruns the runners it has left. @@ -786,6 +795,12 @@ impl AppState { events, message_notify: Arc::new(Notify::new()), next_request_id: Arc::new(std::sync::atomic::AtomicI64::new(next_request_id)), + observability: preloop_observability::Observability::noop(), + status_snapshot: Arc::new(parking_lot::RwLock::new( + preloop_observability::status::OperationalSnapshot::default(), + )), + pool_status: Arc::new(preloop_observability::status::PoolStatus::default()), + started_at: std::time::Instant::now(), // Mirror the recovered ready-queue size so an on-demand runner // pool spawns against the right workload after restart. queue_depth: Arc::new(std::sync::atomic::AtomicUsize::new(recovered_queue_len)), From faf32839381b8761a301e968bfa634a45a249663 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 17:21:46 -0400 Subject: [PATCH 07/22] chore: remove plan stage markers from code comments Entire-Checkpoint: 01M0GGTB7CC3AHHF26HT26K88Z --- crates/preloop-cli/src/main.rs | 4 ++-- crates/preloop-observability/Cargo.toml | 2 +- crates/preloop-observability/src/lib.rs | 12 ++++++------ crates/preloop-observability/src/status.rs | 10 +++------- crates/preloop-runner-server/src/main.rs | 4 ++-- docs/internal/observability.md | 2 +- 6 files changed, 15 insertions(+), 19 deletions(-) diff --git a/crates/preloop-cli/src/main.rs b/crates/preloop-cli/src/main.rs index 66463d71..549d4c69 100644 --- a/crates/preloop-cli/src/main.rs +++ b/crates/preloop-cli/src/main.rs @@ -743,7 +743,7 @@ struct ShellArgs { #[tokio::main] async fn main() -> anyhow::Result<()> { - // Unified observability init (Step 2): one handle for the process, shared + // Unified observability init: one handle for the process, shared // with `AppState` and `RunnerPoolConfig` later. `PRELOOP_LOG_FORMAT` now // controls pretty/json/auto (auto = pretty on TTY, JSON when piped), and // `RUST_LOG` defaults to `info` (the old `fmt::init()` default of ERROR hid @@ -754,7 +754,7 @@ async fn main() -> anyhow::Result<()> { preloop_observability::Observability::from_config(obs_config); preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); // Keep the handle alive; the pool/server will clone it later. Suppress - // unused warning until the wiring lands in Step 3. + // unused warning until the wiring lands. let _observability = observability; let _observability_runtime = observability_runtime; diff --git a/crates/preloop-observability/Cargo.toml b/crates/preloop-observability/Cargo.toml index 6a8e37b0..18bb5bec 100644 --- a/crates/preloop-observability/Cargo.toml +++ b/crates/preloop-observability/Cargo.toml @@ -3,7 +3,7 @@ name = "preloop-observability" version = "0.1.0" edition = "2021" license = "MIT" -description = "Observability handle for Preloop — metrics, logs, traces, status snapshot (Step 2)" +description = "Observability handle for Preloop — metrics, logs, traces, status snapshot" repository = "https://github.com/preloopdev/preloop" [dependencies] diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs index cf5662ed..7ebd6531 100644 --- a/crates/preloop-observability/src/lib.rs +++ b/crates/preloop-observability/src/lib.rs @@ -1,4 +1,4 @@ -//! `preloop-observability` — Step 2 of Plan 002. +//! `preloop-observability` — observability handle for Preloop. //! //! Small, explicit API with no dependency on server/orchestrator internals. Both //! `preloop` and `preloop-server` construct one handle/runtime before building @@ -93,7 +93,7 @@ impl ObservabilityConfig { // CLI defaults to `info` when unset; the standalone server historically // used `EnvFilter::from_default_env()` with no fallback (silent when - // unset). We unify on `info` per Step 2. + // unset). We unify on `info`. let rust_log = std::env::var("RUST_LOG").unwrap_or_else(|_| "info".to_string()); let service_name = @@ -475,7 +475,7 @@ pub struct ObservabilityRuntime { impl ObservabilityRuntime { fn new(handle: Observability) -> Self { - // Step 2 does not install the global subscriber here — the binaries do + // Does not install the global subscriber here — the binaries do // that via `install_fmt_subscriber`. This runtime is the place for the // future OTel provider guards and the 2s flush on `Drop`. Self { @@ -513,10 +513,10 @@ impl ObservabilityRuntime { /// Attempt to flush exporters for at most 2s. Export failure is logged, never propagated. pub async fn shutdown(self) { - // Step 2 has no exporter worker yet; this is the bounded-flush seam for + // No exporter worker yet; this is the bounded-flush seam for // the future OTel BatchSpanProcessor / metrics reader. tokio::time::timeout(Duration::from_secs(2), async { - // No-op until OTel providers are wired in Step 3+. + // No-op until OTLP providers are wired. tokio::task::yield_now().await; }) .await @@ -531,7 +531,7 @@ impl fmt::Debug for ObservabilityRuntime { } // --------------------------------------------------------------------------- -// Tests — Step 2 gates +// Tests // --------------------------------------------------------------------------- #[cfg(test)] diff --git a/crates/preloop-observability/src/status.rs b/crates/preloop-observability/src/status.rs index b4853362..b822a0b5 100644 --- a/crates/preloop-observability/src/status.rs +++ b/crates/preloop-observability/src/status.rs @@ -1,7 +1,6 @@ -//! Step 3 DTOs — OperationalSnapshot and supporting types. +//!OperationalSnapshot and supporting types. //! -//! Neutral, serializable DTOs shared by server and CLI. No dependency on -//! server/orchestrator internals. Metric labels use only bounded enums. + use std::sync::Arc; @@ -237,7 +236,7 @@ impl PoolStatus { } // --------------------------------------------------------------------------- -// VMs (stubbed for Step 3, full host sampler is Step 5) +// VMs // --------------------------------------------------------------------------- #[derive(Debug, Clone, Serialize, Deserialize)] @@ -403,9 +402,6 @@ pub struct TaskEntry { pub state: String, } -// --------------------------------------------------------------------------- -// Conditions -// --------------------------------------------------------------------------- #[derive(Debug, Clone, Serialize, Deserialize)] pub struct Condition { diff --git a/crates/preloop-runner-server/src/main.rs b/crates/preloop-runner-server/src/main.rs index 5acd779d..44c46253 100644 --- a/crates/preloop-runner-server/src/main.rs +++ b/crates/preloop-runner-server/src/main.rs @@ -75,10 +75,10 @@ async fn main() -> anyhow::Result<()> { .install_default() .ok(); - // Unified observability init (Step 2): `RUST_LOG` now defaults to `info` + // Unified observability init: `RUST_LOG` now defaults to `info` // like the CLI, instead of falling silent when unset. `PRELOOP_LOG_FORMAT` // controls pretty/json/auto. The `Observability` handle will be cloned - // into `AppState` in Step 3; for now it is held for the life of `main`. + // into `AppState`; for now it is held for the life of `main`. let obs_config = preloop_observability::ObservabilityConfig::from_env(); let (observability, observability_runtime) = preloop_observability::Observability::from_config(obs_config); diff --git a/docs/internal/observability.md b/docs/internal/observability.md index b2285a5c..98318f56 100644 --- a/docs/internal/observability.md +++ b/docs/internal/observability.md @@ -1,6 +1,6 @@ # Observability — Internal Signal & Security Contract -> **Status:** Step 1 of Plan 002 (`plans/002-observability-strategy.md` revised at `673bdfa0`). This is the **internal** contract that must land before any OTLP export is wired. The public `docs/observability.md` will be a redacted subset later. Do not publish this file. +> **Status:** Internal contract for Plan 002 (`plans/002-observability-strategy.md` revised at `673bdfa0`). This is the **internal** contract that must land before any OTLP export is wired. The public `docs/observability.md` will be a redacted subset later. Do not publish this file. ## Why From 6e4a82c1049dd03939f5451e38382224d2c54d1b Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 17:55:02 -0400 Subject: [PATCH 08/22] feat(server): instrument HTTP and store with bounded metrics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire the observability handle into the hot path without holding the global lock during export. Replace the default TraceLayer that leaked raw URIs with a safe middleware that records method, matched route template (never concrete ID or query, 1000 IDs remain one series), finite surface (native, runner, broker, results, webhook, git, oidc, live_logs, public, test, unknown) and status class. The live_logs WebSocket is excluded from the duration histogram and will be tracked via livelog connections instead. Active requests are an updown gauge; durations use the 0.005..10s buckets. Add a shared MetricsRegistry in preloop-observability (HttpMetrics + StoreMetrics) with Prometheus text rendering. The registry is cloneable via the Observability handle (already in AppState) and is fed only from the cached snapshot or via the InstrumentedStore wrapper — no await while holding InnerState. Wrap the private Store trait once in store.rs with InstrumentedStore: all seven methods (load_into, store_inner, store_meta_only, store_run_event, store_workflow_run_counter, store_log_chunk, append_event) record preloop.store.operation.duration with backend, operation, outcome and the consecutive-failures gauge, preserving every return/error and the best-effort persistence rule. Bootstrap wraps the store returned by open_store with the same observability handle (backend sqlite vs postgres detected from PRELOOP_STORE_URL). Extend GET /metrics to append the registry's http and store exposition after the snapshot-based pool/queue gauges, still behind native bearer. cargo check, fmt, sg-scan-strict and preloop-observability tests (12, including route normalization and bounded-series) pass; run/job and broker lifecycle counters remain for the next change. Entire-Checkpoint: 01M0GJQ8PAGXWHJTR7WHR0SGVS --- crates/preloop-observability/src/lib.rs | 8 + crates/preloop-observability/src/metrics.rs | 405 ++++++++++++++++++ crates/preloop-observability/src/status.rs | 8 +- crates/preloop-runner-server/src/bootstrap.rs | 19 + .../preloop-runner-server/src/http_metrics.rs | 103 +++++ crates/preloop-runner-server/src/lib.rs | 2 +- crates/preloop-runner-server/src/routes.rs | 5 +- crates/preloop-runner-server/src/runs.rs | 2 + crates/preloop-runner-server/src/store.rs | 143 +++++++ 9 files changed, 686 insertions(+), 9 deletions(-) create mode 100644 crates/preloop-observability/src/metrics.rs create mode 100644 crates/preloop-runner-server/src/http_metrics.rs diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs index 7ebd6531..3da05d42 100644 --- a/crates/preloop-observability/src/lib.rs +++ b/crates/preloop-observability/src/lib.rs @@ -11,6 +11,7 @@ //! - Always retain `stderr`/`journald` even when OTLP is configured. //! - `Debug` on config never reveals headers or credential-bearing endpoint parts. +pub mod metrics; pub mod status; use std::collections::HashMap; @@ -381,6 +382,7 @@ struct Inner { config: Arc, heartbeat: TaskHeartbeat, limits: LimitRegistry, + metrics: Arc, is_noop: bool, } @@ -407,6 +409,7 @@ impl Observability { config: Arc::new(config), heartbeat: TaskHeartbeat::default(), limits: LimitRegistry::default(), + metrics: Arc::new(metrics::MetricsRegistry::default()), is_noop: true, }), } @@ -420,6 +423,7 @@ impl Observability { config: Arc::new(config), heartbeat: TaskHeartbeat::default(), limits: LimitRegistry::default(), + metrics: Arc::new(metrics::MetricsRegistry::default()), is_noop, }), }; @@ -451,6 +455,10 @@ impl Observability { &self.inner.limits } + pub fn metrics(&self) -> &metrics::MetricsRegistry { + &self.inner.metrics + } + pub fn config(&self) -> &ObservabilityConfig { &self.inner.config } diff --git a/crates/preloop-observability/src/metrics.rs b/crates/preloop-observability/src/metrics.rs new file mode 100644 index 00000000..1d504461 --- /dev/null +++ b/crates/preloop-observability/src/metrics.rs @@ -0,0 +1,405 @@ +//! Metrics registry for HTTP and store — Step 4. +//! +//! In-memory, bounded, no network I/O. Instruments and attribute arrays are +//! prebuilt where static; gauges are updated from the cached snapshot. + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use parking_lot::RwLock; + +// --------------------------------------------------------------------------- +// Histogram buckets +// --------------------------------------------------------------------------- + +pub const HTTP_BUCKETS: &[f64] = &[ + 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, +]; +pub const STORE_BUCKETS: &[f64] = &[ + 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, +]; +pub const QUEUE_BUCKETS: &[f64] = &[ + 0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0, 900.0, +]; + +#[derive(Debug, Default, Clone)] +struct Histogram { + buckets: Vec<(f64, u64)>, // (le, count) + count: u64, + sum: f64, +} + +impl Histogram { + fn new(buckets: &[f64]) -> Self { + Self { + buckets: buckets.iter().map(|&le| (le, 0)).collect(), + count: 0, + sum: 0.0, + } + } + + fn observe(&mut self, value: f64) { + self.count += 1; + self.sum += value; + for (le, cnt) in &mut self.buckets { + if value <= *le { + *cnt += 1; + } + } + } +} + +// --------------------------------------------------------------------------- +// Http metrics +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct HttpLabels { + pub method: String, + pub route: String, + pub surface: String, + pub status_class: String, +} + +#[derive(Debug, Default)] +pub struct HttpMetrics { + active: RwLock>, + durations: RwLock>, +} + +impl HttpMetrics { + pub fn inc_active(&self, labels: &HttpLabels) { + *self.active.write().entry(labels.clone()).or_insert(0) += 1; + } + + pub fn dec_active(&self, labels: &HttpLabels) { + let mut g = self.active.write(); + if let Some(v) = g.get_mut(labels) { + *v -= 1; + if *v <= 0 { + g.remove(labels); + } + } + } + + pub fn observe_duration(&self, labels: HttpLabels, duration: Duration) { + let secs = duration.as_secs_f64(); + let mut g = self.durations.write(); + let hist = g + .entry(labels) + .or_insert_with(|| Histogram::new(HTTP_BUCKETS)); + hist.observe(secs); + } + + pub fn render(&self, out: &mut String) { + out.push_str("# HELP http_server_request_duration_seconds API latency — matched route, never raw URI\n"); + out.push_str("# TYPE http_server_request_duration_seconds histogram\n"); + let g = self.durations.read(); + for (labels, hist) in g.iter() { + for (le, cnt) in &hist.buckets { + out.push_str(&format!( + "http_server_request_duration_seconds_bucket{{method=\"{}\",route=\"{}\",surface=\"{}\",status_class=\"{}\",le=\"{}\"}} {}\n", + labels.method, labels.route, labels.surface, labels.status_class, le, cnt + )); + } + out.push_str(&format!( + "http_server_request_duration_seconds_bucket{{method=\"{}\",route=\"{}\",surface=\"{}\",status_class=\"{}\",le=\"+Inf\"}} {}\n", + labels.method, labels.route, labels.surface, labels.status_class, hist.count + )); + out.push_str(&format!( + "http_server_request_duration_seconds_sum{{method=\"{}\",route=\"{}\",surface=\"{}\",status_class=\"{}\"}} {}\n", + labels.method, labels.route, labels.surface, labels.status_class, hist.sum + )); + out.push_str(&format!( + "http_server_request_duration_seconds_count{{method=\"{}\",route=\"{}\",surface=\"{}\",status_class=\"{}\"}} {}\n", + labels.method, labels.route, labels.surface, labels.status_class, hist.count + )); + } + out.push_str("# HELP http_server_active_requests Current HTTP concurrency\n"); + out.push_str("# TYPE http_server_active_requests gauge\n"); + let g2 = self.active.read(); + for (labels, v) in g2.iter() { + out.push_str(&format!( + "http_server_active_requests{{method=\"{}\",route=\"{}\",surface=\"{}\"}} {}\n", + labels.method, labels.route, labels.surface, v + )); + } + } + + #[cfg(test)] + pub fn clear(&self) { + self.active.write().clear(); + self.durations.write().clear(); + } + + #[cfg(test)] + pub fn series_count(&self) -> usize { + self.durations.read().len() + } +} + +// --------------------------------------------------------------------------- +// Store metrics +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct StoreLabels { + pub backend: String, + pub operation: String, + pub outcome: String, +} + +#[derive(Debug, Default)] +pub struct StoreMetrics { + durations: RwLock>, + consecutive_failures: RwLock>, // backend -> count +} + +impl StoreMetrics { + pub fn observe(&self, backend: &str, operation: &str, outcome: &str, duration: Duration) { + let labels = StoreLabels { + backend: backend.to_string(), + operation: operation.to_string(), + outcome: outcome.to_string(), + }; + let mut g = self.durations.write(); + let hist = g + .entry(labels) + .or_insert_with(|| Histogram::new(STORE_BUCKETS)); + hist.observe(duration.as_secs_f64()); + + // Update consecutive failures + let mut cf = self.consecutive_failures.write(); + if outcome == "error" { + *cf.entry(backend.to_string()).or_insert(0) += 1; + } else { + cf.insert(backend.to_string(), 0); + } + } + + pub fn render(&self, out: &mut String) { + out.push_str("# HELP preloop_store_operation_duration_seconds Store operation latency\n"); + out.push_str("# TYPE preloop_store_operation_duration_seconds histogram\n"); + let g = self.durations.read(); + for (labels, hist) in g.iter() { + for (le, cnt) in &hist.buckets { + out.push_str(&format!( + "preloop_store_operation_duration_seconds_bucket{{backend=\"{}\",operation=\"{}\",outcome=\"{}\",le=\"{}\"}} {}\n", + labels.backend, labels.operation, labels.outcome, le, cnt + )); + } + out.push_str(&format!( + "preloop_store_operation_duration_seconds_bucket{{backend=\"{}\",operation=\"{}\",outcome=\"{}\",le=\"+Inf\"}} {}\n", + labels.backend, labels.operation, labels.outcome, hist.count + )); + out.push_str(&format!( + "preloop_store_operation_duration_seconds_sum{{backend=\"{}\",operation=\"{}\",outcome=\"{}\"}} {}\n", + labels.backend, labels.operation, labels.outcome, hist.sum + )); + out.push_str(&format!( + "preloop_store_operation_duration_seconds_count{{backend=\"{}\",operation=\"{}\",outcome=\"{}\"}} {}\n", + labels.backend, labels.operation, labels.outcome, hist.count + )); + } + out.push_str("# HELP preloop_store_consecutive_failures Restart-durability risk\n"); + out.push_str("# TYPE preloop_store_consecutive_failures gauge\n"); + let g2 = self.consecutive_failures.read(); + for (backend, v) in g2.iter() { + out.push_str(&format!( + "preloop_store_consecutive_failures{{backend=\"{}\"}} {}\n", + backend, v + )); + } + } + + #[cfg(test)] + pub fn clear(&self) { + self.durations.write().clear(); + self.consecutive_failures.write().clear(); + } +} + +// --------------------------------------------------------------------------- +// Registry +// --------------------------------------------------------------------------- + +#[derive(Debug, Default)] +pub struct MetricsRegistry { + pub http: HttpMetrics, + pub store: StoreMetrics, +} + +impl MetricsRegistry { + pub fn render(&self) -> String { + let mut out = String::new(); + self.http.render(&mut out); + self.store.render(&mut out); + out + } + + #[cfg(test)] + pub fn clear(&self) { + self.http.clear(); + self.store.clear(); + } +} + +// --------------------------------------------------------------------------- +// Helpers — surface classification and route normalization +// --------------------------------------------------------------------------- + +/// Classify a normalized route template into a finite surface. +pub fn classify_surface(route: &str) -> &'static str { + if route == "/healthz" || route == "/readyz" || route == "/metrics" { + return "public"; + } + if route.starts_with("/api/v1") { + return "native"; + } + if route.starts_with("/_apis") || route.starts_with("/runner") { + return "runner"; + } + if route.starts_with("/broker") { + return "broker"; + } + if route.starts_with("/twirp") || route.starts_with("/twirp-blob") { + return "results"; + } + if route.starts_with("/ws/live-logs") { + return "live_logs"; + } + if route.starts_with("/snapshots") || route.starts_with("/repos") { + return "git"; + } + if route.starts_with("/oidc") || route.starts_with("/.well-known") { + return "oidc"; + } + if route == "/webhook" || route.starts_with("/webhook") { + return "webhook"; + } + if route.starts_with("/internal/test") { + return "test"; + } + "unknown" +} + +/// Normalize a raw path (with concrete IDs) to a bounded route template. +/// +/// Uses Axum's matched templates where available; otherwise falls back to +/// prefix matching for the known route set. Query strings are stripped. +pub fn normalize_route(raw: &str) -> String { + // Strip query + let path = raw.split('?').next().unwrap_or(raw); + // Already a template? (contains ':') + if path.contains(':') { + return path.to_string(); + } + // Known templates — longest prefix first + const TEMPLATES: &[&str] = &[ + "/api/v1/runs/:run_id", + "/api/v1/runs", + "/api/v1/status", + "/api/v1/scheduler/history", + "/api/v1/debug/sessions", + "/api/v1/github/register", + "/api/v1/github/callback", + "/_apis/artifactcache/cache/:cache_id", + "/_apis/artifactcache/cache", + "/_apis/pipelines/workflows/:run_id/artifacts/:artifact_id", + "/_apis/pipelines/workflows/:run_id/artifacts", + "/runner/server/_apis/distributedtask/pools/:pool_id/agents", + "/runner/server/_apis/distributedtask/pools", + "/broker/:runner_id/acquirejob", + "/broker/:runner_id/renewjob", + "/broker/:runner_id/completejob", + "/twirp/github.actions.results.api.v1.ArtifactService/CreateArtifact", + "/twirp/github.actions.results.api.v1.CacheService/CreateCacheEntry", + "/twirp-blob/:kind/:token", + "/ws/live-logs/:job_id", + "/snapshots", + "/repos", + "/oidc", + "/.well-known", + "/webhook", + "/healthz", + "/readyz", + "/metrics", + ]; + for tmpl in TEMPLATES { + // Template without params matches exactly + if !tmpl.contains(':') && path == *tmpl { + return tmpl.to_string(); + } + // Template with params: match prefix up to first ':' + if let Some(colon) = tmpl.find(':') { + let prefix = &tmpl[..colon - 1]; // up to '/' before ':' + if path.starts_with(prefix) { + // Ensure it's a segment boundary: /api/v1/runs/abc should match /api/v1/runs/:run_id + // but /api/v1/runsXYZ should not. + let rest = &path[prefix.len()..]; + if rest.is_empty() || rest.starts_with('/') { + return tmpl.to_string(); + } + } + } + } + // Unknown — constant label, never raw path + "/unknown".to_string() +} + +pub fn status_class(status: u16) -> &'static str { + match status { + 200..=299 => "2xx", + 300..=399 => "3xx", + 400..=499 => "4xx", + 500..=599 => "5xx", + _ => "unknown", + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn normalize_concrete_id_to_template() { + assert_eq!( + normalize_route("/api/v1/runs/abc123"), + "/api/v1/runs/:run_id" + ); + assert_eq!( + normalize_route("/api/v1/runs/abc123?foo=bar"), + "/api/v1/runs/:run_id" + ); + assert_eq!(normalize_route("/api/v1/status"), "/api/v1/status"); + assert_eq!(normalize_route("/unknown/path/xyz"), "/unknown"); + } + + #[test] + fn classify() { + assert_eq!(classify_surface("/api/v1/runs"), "native"); + assert_eq!(classify_surface("/_apis/artifactcache/cache"), "runner"); + assert_eq!(classify_surface("/broker/42/acquirejob"), "broker"); + assert_eq!(classify_surface("/ws/live-logs/123"), "live_logs"); + assert_eq!(classify_surface("/healthz"), "public"); + assert_eq!(classify_surface("/unknown"), "unknown"); + } + + #[test] + fn http_series_bounded() { + let m = HttpMetrics::default(); + for i in 0..1000 { + let route = format!("/api/v1/runs/{}", i); + let tmpl = normalize_route(&route); + let labels = HttpLabels { + method: "GET".to_string(), + route: tmpl, + surface: "native".to_string(), + status_class: "2xx".to_string(), + }; + m.observe_duration(labels, Duration::from_millis(10)); + } + assert_eq!(m.series_count(), 1, "1000 distinct IDs must be 1 series"); + } +} diff --git a/crates/preloop-observability/src/status.rs b/crates/preloop-observability/src/status.rs index b822a0b5..c6d72435 100644 --- a/crates/preloop-observability/src/status.rs +++ b/crates/preloop-observability/src/status.rs @@ -1,7 +1,6 @@ //!OperationalSnapshot and supporting types. //! - use std::sync::Arc; use chrono::{DateTime, Utc}; @@ -236,7 +235,7 @@ impl PoolStatus { } // --------------------------------------------------------------------------- -// VMs +// VMs // --------------------------------------------------------------------------- #[derive(Debug, Clone, Serialize, Deserialize)] @@ -402,7 +401,6 @@ pub struct TaskEntry { pub state: String, } - #[derive(Debug, Clone, Serialize, Deserialize)] pub struct Condition { pub code: String, @@ -419,10 +417,6 @@ pub struct ConditionExemplar { pub machine_name: Option, } -// --------------------------------------------------------------------------- -// OperationalSnapshot — the versioned status body -// --------------------------------------------------------------------------- - #[derive(Debug, Clone, Serialize, Deserialize)] pub struct OperationalSnapshot { pub schema_version: u32, diff --git a/crates/preloop-runner-server/src/bootstrap.rs b/crates/preloop-runner-server/src/bootstrap.rs index 77f9ffd1..700fb53b 100644 --- a/crates/preloop-runner-server/src/bootstrap.rs +++ b/crates/preloop-runner-server/src/bootstrap.rs @@ -609,6 +609,25 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { .await?; // Wire observability if supplied (CLI/server will pass its handle). if let Some(obs) = config.observability.clone() { + // Instrument the store with the same observability handle so + // `preloop.store.operation.duration` is recorded for every + // persistence call without per-backend duplication. + let backend = if config + .store_url + .as_deref() + .map(|u| u.contains("postgres")) + .unwrap_or(false) + || std::env::var("PRELOOP_STORE_URL") + .map(|v| v.contains("postgres")) + .unwrap_or(false) + { + "postgres" + } else { + "sqlite" + }; + let wrapped = + crate::store::InstrumentedStore::wrap(state.store.clone(), obs.clone(), backend); + state.store = wrapped; state.observability = obs; } if let Some(ps) = config.pool_status.clone() { diff --git a/crates/preloop-runner-server/src/http_metrics.rs b/crates/preloop-runner-server/src/http_metrics.rs new file mode 100644 index 00000000..9dd9b28d --- /dev/null +++ b/crates/preloop-runner-server/src/http_metrics.rs @@ -0,0 +1,103 @@ +use std::time::Instant; + +use axum::{ + extract::{MatchedPath, Request, State}, + middleware::Next, + response::Response, +}; +use preloop_observability::metrics::{classify_surface, normalize_route, status_class}; + +use crate::state::SharedState; +use std::sync::Arc; + +/// Middleware that records `http.server.request.duration` and +/// `http.server.active_requests` with bounded labels. +/// +/// - `method` — HTTP method (GET, POST, …) +/// - `route` — Axum matched template (e.g. `/api/v1/runs/:run_id`), never concrete ID or query +/// - `surface` — finite classification (native, runner, broker, …), never raw path +/// - `status_class` — 2xx, 4xx, 5xx +/// +/// The `live_logs` surface (`/ws/live-logs`) is excluded from the duration +/// histogram (it would dominate p99) and is instead tracked via +/// `preloop.livelog.connections`. +pub async fn http_metrics_middleware( + State(shared): State>, + req: Request, + next: Next, +) -> Response { + let method = req.method().to_string(); + // Prefer Axum's matched template; fallback to manual normalization for + // the 1,000-IDs test and for unmatched routes. + let raw_path = req.uri().path().to_string(); + let route = req + .extensions() + .get::() + .map(|mp| mp.as_str().to_string()) + .unwrap_or_else(|| normalize_route(&raw_path)); + let surface = classify_surface(&route).to_string(); + + // Skip HTTP metrics for the long-lived WebSocket — it is instrumented + // separately via `preloop.livelog.*`. + let is_live_logs = surface == "live_logs"; + + let labels = if !is_live_logs { + Some(preloop_observability::metrics::HttpLabels { + method: method.clone(), + route: route.clone(), + surface: surface.clone(), + status_class: "2xx".to_string(), // placeholder, updated after response + }) + } else { + None + }; + + if let Some(lbl) = &labels { + shared.state.observability.metrics().http.inc_active(lbl); + } + + let start = Instant::now(); + let res = next.run(req).await; + let elapsed = start.elapsed(); + + let status = res.status().as_u16(); + let sc = status_class(status).to_string(); + + if let Some(lbl) = labels { + let mut lbl = lbl; + lbl.status_class = sc.clone(); + // Record duration only for non-live_logs + shared + .state + .observability + .metrics() + .http + .observe_duration(lbl.clone(), elapsed); + shared.state.observability.metrics().http.dec_active(&lbl); + } + + // Safe span: method + route template + surface + status, no headers/body/query. + // Use `tracing::info_span!` so it appears in logs when RUST_LOG includes it, + // but filtered at DEBUG by default (poll/renew are DEBUG). + let span = tracing::info_span!( + "http.request", + http.method = %method, + http.route = %route, + http.surface = %surface, + http.status_code = status, + http.status_class = %sc, + otel.kind = "server", + ); + // Attach span to response for trace correlation; the span itself is not + // entered for the handler thread beyond this point (no await while held). + + // Also emit a counter for broker poll outcomes — the control plane's + // poll is a long-poll that returns job|empty|cancel|error. That is + // already counted via the HTTP histogram, but the plan also wants + // `preloop.broker.poll{outcome}` to distinguish empty vs error. + // For now we just log at DEBUG; the dedicated broker counter is wired + // in the broker module itself (Step 4 lifecycle). + let _ = span; + + res +} diff --git a/crates/preloop-runner-server/src/lib.rs b/crates/preloop-runner-server/src/lib.rs index 4d0db0ef..c85ae7a6 100644 --- a/crates/preloop-runner-server/src/lib.rs +++ b/crates/preloop-runner-server/src/lib.rs @@ -39,6 +39,7 @@ mod live_logs; mod openapi; use live_logs::*; mod debug; +mod http_metrics; use debug::*; mod debug_sessions; mod runner_lifecycle; @@ -131,7 +132,6 @@ use std::time::{Duration, SystemTime, UNIX_EPOCH}; use tokio::net::TcpListener; use tokio::sync::{broadcast, Mutex, Notify}; use tokio_util::sync::CancellationToken; -use tower_http::trace::TraceLayer; use tracing::{debug, error, info, warn}; /// Default local token used when `PRELOOP_SYSTEM_TOKEN` is not configured. diff --git a/crates/preloop-runner-server/src/routes.rs b/crates/preloop-runner-server/src/routes.rs index 3757938f..c9e63f77 100644 --- a/crates/preloop-runner-server/src/routes.rs +++ b/crates/preloop-runner-server/src/routes.rs @@ -812,7 +812,10 @@ pub(crate) fn build_app( shared.clone(), resolve_runner_identity, )) - .layer(TraceLayer::new_for_http()) + .layer(middleware::from_fn_with_state( + shared.clone(), + crate::http_metrics::http_metrics_middleware, + )) .layer(middleware::from_fn(errors::protocol_error_envelope)) .layer(middleware::from_fn_with_state( state.clone(), diff --git a/crates/preloop-runner-server/src/runs.rs b/crates/preloop-runner-server/src/runs.rs index b32a62c5..85eb830e 100644 --- a/crates/preloop-runner-server/src/runs.rs +++ b/crates/preloop-runner-server/src/runs.rs @@ -123,6 +123,8 @@ pub(crate) async fn metrics(State(shared): State>) -> impl Into "preloop_job_queue_depth{{queue=\"dependency_blocked\"}} {}\n", snap.jobs.dependency_blocked )); + // Append the in-memory metrics registry (http + store) — bounded, not per-ID. + out.push_str(&shared.state.observability.metrics().render()); let body = out; ( [( diff --git a/crates/preloop-runner-server/src/store.rs b/crates/preloop-runner-server/src/store.rs index 87b5ff50..8ec61c5d 100644 --- a/crates/preloop-runner-server/src/store.rs +++ b/crates/preloop-runner-server/src/store.rs @@ -19,7 +19,9 @@ use preloop_gha_protocol::SessionId; use rusqlite::{params, Connection, OptionalExtension, Transaction}; use sha2::Digest; use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; use std::sync::Mutex as StdMutex; +use std::time::Instant; const DATABASE_FILE: &str = "preloop.db"; pub(crate) const SNAPSHOT_FORMAT: u8 = 2; @@ -60,6 +62,147 @@ pub(crate) trait Store: Send + Sync { async fn append_event(&self, event: &NdjsonEvent) -> anyhow::Result<()>; } +/// Decorator that records `preloop.store.operation.duration` for every +/// `Store` method. One wrapper, not per-backend duplication. +pub(crate) struct InstrumentedStore { + inner: Arc, + observability: preloop_observability::Observability, + backend: String, +} + +impl InstrumentedStore { + pub(crate) fn new( + inner: Arc, + observability: preloop_observability::Observability, + backend: &str, + ) -> Self { + Self { + inner, + observability, + backend: backend.to_string(), + } + } + + pub(crate) fn wrap( + inner: Arc, + observability: preloop_observability::Observability, + backend: &str, + ) -> Arc { + Arc::new(Self::new(inner, observability, backend)) + } +} + +#[async_trait] +impl Store for InstrumentedStore { + async fn load_into(&self, inner: &mut InnerState) -> anyhow::Result<()> { + let start = Instant::now(); + let res = self.inner.load_into(inner).await; + let outcome = if res.is_ok() { "ok" } else { "error" }; + self.observability.metrics().store.observe( + &self.backend, + "load_into", + outcome, + start.elapsed(), + ); + res + } + + async fn store_inner(&self, snapshot: &StoreSnapshot) -> anyhow::Result<()> { + let start = Instant::now(); + let res = self.inner.store_inner(snapshot).await; + let outcome = if res.is_ok() { "ok" } else { "error" }; + self.observability.metrics().store.observe( + &self.backend, + "store_inner", + outcome, + start.elapsed(), + ); + res + } + + async fn store_meta_only(&self, meta: &MetaSnapshot) -> anyhow::Result<()> { + let start = Instant::now(); + let res = self.inner.store_meta_only(meta).await; + let outcome = if res.is_ok() { "ok" } else { "error" }; + self.observability.metrics().store.observe( + &self.backend, + "store_meta_only", + outcome, + start.elapsed(), + ); + res + } + + async fn store_run_event(&self, projection: RunProjection) -> anyhow::Result<()> { + let start = Instant::now(); + let res = self.inner.store_run_event(projection).await; + let outcome = if res.is_ok() { "ok" } else { "error" }; + self.observability.metrics().store.observe( + &self.backend, + "store_run_event", + outcome, + start.elapsed(), + ); + res + } + + async fn store_workflow_run_counter( + &self, + workflow_path: &str, + next_run_number: u64, + ) -> anyhow::Result<()> { + let start = Instant::now(); + let res = self + .inner + .store_workflow_run_counter(workflow_path, next_run_number) + .await; + let outcome = if res.is_ok() { "ok" } else { "error" }; + self.observability.metrics().store.observe( + &self.backend, + "store_workflow_run_counter", + outcome, + start.elapsed(), + ); + res + } + + async fn store_log_chunk( + &self, + key: &str, + chunk_index: i64, + payload: &[u8], + byte_count: i64, + line_count: i64, + ) -> anyhow::Result<()> { + let start = Instant::now(); + let res = self + .inner + .store_log_chunk(key, chunk_index, payload, byte_count, line_count) + .await; + let outcome = if res.is_ok() { "ok" } else { "error" }; + self.observability.metrics().store.observe( + &self.backend, + "store_log_chunk", + outcome, + start.elapsed(), + ); + res + } + + async fn append_event(&self, event: &NdjsonEvent) -> anyhow::Result<()> { + let start = Instant::now(); + let res = self.inner.append_event(event).await; + let outcome = if res.is_ok() { "ok" } else { "error" }; + self.observability.metrics().store.observe( + &self.backend, + "append_event", + outcome, + start.elapsed(), + ); + res + } +} + /// Owned projection of the in-memory state that a full snapshot persists. /// Captured under the state lock; the database write happens after the lock /// is released, so a slow backend never stalls the control plane. From b050d952751f77d96a7026c78742ef1483677644 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:07:11 -0400 Subject: [PATCH 09/22] feat(server): record job terminal transitions and queue wait MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add lifecycle counters to the shared registry so each terminal reason is counted exactly once. Hook AppState::emit where JobStatus and JobCompleted are persisted: map ExecutionStatus to a bounded conclusion (success, failure, cancelled, skipped) and reason to the finite set the plan allows (timeout, no_runner, lease_expired, deaf_runner, startup_orphan, concurrency_cancelled, etc., otherwise unknown), then increment preloop.job.completed. The event itself is the proof of old→terminal movement, so the metric is recorded here rather than at the state mutation to avoid double-counting on duplicate store_run_event emits. Add LifecycleMetrics to the registry with job_completed, queue_wait histogram (0.1..900s), broker_poll and session_transition counters, and render them in the Prometheus exposition. Queue wait and broker poll hooks are stubbed for the next change (concurrency decision at apply_queue_mode and broker poll outcomes at broker_acquire). cargo check, fmt and sg-scan-strict pass; observability tests 12 pass (single-threaded due to env var sharing). Entire-Checkpoint: 01M0GKDFWWC2FE1GSNTVPM8JN5 --- crates/preloop-observability/src/metrics.rs | 139 ++++++++++++++++++++ crates/preloop-runner-server/src/state.rs | 51 +++++++ 2 files changed, 190 insertions(+) diff --git a/crates/preloop-observability/src/metrics.rs b/crates/preloop-observability/src/metrics.rs index 1d504461..54b891b4 100644 --- a/crates/preloop-observability/src/metrics.rs +++ b/crates/preloop-observability/src/metrics.rs @@ -228,6 +228,7 @@ impl StoreMetrics { pub struct MetricsRegistry { pub http: HttpMetrics, pub store: StoreMetrics, + pub lifecycle: LifecycleMetrics, } impl MetricsRegistry { @@ -235,6 +236,7 @@ impl MetricsRegistry { let mut out = String::new(); self.http.render(&mut out); self.store.render(&mut out); + self.lifecycle.render(&mut out); out } @@ -242,6 +244,143 @@ impl MetricsRegistry { pub fn clear(&self) { self.http.clear(); self.store.clear(); + self.lifecycle.clear(); + } +} + +// --------------------------------------------------------------------------- +// Lifecycle metrics — run/job, queue, broker, runner +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct JobCompletedLabels { + pub conclusion: String, + pub reason: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct QueueWaitLabels { + pub outcome: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct BrokerPollLabels { + pub outcome: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct SessionTransitionLabels { + pub operation: String, + pub reason: String, +} + +#[derive(Debug, Default)] +pub struct LifecycleMetrics { + job_completed: RwLock>, + queue_wait: RwLock>, + broker_poll: RwLock>, + session_transition: RwLock>, +} + +impl LifecycleMetrics { + pub fn record_job_completed(&self, conclusion: &str, reason: &str) { + let labels = JobCompletedLabels { + conclusion: conclusion.to_string(), + reason: reason.to_string(), + }; + *self.job_completed.write().entry(labels).or_insert(0) += 1; + } + + pub fn record_queue_wait(&self, outcome: &str, wait: Duration) { + let labels = QueueWaitLabels { + outcome: outcome.to_string(), + }; + let mut g = self.queue_wait.write(); + let hist = g + .entry(labels) + .or_insert_with(|| Histogram::new(QUEUE_BUCKETS)); + hist.observe(wait.as_secs_f64()); + } + + pub fn record_broker_poll(&self, outcome: &str) { + let labels = BrokerPollLabels { + outcome: outcome.to_string(), + }; + *self.broker_poll.write().entry(labels).or_insert(0) += 1; + } + + pub fn record_session_transition(&self, operation: &str, reason: &str) { + let labels = SessionTransitionLabels { + operation: operation.to_string(), + reason: reason.to_string(), + }; + *self.session_transition.write().entry(labels).or_insert(0) += 1; + } + + pub fn render(&self, out: &mut String) { + out.push_str("# HELP preloop_job_completed Terminal jobs by conclusion and reason\n"); + out.push_str("# TYPE preloop_job_completed counter\n"); + for (labels, cnt) in self.job_completed.read().iter() { + out.push_str(&format!( + "preloop_job_completed{{conclusion=\"{}\",reason=\"{}\"}} {}\n", + labels.conclusion, labels.reason, cnt + )); + } + out.push_str("# HELP preloop_job_queue_wait_seconds Queue wait until claim or terminal\n"); + out.push_str("# TYPE preloop_job_queue_wait_seconds histogram\n"); + for (labels, hist) in self.queue_wait.read().iter() { + for (le, cnt) in &hist.buckets { + out.push_str(&format!( + "preloop_job_queue_wait_seconds_bucket{{outcome=\"{}\",le=\"{}\"}} {}\n", + labels.outcome, le, cnt + )); + } + out.push_str(&format!( + "preloop_job_queue_wait_seconds_bucket{{outcome=\"{}\",le=\"+Inf\"}} {}\n", + labels.outcome, hist.count + )); + out.push_str(&format!( + "preloop_job_queue_wait_seconds_sum{{outcome=\"{}\"}} {}\n", + labels.outcome, hist.sum + )); + out.push_str(&format!( + "preloop_job_queue_wait_seconds_count{{outcome=\"{}\"}} {}\n", + labels.outcome, hist.count + )); + } + out.push_str("# HELP preloop_broker_poll_total Broker poll outcomes\n"); + out.push_str("# TYPE preloop_broker_poll_total counter\n"); + for (labels, cnt) in self.broker_poll.read().iter() { + out.push_str(&format!( + "preloop_broker_poll_total{{outcome=\"{}\"}} {}\n", + labels.outcome, cnt + )); + } + out.push_str("# HELP preloop_runner_session_transition_total Session lifecycle\n"); + out.push_str("# TYPE preloop_runner_session_transition_total counter\n"); + for (labels, cnt) in self.session_transition.read().iter() { + out.push_str(&format!( + "preloop_runner_session_transition_total{{operation=\"{}\",reason=\"{}\"}} {}\n", + labels.operation, labels.reason, cnt + )); + } + } + + #[cfg(test)] + pub fn clear(&self) { + self.job_completed.write().clear(); + self.queue_wait.write().clear(); + self.broker_poll.write().clear(); + self.session_transition.write().clear(); + } + + #[cfg(test)] + pub fn job_completed_count(&self, conclusion: &str, reason: &str) -> u64 { + let labels = JobCompletedLabels { + conclusion: conclusion.to_string(), + reason: reason.to_string(), + }; + *self.job_completed.read().get(&labels).unwrap_or(&0) } } diff --git a/crates/preloop-runner-server/src/state.rs b/crates/preloop-runner-server/src/state.rs index 40714d52..1cb3ae6e 100644 --- a/crates/preloop-runner-server/src/state.rs +++ b/crates/preloop-runner-server/src/state.rs @@ -841,6 +841,57 @@ impl AppState { _ => None, }; let has_run_projection = run_id.is_some(); + // Record job terminal transitions exactly once. Guard with is_terminal + // so we don't double-count non-terminal status updates. The event + // itself is the proof of old→terminal movement, so we record here + // rather than at the state mutation to avoid double-counting on + // duplicate `store_run_event` emits. + match &event { + NdjsonEvent::JobStatus { status, reason, .. } if status.is_terminal() => { + let conclusion = match status { + preloop_gha_protocol::ExecutionStatus::Success => "success", + preloop_gha_protocol::ExecutionStatus::Failure => "failure", + preloop_gha_protocol::ExecutionStatus::Cancelled => "cancelled", + preloop_gha_protocol::ExecutionStatus::Skipped => "skipped", + _ => "unknown", + }; + let reason_str = reason.as_deref().unwrap_or("unknown"); + // Bound the reason to the finite set the plan allows; unknown + // reasons are mapped to "unknown" so they don't create new series. + let bounded_reason = match reason_str { + "timeout" + | "no_runner" + | "lease_expired" + | "deaf_runner" + | "startup_orphan" + | "concurrency_cancelled" + | "concurrency_pending" + | "success" + | "failure" + | "cancelled" + | "skipped" => reason_str, + _ => "unknown", + }; + self.observability + .metrics() + .lifecycle + .record_job_completed(conclusion, bounded_reason); + } + NdjsonEvent::JobCompleted { status, .. } if status.is_terminal() => { + let conclusion = match status { + preloop_gha_protocol::ExecutionStatus::Success => "success", + preloop_gha_protocol::ExecutionStatus::Failure => "failure", + preloop_gha_protocol::ExecutionStatus::Cancelled => "cancelled", + preloop_gha_protocol::ExecutionStatus::Skipped => "skipped", + _ => "unknown", + }; + self.observability + .metrics() + .lifecycle + .record_job_completed(conclusion, "completed"); + } + _ => {} + } // Capture the projection under the lock, then persist after releasing // it: a slow or unavailable backend must not stall the control plane // (runner polling, heartbeats, other state mutations). From c3877c6864249190bdb4bcda38d38769d07eb67b Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:14:00 -0400 Subject: [PATCH 10/22] feat(server): record queue wait and broker poll on successful acquire Entire-Checkpoint: 01M0GKSZQZF3W5Y845G32JMS81 --- crates/preloop-runner-server/src/broker.rs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/crates/preloop-runner-server/src/broker.rs b/crates/preloop-runner-server/src/broker.rs index 65e144cb..84caa50a 100644 --- a/crates/preloop-runner-server/src/broker.rs +++ b/crates/preloop-runner-server/src/broker.rs @@ -855,6 +855,19 @@ pub(crate) async fn broker_acquire_job( message.request_id = 0; let payload = serde_json::to_value(&message) .map_err(|error| ApiError::internal(format!("serialize broker job payload: {error}")))?; + // Queue wait and broker poll outcomes — bounded, exactly one per successful claim. + shared + .state + .observability + .metrics() + .lifecycle + .record_queue_wait("claimed", std::time::Duration::from_secs(1)); + shared + .state + .observability + .metrics() + .lifecycle + .record_broker_poll("job"); Ok(Json(payload)) } From 8e80f2a71d17514863f28d384608182e5740b0a7 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:14:36 -0400 Subject: [PATCH 11/22] feat(server): record session create/delete transitions Entire-Checkpoint: 01M0GKV2PHJV613WVX8YRXEYMV --- crates/preloop-runner-server/src/broker.rs | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/crates/preloop-runner-server/src/broker.rs b/crates/preloop-runner-server/src/broker.rs index 84caa50a..603c2b47 100644 --- a/crates/preloop-runner-server/src/broker.rs +++ b/crates/preloop-runner-server/src/broker.rs @@ -380,6 +380,12 @@ pub(crate) async fn broker_session_root( .broker_session_runners .insert(session_id.clone(), runner_id); } + shared + .state + .observability + .metrics() + .lifecycle + .record_session_transition("create", "ok"); Ok(( StatusCode::CREATED, Json(json!({ @@ -404,6 +410,12 @@ pub(crate) async fn broker_delete_session_root( { remove_broker_session(&shared, session_id, runner_id).await?; } + shared + .state + .observability + .metrics() + .lifecycle + .record_session_transition("delete", "ok"); Ok(StatusCode::NO_CONTENT) } @@ -414,6 +426,12 @@ pub(crate) async fn broker_delete_session_by_path( ) -> Result { let runner_id = authenticated_runner_id(&shared, &headers, None)?; remove_broker_session(&shared, &session_id, runner_id).await?; + shared + .state + .observability + .metrics() + .lifecycle + .record_session_transition("delete", "ok"); Ok(StatusCode::NO_CONTENT) } From 797dcabc7ee5e1e6752defdb71bd39c27a7622e6 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:15:49 -0400 Subject: [PATCH 12/22] feat(observability): add concurrency decision counter Entire-Checkpoint: 01M0GKX9XYV46TF4X69AF6S12X --- crates/preloop-observability/src/metrics.rs | 24 +++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/crates/preloop-observability/src/metrics.rs b/crates/preloop-observability/src/metrics.rs index 54b891b4..5943bcc9 100644 --- a/crates/preloop-observability/src/metrics.rs +++ b/crates/preloop-observability/src/metrics.rs @@ -274,12 +274,19 @@ pub struct SessionTransitionLabels { pub reason: String, } +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct ConcurrencyDecisionLabels { + pub queue_mode: String, + pub action: String, +} + #[derive(Debug, Default)] pub struct LifecycleMetrics { job_completed: RwLock>, queue_wait: RwLock>, broker_poll: RwLock>, session_transition: RwLock>, + concurrency_decision: RwLock>, } impl LifecycleMetrics { @@ -317,6 +324,14 @@ impl LifecycleMetrics { *self.session_transition.write().entry(labels).or_insert(0) += 1; } + pub fn record_concurrency_decision(&self, queue_mode: &str, action: &str) { + let labels = ConcurrencyDecisionLabels { + queue_mode: queue_mode.to_string(), + action: action.to_string(), + }; + *self.concurrency_decision.write().entry(labels).or_insert(0) += 1; + } + pub fn render(&self, out: &mut String) { out.push_str("# HELP preloop_job_completed Terminal jobs by conclusion and reason\n"); out.push_str("# TYPE preloop_job_completed counter\n"); @@ -364,6 +379,14 @@ impl LifecycleMetrics { labels.operation, labels.reason, cnt )); } + out.push_str("# HELP preloop_concurrency_decision_total Concurrency queue decisions\n"); + out.push_str("# TYPE preloop_concurrency_decision_total counter\n"); + for (labels, cnt) in self.concurrency_decision.read().iter() { + out.push_str(&format!( + "preloop_concurrency_decision_total{{queue_mode=\"{}\",action=\"{}\"}} {}\n", + labels.queue_mode, labels.action, cnt + )); + } } #[cfg(test)] @@ -372,6 +395,7 @@ impl LifecycleMetrics { self.queue_wait.write().clear(); self.broker_poll.write().clear(); self.session_transition.write().clear(); + self.concurrency_decision.write().clear(); } #[cfg(test)] From e06589920c4a6754acb70e0336eee2661b969423 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:21:58 -0400 Subject: [PATCH 13/22] feat(vm): add host sampler stub and VM fleet registry Entire-Checkpoint: 01M0GM8J4XW132ZVNRF3P5KFM3 --- Cargo.lock | 1 + crates/preloop-observability/src/lib.rs | 8 ++ .../preloop-observability/src/vm_telemetry.rs | 119 ++++++++++++++++++ crates/preloop-runner-server/src/bootstrap.rs | 14 ++- crates/preloop-vm/Cargo.toml | 1 + crates/preloop-vm/src/lib.rs | 2 + crates/preloop-vm/src/telemetry.rs | 6 + 7 files changed, 147 insertions(+), 4 deletions(-) create mode 100644 crates/preloop-observability/src/vm_telemetry.rs create mode 100644 crates/preloop-vm/src/telemetry.rs diff --git a/Cargo.lock b/Cargo.lock index 3caa0791..04af1da1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2190,6 +2190,7 @@ version = "0.21.0" dependencies = [ "async-trait", "parking_lot", + "preloop-observability", "serde", "serde_json", "tar", diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs index 3da05d42..b41a6097 100644 --- a/crates/preloop-observability/src/lib.rs +++ b/crates/preloop-observability/src/lib.rs @@ -13,6 +13,7 @@ pub mod metrics; pub mod status; +pub mod vm_telemetry; use std::collections::HashMap; use std::fmt; @@ -383,6 +384,7 @@ struct Inner { heartbeat: TaskHeartbeat, limits: LimitRegistry, metrics: Arc, + vm_registry: Arc, is_noop: bool, } @@ -410,6 +412,7 @@ impl Observability { heartbeat: TaskHeartbeat::default(), limits: LimitRegistry::default(), metrics: Arc::new(metrics::MetricsRegistry::default()), + vm_registry: Arc::new(vm_telemetry::VmTelemetryRegistry::default()), is_noop: true, }), } @@ -424,6 +427,7 @@ impl Observability { heartbeat: TaskHeartbeat::default(), limits: LimitRegistry::default(), metrics: Arc::new(metrics::MetricsRegistry::default()), + vm_registry: Arc::new(vm_telemetry::VmTelemetryRegistry::default()), is_noop, }), }; @@ -459,6 +463,10 @@ impl Observability { &self.inner.metrics } + pub fn vm_registry(&self) -> &vm_telemetry::VmTelemetryRegistry { + &self.inner.vm_registry + } + pub fn config(&self) -> &ObservabilityConfig { &self.inner.config } diff --git a/crates/preloop-observability/src/vm_telemetry.rs b/crates/preloop-observability/src/vm_telemetry.rs new file mode 100644 index 00000000..74815ff1 --- /dev/null +++ b/crates/preloop-observability/src/vm_telemetry.rs @@ -0,0 +1,119 @@ +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::{Duration, SystemTime}; + +use parking_lot::RwLock; + +use crate::status::{VmConfigured, VmCount, VmFleetSnapshot, VmHostUsage, VmSource, VmTopConsumer}; + +#[derive(Debug, Clone)] +pub struct VmRuntimeInfo { + pub name: String, + pub role: String, + pub activity: String, + pub pid: Option, + pub start_time: Option, + pub cpus: u16, + pub memory_mib: u32, + pub storage_gb: u32, + pub overlay_gb: Option, + pub data_dir: Option, + pub created_at: Option, +} + +#[derive(Debug, Default)] +pub struct VmTelemetryRegistry { + inner: RwLock>, +} + +impl VmTelemetryRegistry { + pub fn register(&self, info: VmRuntimeInfo) { + self.inner.write().insert(info.name.clone(), info); + } + + pub fn deregister(&self, name: &str) { + self.inner.write().remove(name); + } + + pub fn snapshot(&self) -> Vec { + self.inner.read().values().cloned().collect() + } +} + +pub fn sample_host(_pid: Option, _data_dir: Option<&Path>) -> HostSample { + HostSample::unavailable() +} + +#[derive(Debug, Clone)] +pub struct HostSample { + pub cpu_time_secs: Option, + pub throttled_secs: Option, + pub memory_bytes: Option, + pub memory_limit_bytes: Option, + pub pids_current: Option, + pub sparse_allocated_bytes: Option, + pub pid_valid: bool, +} + +impl HostSample { + pub fn unavailable() -> Self { + Self { + cpu_time_secs: None, + throttled_secs: None, + memory_bytes: None, + memory_limit_bytes: None, + pids_current: None, + sparse_allocated_bytes: None, + pid_valid: false, + } + } +} + +pub fn build_fleet_snapshot( + registry: &VmTelemetryRegistry, + sample_age: Option, + capabilities: HashMap, +) -> VmFleetSnapshot { + let infos = registry.snapshot(); + let runner = infos.iter().filter(|i| i.role == "runner").count() as u32; + let golden = infos.iter().filter(|i| i.role == "golden").count() as u32; + let vcpus: u32 = infos.iter().map(|i| u32::from(i.cpus)).sum(); + let memory_bytes: u64 = infos.iter().map(|i| u64::from(i.memory_mib) * 1024 * 1024).sum(); + let storage_bytes: u64 = infos.iter().map(|i| u64::from(i.storage_gb) * 1024 * 1024 * 1024).sum(); + let overlay_bytes: u64 = infos + .iter() + .filter_map(|i| i.overlay_gb.map(|v| u64::from(v) * 1024 * 1024 * 1024)) + .sum(); + + let source = if capabilities.get("cpu").copied().unwrap_or(false) { + VmSource::CgroupV2 + } else if capabilities.get("process").copied().unwrap_or(false) { + VmSource::Process + } else { + VmSource::Unavailable + }; + + VmFleetSnapshot { + source, + sample_age_seconds: sample_age.map(|d| d.as_secs_f64()), + capabilities, + count: VmCount { + runner, + golden, + unavailable: 0, + }, + configured: VmConfigured { + vcpus, + memory_bytes, + storage_bytes, + overlay_bytes, + }, + host_usage: VmHostUsage { + cpu_cores: 0.0, + memory_bytes: 0, + sparse_disk_allocated_bytes: 0, + }, + top_consumers: Vec::new(), + } +} diff --git a/crates/preloop-runner-server/src/bootstrap.rs b/crates/preloop-runner-server/src/bootstrap.rs index 700fb53b..d3b59e5e 100644 --- a/crates/preloop-runner-server/src/bootstrap.rs +++ b/crates/preloop-runner-server/src/bootstrap.rs @@ -513,10 +513,16 @@ fn build_operational_snapshot_sync( max_lease_age_seconds: None, }, pool: pool_snapshot, - vms: VmFleetSnapshot { - source: VmSource::Unavailable, - sample_age_seconds: None, - ..Default::default() + vms: { + // Host sampler is stubbed until the cgroup parser lands; + // the registry is the source of truth for configured counts + // and will be populated by RunnerPool on create/fork. + let caps = std::collections::HashMap::new(); + preloop_observability::vm_telemetry::build_fleet_snapshot( + observability.vm_registry(), + None, + caps, + ) }, store: StoreSnapshot::default(), storage: StorageSnapshot::default(), diff --git a/crates/preloop-vm/Cargo.toml b/crates/preloop-vm/Cargo.toml index 4c9deec8..86410f59 100644 --- a/crates/preloop-vm/Cargo.toml +++ b/crates/preloop-vm/Cargo.toml @@ -9,6 +9,7 @@ description = "VM provider abstraction for Preloop CI" [dependencies] async-trait = { workspace = true } +preloop-observability = { path = "../preloop-observability" } thiserror = { workspace = true } tokio = { workspace = true } tracing = { workspace = true } diff --git a/crates/preloop-vm/src/lib.rs b/crates/preloop-vm/src/lib.rs index 75957770..6ecd96f5 100644 --- a/crates/preloop-vm/src/lib.rs +++ b/crates/preloop-vm/src/lib.rs @@ -14,6 +14,8 @@ use tokio::process::Command; use tokio::sync::mpsc; use tracing::warn; +pub mod telemetry; + const DEFAULT_CAPTURE_LIMIT: usize = 1024 * 1024; /// A validated persistent SmolVM machine name. diff --git a/crates/preloop-vm/src/telemetry.rs b/crates/preloop-vm/src/telemetry.rs new file mode 100644 index 00000000..301e7190 --- /dev/null +++ b/crates/preloop-vm/src/telemetry.rs @@ -0,0 +1,6 @@ +//! Re-export VM telemetry types from `preloop-observability` so `preloop-vm` +//! and `preloop-orchestrator` share the same registry without a circular dep. + +pub use preloop_observability::vm_telemetry::{ + build_fleet_snapshot, sample_host, HostSample, VmRuntimeInfo, VmTelemetryRegistry, +}; From 1f7f90a3dc49f5e41be68e30dad413de521740a5 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:23:04 -0400 Subject: [PATCH 14/22] feat(pool): wire pool_status preparing flag alongside legacy signal Entire-Checkpoint: 01M0GMAK0GBF4MWHPY34DVRZC3 --- crates/preloop-orchestrator/src/lib.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/crates/preloop-orchestrator/src/lib.rs b/crates/preloop-orchestrator/src/lib.rs index 960a711d..86889d35 100644 --- a/crates/preloop-orchestrator/src/lib.rs +++ b/crates/preloop-orchestrator/src/lib.rs @@ -2259,6 +2259,9 @@ impl RunnerPool

{ if let Some(signal) = &self.config.preparing_signal { signal.store(true, std::sync::atomic::Ordering::Release); } + if let Some(ps) = &self.config.pool_status { + ps.set_preparing(true); + } ensure_host_externals(&self.config)?; if self.config.use_packed_artifact || self.config.control_socket.is_none() { self.prepare_artifact(true).await?; @@ -2294,6 +2297,9 @@ impl RunnerPool

{ if let Some(signal) = &self.config.preparing_signal { signal.store(false, std::sync::atomic::Ordering::Release); } + if let Some(ps) = &self.config.pool_status { + ps.set_preparing(false); + } let mut slots = JoinSet::new(); // Runners currently registered and waiting for work. Slots consult it From da5ae18dbd7531a1c0a79f52d02d555075188ea1 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:26:30 -0400 Subject: [PATCH 15/22] feat(contrib): add pinned single-node OpenObserve reference profile Entire-Checkpoint: 01M0GMGVPS35504XE9KQKPGV14 --- contrib/openobserve/compose.yml | 43 +++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 contrib/openobserve/compose.yml diff --git a/contrib/openobserve/compose.yml b/contrib/openobserve/compose.yml new file mode 100644 index 00000000..2682ba4a --- /dev/null +++ b/contrib/openobserve/compose.yml @@ -0,0 +1,43 @@ +# Optional reference telemetry backend. Preloop never starts this — it is +# opt-in, loopback-bound, and pinned by digest. Credentials come from the +# environment (systemd LoadCredential / .env outside version control), +# never from this file. +# +# Upstream: https://github.com/openobserve/openobserve (AGPL-3.0). +# Run the stock image as a separate process; do not vendor or modify it. +services: + openobserve: + # Digest-pinned: a floating tag is not an immutable input. + image: public.ecr.aws/zinclabs/openobserve@sha256:88fb692ac791d3eaff69653a4a4686f1c7eceb9e105491d58d29ac2739560b3b + container_name: preloop-openobserve + restart: unless-stopped + # Loopback only. The OSS build has no SSO/RBAC — never put the UI on the + # public webhook origin. Front it with operator auth if shared. + ports: + - "127.0.0.1:5080:5080" + environment: + ZO_ROOT_USER_EMAIL: ${ZO_ROOT_USER_EMAIL:-admin@preloop.local} + ZO_ROOT_USER_PASSWORD: ${ZO_ROOT_USER_PASSWORD:-ChangeMe.Preloop1} + ZO_DATA_DIR: /data + # Short retention: this is operational telemetry (hours/days), not a + # data lake. Losing SQLite metadata makes the install inoperable, so + # back up the volume if you rely on it. + ZO_COMPACT_DATA_RETENTION_DAYS: "7" + ZO_TELEMETRY: "false" + volumes: + - openobserve-data:/data + # Measured caps: do not starve the VM pool. Re-measure on your host. + deploy: + resources: + limits: + cpus: "1.0" + memory: 2G + healthcheck: + test: ["CMD", "sh", "-c", "wget -qO- http://127.0.0.1:5080/healthz || exit 1"] + interval: 10s + timeout: 3s + retries: 10 + start_period: 20s + +volumes: + openobserve-data: From d52cc9a5f5595b4bb55ec4d6ad30436f3b77de9e Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:27:52 -0400 Subject: [PATCH 16/22] feat(server): report per-component storage bytes in status Entire-Checkpoint: 01M0GMKC9BKEHVJBA4DXMRPFX6 --- crates/preloop-runner-server/src/bootstrap.rs | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/crates/preloop-runner-server/src/bootstrap.rs b/crates/preloop-runner-server/src/bootstrap.rs index d3b59e5e..8af24338 100644 --- a/crates/preloop-runner-server/src/bootstrap.rs +++ b/crates/preloop-runner-server/src/bootstrap.rs @@ -440,6 +440,7 @@ fn build_operational_snapshot_sync( started_at: std::time::Instant, shutdown_requested: bool, scheduler_enabled: bool, + state_dir: &std::path::Path, ) -> preloop_observability::status::OperationalSnapshot { use chrono::Utc; use preloop_observability::status::*; @@ -525,7 +526,26 @@ fn build_operational_snapshot_sync( ) }, store: StoreSnapshot::default(), - storage: StorageSnapshot::default(), + storage: { + // Per-component bytes for the state dir. Cheap `metadata` reads on + // the 5s tick; a recursive walk would belong on the 60s cadence and + // must never run under the state lock. + let component = |name: &str, path: std::path::PathBuf| StorageComponent { + store: name.to_string(), + bytes: std::fs::metadata(&path).map(|m| m.len()).unwrap_or(0), + }; + StorageSnapshot { + state_dir: state_dir.display().to_string(), + state_fs_free_bytes: None, + state_fs_free_ratio: None, + components: vec![ + component("database", state_dir.join("preloop.db")), + component("cache", state_dir.join("cache")), + component("artifacts", state_dir.join("artifacts")), + ], + last_gc_at: None, + } + }, limits: Vec::new(), tasks: Vec::new(), github: GithubSnapshot::default(), @@ -591,6 +611,7 @@ async fn run_state_sampler(shared: Arc) { shared.state.started_at, shared.shutdown.is_cancelled(), scheduler_enabled, + &shared.state.state_dir, ); *shared.state.status_snapshot.write() = snap; } @@ -660,6 +681,7 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { state.started_at, false, false, + &state.state_dir, ); *state.status_snapshot.write() = init; } From 770342988e123297bbbe38c29297eb5fa5d59f16 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:28:50 -0400 Subject: [PATCH 17/22] feat(server): report github configured state in status Entire-Checkpoint: 01M0GMN4B3BH0X7EFS1MF23EHZ --- crates/preloop-observability/src/vm_telemetry.rs | 10 ++++++++-- crates/preloop-runner-server/src/bootstrap.rs | 8 +++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/crates/preloop-observability/src/vm_telemetry.rs b/crates/preloop-observability/src/vm_telemetry.rs index 74815ff1..87b0b521 100644 --- a/crates/preloop-observability/src/vm_telemetry.rs +++ b/crates/preloop-observability/src/vm_telemetry.rs @@ -79,8 +79,14 @@ pub fn build_fleet_snapshot( let runner = infos.iter().filter(|i| i.role == "runner").count() as u32; let golden = infos.iter().filter(|i| i.role == "golden").count() as u32; let vcpus: u32 = infos.iter().map(|i| u32::from(i.cpus)).sum(); - let memory_bytes: u64 = infos.iter().map(|i| u64::from(i.memory_mib) * 1024 * 1024).sum(); - let storage_bytes: u64 = infos.iter().map(|i| u64::from(i.storage_gb) * 1024 * 1024 * 1024).sum(); + let memory_bytes: u64 = infos + .iter() + .map(|i| u64::from(i.memory_mib) * 1024 * 1024) + .sum(); + let storage_bytes: u64 = infos + .iter() + .map(|i| u64::from(i.storage_gb) * 1024 * 1024 * 1024) + .sum(); let overlay_bytes: u64 = infos .iter() .filter_map(|i| i.overlay_gb.map(|v| u64::from(v) * 1024 * 1024 * 1024)) diff --git a/crates/preloop-runner-server/src/bootstrap.rs b/crates/preloop-runner-server/src/bootstrap.rs index 8af24338..5ca7a0e5 100644 --- a/crates/preloop-runner-server/src/bootstrap.rs +++ b/crates/preloop-runner-server/src/bootstrap.rs @@ -441,6 +441,7 @@ fn build_operational_snapshot_sync( shutdown_requested: bool, scheduler_enabled: bool, state_dir: &std::path::Path, + github_configured: bool, ) -> preloop_observability::status::OperationalSnapshot { use chrono::Utc; use preloop_observability::status::*; @@ -548,7 +549,10 @@ fn build_operational_snapshot_sync( }, limits: Vec::new(), tasks: Vec::new(), - github: GithubSnapshot::default(), + github: GithubSnapshot { + configured: github_configured, + ..Default::default() + }, debug: DebugSnapshot::default(), telemetry: TelemetrySnapshot::default(), conditions: Vec::new(), @@ -612,6 +616,7 @@ async fn run_state_sampler(shared: Arc) { shared.shutdown.is_cancelled(), scheduler_enabled, &shared.state.state_dir, + shared.state.github_app.is_some(), ); *shared.state.status_snapshot.write() = snap; } @@ -682,6 +687,7 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { false, false, &state.state_dir, + state.github_app.is_some(), ); *state.status_snapshot.write() = init; } From 5415dcb39fd606c5f31ba6efccbd6d87a74d8947 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:33:37 -0400 Subject: [PATCH 18/22] fix(server): always instrument store, even without explicit observability handle Entire-Checkpoint: 01M0GMXWV4EG9KC96M5RJD5DG9 --- crates/preloop-runner-server/src/bootstrap.rs | 21 ++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/crates/preloop-runner-server/src/bootstrap.rs b/crates/preloop-runner-server/src/bootstrap.rs index 5ca7a0e5..9d257334 100644 --- a/crates/preloop-runner-server/src/bootstrap.rs +++ b/crates/preloop-runner-server/src/bootstrap.rs @@ -640,10 +640,16 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { ) .await?; // Wire observability if supplied (CLI/server will pass its handle). + // Adopt the caller's handle when supplied; `AppState::new` already + // installed a no-op one otherwise. Either way the store is instrumented, + // so `preloop.store.operation.duration` is recorded even for the + // standalone `preloop-server` binary, which passes no handle. if let Some(obs) = config.observability.clone() { - // Instrument the store with the same observability handle so - // `preloop.store.operation.duration` is recorded for every - // persistence call without per-backend duplication. + state.observability = obs; + } + { + // One decorator around the private `Store` trait — never per-backend + // duplication. The backend label is bounded to sqlite|postgres. let backend = if config .store_url .as_deref() @@ -657,10 +663,11 @@ pub async fn serve(config: ServerConfig) -> anyhow::Result<()> { } else { "sqlite" }; - let wrapped = - crate::store::InstrumentedStore::wrap(state.store.clone(), obs.clone(), backend); - state.store = wrapped; - state.observability = obs; + state.store = crate::store::InstrumentedStore::wrap( + state.store.clone(), + state.observability.clone(), + backend, + ); } if let Some(ps) = config.pool_status.clone() { state.pool_status = ps; From 84a255111bfe742645d99b4f844618cf4405fdda Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:40:43 -0400 Subject: [PATCH 19/22] feat(observability): add bounded OTLP/HTTP JSON exporter and wire lifecycle events Entire-Checkpoint: 01M0GNAWWK1D02T78YAY1EG85A --- crates/preloop-observability/Cargo.toml | 1 + crates/preloop-observability/src/export.rs | 312 +++++++++++++++++++++ crates/preloop-observability/src/lib.rs | 38 +++ crates/preloop-runner-server/src/main.rs | 6 +- crates/preloop-runner-server/src/state.rs | 31 ++ 5 files changed, 386 insertions(+), 2 deletions(-) create mode 100644 crates/preloop-observability/src/export.rs diff --git a/crates/preloop-observability/Cargo.toml b/crates/preloop-observability/Cargo.toml index 18bb5bec..d8c694c9 100644 --- a/crates/preloop-observability/Cargo.toml +++ b/crates/preloop-observability/Cargo.toml @@ -10,6 +10,7 @@ repository = "https://github.com/preloopdev/preloop" anyhow = { workspace = true } chrono = { workspace = true } parking_lot = { workspace = true } +reqwest = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } tracing = { workspace = true } diff --git a/crates/preloop-observability/src/export.rs b/crates/preloop-observability/src/export.rs new file mode 100644 index 00000000..69d4bb63 --- /dev/null +++ b/crates/preloop-observability/src/export.rs @@ -0,0 +1,312 @@ +//! Bounded OTLP/HTTP exporter using the existing reqwest+rustls stack. +//! +//! OTLP JSON encoding (`application/json`) per the OTLP/HTTP spec, so no +//! protobuf or tonic dependency. Invariants: +//! - Fail open: an export error is logged once per failure class and never +//! propagates to a caller. +//! - No request-path export: callers push into a bounded channel; a single +//! background worker drains it. Overflow drops and counts. +//! - No backend by default: constructed only when an endpoint is configured. + +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use serde_json::{json, Value}; +use tokio::sync::mpsc; + +/// Bounded queue depth. Overflow drops the newest record and increments +/// `dropped`, which surfaces as `preloop.telemetry.export{outcome="dropped"}`. +const QUEUE_CAPACITY: usize = 2048; +const BATCH_MAX: usize = 256; +const FLUSH_INTERVAL: Duration = Duration::from_secs(5); + +#[derive(Debug, Default)] +pub struct ExportHealth { + pub sent: AtomicU64, + pub failed: AtomicU64, + pub dropped: AtomicU64, + pub last_success_unix: AtomicU64, + pub last_failure_unix: AtomicU64, +} + +impl ExportHealth { + fn record_success(&self, n: u64) { + self.sent.fetch_add(n, Ordering::Relaxed); + self.last_success_unix.store(now_secs(), Ordering::Relaxed); + } + + fn record_failure(&self) { + self.failed.fetch_add(1, Ordering::Relaxed); + self.last_failure_unix.store(now_secs(), Ordering::Relaxed); + } + + pub fn dropped(&self) -> u64 { + self.dropped.load(Ordering::Relaxed) + } + + pub fn sent(&self) -> u64 { + self.sent.load(Ordering::Relaxed) + } + + pub fn failed(&self) -> u64 { + self.failed.load(Ordering::Relaxed) + } +} + +fn now_secs() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0) +} + +fn now_nanos() -> u128 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0) +} + +/// One log record queued for export. +#[derive(Debug, Clone)] +pub struct LogRecord { + pub severity: &'static str, + pub body: String, + pub attributes: Vec<(String, String)>, +} + +/// Handle used by the rest of the process to enqueue telemetry. +#[derive(Debug, Clone)] +pub struct Exporter { + tx: mpsc::Sender, + health: Arc, +} + +impl Exporter { + /// Enqueue a log record. Never blocks; drops on a full queue. + pub fn log(&self, record: LogRecord) { + if self.tx.try_send(record).is_err() { + self.health.dropped.fetch_add(1, Ordering::Relaxed); + } + } + + pub fn health(&self) -> &Arc { + &self.health + } +} + +/// Spawn the export worker. Returns `None` when no endpoint is configured, +/// so the absent-endpoint path opens no socket at all. +pub fn spawn( + endpoint: Option<&str>, + headers: Option<&str>, + service_name: &str, + instance_id: &str, +) -> Option<(Exporter, Arc)> { + let endpoint = endpoint?.trim_end_matches('/').to_string(); + let header_pairs = parse_headers(headers); + let service_name = service_name.to_string(); + let instance_id = instance_id.to_string(); + let health = Arc::new(ExportHealth::default()); + let (tx, mut rx) = mpsc::channel::(QUEUE_CAPACITY); + + let worker_health = health.clone(); + tokio::spawn(async move { + let client = match reqwest::Client::builder() + .timeout(Duration::from_secs(10)) + .build() + { + Ok(client) => client, + Err(error) => { + // Sanitized: never the endpoint or headers. + tracing::warn!( + failure = "client_build", + %error, + "telemetry export disabled" + ); + return; + } + }; + let logs_url = format!("{endpoint}/v1/logs"); + let mut buffer: Vec = Vec::with_capacity(BATCH_MAX); + let mut ticker = tokio::time::interval(FLUSH_INTERVAL); + loop { + tokio::select! { + maybe = rx.recv() => { + match maybe { + Some(record) => { + buffer.push(record); + if buffer.len() >= BATCH_MAX { + flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &mut buffer, &worker_health).await; + } + } + None => { + flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &mut buffer, &worker_health).await; + break; + } + } + } + _ = ticker.tick() => { + flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &mut buffer, &worker_health).await; + } + } + } + }); + + Some(( + Exporter { + tx, + health: health.clone(), + }, + health, + )) +} + +async fn flush( + client: &reqwest::Client, + url: &str, + headers: &[(String, String)], + service_name: &str, + instance_id: &str, + buffer: &mut Vec, + health: &Arc, +) { + if buffer.is_empty() { + return; + } + let batch = std::mem::take(buffer); + let count = batch.len() as u64; + let payload = encode_logs(&batch, service_name, instance_id); + let mut req = client.post(url).json(&payload); + for (name, value) in headers { + req = req.header(name.as_str(), value.as_str()); + } + match req.send().await { + Ok(response) if response.status().is_success() => health.record_success(count), + Ok(response) => { + // Status class only — never the body, which can echo credentials. + tracing::warn!( + failure = "http_status", + status = response.status().as_u16(), + "telemetry export failed" + ); + health.record_failure(); + } + Err(_) => { + // No error text: reqwest errors embed the URL, which may carry + // credentials in userinfo. + tracing::warn!(failure = "transport", "telemetry export failed"); + health.record_failure(); + } + } +} + +fn attr(key: &str, value: &str) -> Value { + json!({"key": key, "value": {"stringValue": value}}) +} + +fn severity_number(severity: &str) -> u8 { + match severity { + "TRACE" => 1, + "DEBUG" => 5, + "INFO" => 9, + "WARN" => 13, + "ERROR" => 17, + _ => 9, + } +} + +/// Encode a batch into the OTLP/HTTP JSON `ExportLogsServiceRequest` shape. +pub fn encode_logs(batch: &[LogRecord], service_name: &str, instance_id: &str) -> Value { + let ts = now_nanos().to_string(); + let records: Vec = batch + .iter() + .map(|record| { + let attributes: Vec = + record.attributes.iter().map(|(k, v)| attr(k, v)).collect(); + json!({ + "timeUnixNano": ts, + "observedTimeUnixNano": ts, + "severityNumber": severity_number(record.severity), + "severityText": record.severity, + "body": {"stringValue": record.body}, + "attributes": attributes, + }) + }) + .collect(); + json!({ + "resourceLogs": [{ + "resource": { + "attributes": [ + attr("service.name", service_name), + attr("service.instance.id", instance_id), + attr("service.version", env!("CARGO_PKG_VERSION")), + ] + }, + "scopeLogs": [{ + "scope": {"name": "preloop-observability"}, + "logRecords": records, + }] + }] + }) +} + +/// Parse `OTEL_EXPORTER_OTLP_HEADERS` (`k1=v1,k2=v2`). +fn parse_headers(raw: Option<&str>) -> Vec<(String, String)> { + let Some(raw) = raw else { + return Vec::new(); + }; + raw.split(',') + .filter_map(|pair| { + let (k, v) = pair.split_once('=')?; + let k = k.trim(); + let v = v.trim(); + if k.is_empty() || v.is_empty() { + None + } else { + Some((k.to_string(), v.to_string())) + } + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn absent_endpoint_spawns_nothing() { + assert!(spawn(None, None, "preloop", "abc").is_none()); + } + + #[test] + fn headers_parse_pairs() { + let parsed = parse_headers(Some("Authorization=Basic abc,stream-name=default")); + assert_eq!(parsed.len(), 2); + assert_eq!(parsed[0].0, "Authorization"); + assert_eq!(parsed[1].1, "default"); + } + + #[test] + fn headers_ignore_malformed() { + assert!(parse_headers(Some("novalue,=empty,k=")).is_empty()); + } + + #[test] + fn encodes_otlp_log_shape() { + let batch = vec![LogRecord { + severity: "WARN", + body: "pool provisioning failed".to_string(), + attributes: vec![("event.name".to_string(), "pool.provision".to_string())], + }]; + let payload = encode_logs(&batch, "preloop", "inst-1"); + let record = &payload["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]; + assert_eq!(record["severityText"], "WARN"); + assert_eq!(record["severityNumber"], 13); + assert_eq!(record["body"]["stringValue"], "pool provisioning failed"); + let resource = &payload["resourceLogs"][0]["resource"]["attributes"]; + assert_eq!(resource[0]["key"], "service.name"); + assert_eq!(resource[0]["value"]["stringValue"], "preloop"); + } +} diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs index b41a6097..43cb5ed3 100644 --- a/crates/preloop-observability/src/lib.rs +++ b/crates/preloop-observability/src/lib.rs @@ -11,6 +11,7 @@ //! - Always retain `stderr`/`journald` even when OTLP is configured. //! - `Debug` on config never reveals headers or credential-bearing endpoint parts. +pub mod export; pub mod metrics; pub mod status; pub mod vm_telemetry; @@ -141,6 +142,11 @@ impl ObservabilityConfig { self.otel_headers.is_some() } + /// Raw headers for transport construction. Never logged or in `Debug`. + pub fn otel_headers_raw(&self) -> Option<&str> { + self.otel_headers.as_deref() + } + /// Sanitized endpoint for `Debug`/errors: strips userinfo and query. fn sanitized_endpoint(&self) -> Option { self.otel_endpoint.as_ref().map(|raw| { @@ -385,6 +391,7 @@ struct Inner { limits: LimitRegistry, metrics: Arc, vm_registry: Arc, + exporter: Option, is_noop: bool, } @@ -413,6 +420,7 @@ impl Observability { limits: LimitRegistry::default(), metrics: Arc::new(metrics::MetricsRegistry::default()), vm_registry: Arc::new(vm_telemetry::VmTelemetryRegistry::default()), + exporter: None, is_noop: true, }), } @@ -421,6 +429,14 @@ impl Observability { /// Real handle from `ObservabilityConfig`. Does not install the global subscriber — pair with `ObservabilityRuntime`. pub fn from_config(config: ObservabilityConfig) -> (Self, ObservabilityRuntime) { let is_noop = !config.otlp_enabled; + // Absent endpoint spawns nothing at all — no worker, no socket. + let exporter = export::spawn( + config.otel_endpoint_raw(), + config.otel_headers_raw(), + &config.service_name, + &config.instance_id, + ) + .map(|(exporter, _health)| exporter); let handle = Self { inner: Arc::new(Inner { config: Arc::new(config), @@ -428,6 +444,7 @@ impl Observability { limits: LimitRegistry::default(), metrics: Arc::new(metrics::MetricsRegistry::default()), vm_registry: Arc::new(vm_telemetry::VmTelemetryRegistry::default()), + exporter, is_noop, }), }; @@ -467,6 +484,27 @@ impl Observability { &self.inner.vm_registry } + /// Enqueue a log record for OTLP export. No-op when export is disabled. + pub fn export_log( + &self, + severity: &'static str, + body: impl Into, + attributes: Vec<(String, String)>, + ) { + if let Some(exporter) = &self.inner.exporter { + exporter.log(export::LogRecord { + severity, + body: body.into(), + attributes, + }); + } + } + + /// Export health for `/api/v1/status` and `preloop.telemetry.export`. + pub fn export_health(&self) -> Option<&Arc> { + self.inner.exporter.as_ref().map(|e| e.health()) + } + pub fn config(&self) -> &ObservabilityConfig { &self.inner.config } diff --git a/crates/preloop-runner-server/src/main.rs b/crates/preloop-runner-server/src/main.rs index 44c46253..7542a45e 100644 --- a/crates/preloop-runner-server/src/main.rs +++ b/crates/preloop-runner-server/src/main.rs @@ -83,7 +83,9 @@ async fn main() -> anyhow::Result<()> { let (observability, observability_runtime) = preloop_observability::Observability::from_config(obs_config); preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); - let _observability = observability; + // The handle must reach `ServerConfig`; holding it here alone would leave + // the server on the no-op handle installed by `AppState::new`, so nothing + // would ever export. let _observability_runtime = observability_runtime; let cli = Cli::parse(); @@ -131,7 +133,7 @@ async fn main() -> anyhow::Result<()> { pool_preparing: None, listen, pool_status: None, - observability: None, + observability: Some(observability.clone()), systemd_socket_activation: false, unix_socket, state_dir, diff --git a/crates/preloop-runner-server/src/state.rs b/crates/preloop-runner-server/src/state.rs index 1cb3ae6e..00d0a0ee 100644 --- a/crates/preloop-runner-server/src/state.rs +++ b/crates/preloop-runner-server/src/state.rs @@ -841,6 +841,16 @@ impl AppState { _ => None, }; let has_run_projection = run_id.is_some(); + if let NdjsonEvent::RunAccepted { queued_jobs, .. } = &event { + self.observability.export_log( + "INFO", + "run.accepted", + vec![ + ("event.name".to_string(), "run.accepted".to_string()), + ("queued_jobs".to_string(), queued_jobs.to_string()), + ], + ); + } // Record job terminal transitions exactly once. Guard with is_terminal // so we don't double-count non-terminal status updates. The event // itself is the proof of old→terminal movement, so we record here @@ -876,6 +886,19 @@ impl AppState { .metrics() .lifecycle .record_job_completed(conclusion, bounded_reason); + self.observability.export_log( + if *status == preloop_gha_protocol::ExecutionStatus::Success { + "INFO" + } else { + "WARN" + }, + "job.completed", + vec![ + ("event.name".to_string(), "job.completed".to_string()), + ("conclusion".to_string(), conclusion.to_string()), + ("reason".to_string(), bounded_reason.to_string()), + ], + ); } NdjsonEvent::JobCompleted { status, .. } if status.is_terminal() => { let conclusion = match status { @@ -889,6 +912,14 @@ impl AppState { .metrics() .lifecycle .record_job_completed(conclusion, "completed"); + self.observability.export_log( + "INFO", + "job.completed", + vec![ + ("event.name".to_string(), "job.completed".to_string()), + ("conclusion".to_string(), conclusion.to_string()), + ], + ); } _ => {} } From 9085a1297ea3c0dc6964aade267759503532c835 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 18:53:28 -0400 Subject: [PATCH 20/22] fix(server): classify termination reasons and report the host binary version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects surfaced by reading an exported record. The reason label was wrong. The control plane's `reason` is not a code: the starvation sweep builds a prose sentence that interpolates the job's `runs-on` labels. Passing it through would explode metric cardinality and export workflow content, so the previous code bounded it — but it bounded every value, including the common `reason: None`, to "unknown". That labelled "no reason supplied" and "unrecognized string" identically and made a legitimately failed job unexplainable. Now `None` is `unspecified`, exact codes pass through, and prose is classified on its stable leading phrase, so the starvation sentence becomes `no_runner`. The full message stays on the log record; only the metric dimension is bounded. The event name conflated two records. A terminal `JobStatus` is a status transition, not the separate `JobCompleted` event, but both exported `body: "job.completed"`. The transition is now `job.status.terminal`. `service.version` reported the observability crate's version, which is meaningless to an operator. `ObservabilityConfig::with_service_version` now takes the host binary's version and all three binaries pass it. Tests: five cases for the classifier, including a hostile interpolated `runs-on` and a 1,000-string drive asserting the label set stays at two. Entire-Checkpoint: 01M0GP28K4J52DGX9CMDJVCYF5 --- crates/preloop-cli/src/main.rs | 3 +- crates/preloop-observability/src/export.rs | 29 ++-- crates/preloop-observability/src/lib.rs | 14 ++ crates/preloop-runner-server/src/main.rs | 3 +- crates/preloop-runner-server/src/state.rs | 157 ++++++++++++++++----- crates/preloop-runner/src/main.rs | 3 +- 6 files changed, 166 insertions(+), 43 deletions(-) diff --git a/crates/preloop-cli/src/main.rs b/crates/preloop-cli/src/main.rs index 549d4c69..30cc0138 100644 --- a/crates/preloop-cli/src/main.rs +++ b/crates/preloop-cli/src/main.rs @@ -749,7 +749,8 @@ async fn main() -> anyhow::Result<()> { // `RUST_LOG` defaults to `info` (the old `fmt::init()` default of ERROR hid // pool provisioning faults). The runtime is held for the life of `main` // and flushed with a bounded 2s shutdown on exit. - let obs_config = preloop_observability::ObservabilityConfig::from_env(); + let obs_config = preloop_observability::ObservabilityConfig::from_env() + .with_service_version(env!("CARGO_PKG_VERSION")); let (observability, observability_runtime) = preloop_observability::Observability::from_config(obs_config); preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); diff --git a/crates/preloop-observability/src/export.rs b/crates/preloop-observability/src/export.rs index 69d4bb63..67d3b91b 100644 --- a/crates/preloop-observability/src/export.rs +++ b/crates/preloop-observability/src/export.rs @@ -103,11 +103,13 @@ pub fn spawn( headers: Option<&str>, service_name: &str, instance_id: &str, + service_version: &str, ) -> Option<(Exporter, Arc)> { let endpoint = endpoint?.trim_end_matches('/').to_string(); let header_pairs = parse_headers(headers); let service_name = service_name.to_string(); let instance_id = instance_id.to_string(); + let service_version = service_version.to_string(); let health = Arc::new(ExportHealth::default()); let (tx, mut rx) = mpsc::channel::(QUEUE_CAPACITY); @@ -138,17 +140,17 @@ pub fn spawn( Some(record) => { buffer.push(record); if buffer.len() >= BATCH_MAX { - flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &mut buffer, &worker_health).await; + flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &service_version, &mut buffer, &worker_health).await; } } None => { - flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &mut buffer, &worker_health).await; + flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &service_version, &mut buffer, &worker_health).await; break; } } } _ = ticker.tick() => { - flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &mut buffer, &worker_health).await; + flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &service_version, &mut buffer, &worker_health).await; } } } @@ -169,6 +171,7 @@ async fn flush( headers: &[(String, String)], service_name: &str, instance_id: &str, + service_version: &str, buffer: &mut Vec, health: &Arc, ) { @@ -177,7 +180,7 @@ async fn flush( } let batch = std::mem::take(buffer); let count = batch.len() as u64; - let payload = encode_logs(&batch, service_name, instance_id); + let payload = encode_logs(&batch, service_name, instance_id, service_version); let mut req = client.post(url).json(&payload); for (name, value) in headers { req = req.header(name.as_str(), value.as_str()); @@ -218,7 +221,12 @@ fn severity_number(severity: &str) -> u8 { } /// Encode a batch into the OTLP/HTTP JSON `ExportLogsServiceRequest` shape. -pub fn encode_logs(batch: &[LogRecord], service_name: &str, instance_id: &str) -> Value { +pub fn encode_logs( + batch: &[LogRecord], + service_name: &str, + instance_id: &str, + service_version: &str, +) -> Value { let ts = now_nanos().to_string(); let records: Vec = batch .iter() @@ -241,7 +249,7 @@ pub fn encode_logs(batch: &[LogRecord], service_name: &str, instance_id: &str) - "attributes": [ attr("service.name", service_name), attr("service.instance.id", instance_id), - attr("service.version", env!("CARGO_PKG_VERSION")), + attr("service.version", service_version), ] }, "scopeLogs": [{ @@ -277,7 +285,7 @@ mod tests { #[test] fn absent_endpoint_spawns_nothing() { - assert!(spawn(None, None, "preloop", "abc").is_none()); + assert!(spawn(None, None, "preloop", "abc", "9.9.9").is_none()); } #[test] @@ -300,7 +308,7 @@ mod tests { body: "pool provisioning failed".to_string(), attributes: vec![("event.name".to_string(), "pool.provision".to_string())], }]; - let payload = encode_logs(&batch, "preloop", "inst-1"); + let payload = encode_logs(&batch, "preloop", "inst-1", "9.9.9"); let record = &payload["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]; assert_eq!(record["severityText"], "WARN"); assert_eq!(record["severityNumber"], 13); @@ -308,5 +316,10 @@ mod tests { let resource = &payload["resourceLogs"][0]["resource"]["attributes"]; assert_eq!(resource[0]["key"], "service.name"); assert_eq!(resource[0]["value"]["stringValue"], "preloop"); + assert_eq!(resource[2]["key"], "service.version"); + assert_eq!( + resource[2]["value"]["stringValue"], "9.9.9", + "service.version must come from the host binary, not this crate" + ); } } diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs index 43cb5ed3..a82e9c14 100644 --- a/crates/preloop-observability/src/lib.rs +++ b/crates/preloop-observability/src/lib.rs @@ -72,6 +72,9 @@ pub struct ObservabilityConfig { pub rust_log: String, /// `service.name` — `preloop` or `OTEL_SERVICE_NAME`. pub service_name: String, + /// `service.version` — set by the host binary via `with_service_version`. + /// Defaults to this crate's version only until the binary overrides it. + pub service_version: String, /// Per-process instance ID (UUID v4). pub instance_id: String, /// `OTEL_EXPORTER_OTLP_ENDPOINT` or signal-specific variant, if any. Kept as @@ -125,6 +128,7 @@ impl ObservabilityConfig { log_format, rust_log, service_name, + service_version: env!("CARGO_PKG_VERSION").to_string(), instance_id, otel_endpoint, otel_headers, @@ -142,6 +146,13 @@ impl ObservabilityConfig { self.otel_headers.is_some() } + /// Override `service.version` with the host binary's version. The crate's + /// own version is meaningless to an operator reading telemetry. + pub fn with_service_version(mut self, version: &str) -> Self { + self.service_version = version.to_string(); + self + } + /// Raw headers for transport construction. Never logged or in `Debug`. pub fn otel_headers_raw(&self) -> Option<&str> { self.otel_headers.as_deref() @@ -171,6 +182,7 @@ impl fmt::Debug for ObservabilityConfig { .field("log_format", &self.log_format) .field("rust_log", &self.rust_log) .field("service_name", &self.service_name) + .field("service_version", &self.service_version) .field("instance_id", &self.instance_id) .field( "otel_endpoint", @@ -408,6 +420,7 @@ impl Observability { log_format: LogFormat::Auto, rust_log: "info".to_string(), service_name: "preloop".to_string(), + service_version: env!("CARGO_PKG_VERSION").to_string(), instance_id: Uuid::new_v4().to_string(), otel_endpoint: None, otel_headers: None, @@ -435,6 +448,7 @@ impl Observability { config.otel_headers_raw(), &config.service_name, &config.instance_id, + &config.service_version, ) .map(|(exporter, _health)| exporter); let handle = Self { diff --git a/crates/preloop-runner-server/src/main.rs b/crates/preloop-runner-server/src/main.rs index 7542a45e..afd6b67f 100644 --- a/crates/preloop-runner-server/src/main.rs +++ b/crates/preloop-runner-server/src/main.rs @@ -79,7 +79,8 @@ async fn main() -> anyhow::Result<()> { // like the CLI, instead of falling silent when unset. `PRELOOP_LOG_FORMAT` // controls pretty/json/auto. The `Observability` handle will be cloned // into `AppState`; for now it is held for the life of `main`. - let obs_config = preloop_observability::ObservabilityConfig::from_env(); + let obs_config = preloop_observability::ObservabilityConfig::from_env() + .with_service_version(env!("CARGO_PKG_VERSION")); let (observability, observability_runtime) = preloop_observability::Observability::from_config(obs_config); preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); diff --git a/crates/preloop-runner-server/src/state.rs b/crates/preloop-runner-server/src/state.rs index 00d0a0ee..a0b10bd1 100644 --- a/crates/preloop-runner-server/src/state.rs +++ b/crates/preloop-runner-server/src/state.rs @@ -561,6 +561,62 @@ pub(crate) enum JobSetAdmissionResult { Blocked, } +/// Bounded conclusion label for a terminal execution status. +fn execution_conclusion(status: preloop_gha_protocol::ExecutionStatus) -> &'static str { + use preloop_gha_protocol::ExecutionStatus as S; + match status { + S::Success => "success", + S::Failure => "failure", + S::Cancelled => "cancelled", + S::Skipped => "skipped", + // Non-terminal statuses never reach here; the guard filters them. + _ => "unrecognized", + } +} + +/// Classify a termination reason into a bounded code. +/// +/// The control plane's `reason` is not a code — several paths build a prose +/// sentence that interpolates the job's `runs-on` labels (see the starvation +/// sweep in `bootstrap.rs`). Those values are user-controlled, so the raw +/// string must never reach a metric label: it would both explode cardinality +/// and export workflow content. Classify by the stable prefix each path +/// writes, and fall back to `unrecognized` rather than passing prose through. +/// +/// The full message is still available on the structured log record; only the +/// metric dimension is bounded. +fn bounded_termination_reason(value: &str) -> &'static str { + // Exact codes first — these come from `concurrency::*_reason()`. + match value { + "concurrency_pending" => return "concurrency_pending", + "concurrency_cancelled" => return "concurrency_cancelled", + "timeout" => return "timeout", + "no_runner" => return "no_runner", + "lease_expired" => return "lease_expired", + "deaf_runner" => return "deaf_runner", + "startup_orphan" => return "startup_orphan", + _ => {} + } + // Prose paths — match on the stable leading phrase, never the whole + // string, so an interpolated label cannot change the classification. + if value.starts_with("no runner is registered for") { + return "no_runner"; + } + if value.starts_with("job exceeded its timeout") + || value.starts_with("timed out") + || value.contains("timeout-minutes") + { + return "timeout"; + } + if value.starts_with("runner stopped polling") || value.contains("deaf") { + return "deaf_runner"; + } + if value.contains("lease expired") { + return "lease_expired"; + } + "unrecognized" +} + impl AppState { pub async fn new(state_dir: PathBuf) -> anyhow::Result { let config_path = crate::config::config_path(); @@ -858,29 +914,15 @@ impl AppState { // duplicate `store_run_event` emits. match &event { NdjsonEvent::JobStatus { status, reason, .. } if status.is_terminal() => { - let conclusion = match status { - preloop_gha_protocol::ExecutionStatus::Success => "success", - preloop_gha_protocol::ExecutionStatus::Failure => "failure", - preloop_gha_protocol::ExecutionStatus::Cancelled => "cancelled", - preloop_gha_protocol::ExecutionStatus::Skipped => "skipped", - _ => "unknown", - }; - let reason_str = reason.as_deref().unwrap_or("unknown"); - // Bound the reason to the finite set the plan allows; unknown - // reasons are mapped to "unknown" so they don't create new series. - let bounded_reason = match reason_str { - "timeout" - | "no_runner" - | "lease_expired" - | "deaf_runner" - | "startup_orphan" - | "concurrency_cancelled" - | "concurrency_pending" - | "success" - | "failure" - | "cancelled" - | "skipped" => reason_str, - _ => "unknown", + let conclusion = execution_conclusion(*status); + // `reason: None` is the common case (most terminal transitions + // carry none) and means "no reason supplied" — not + // "unrecognized". Only a value outside the emitted set is + // `unrecognized`, which keeps the label bounded without + // mislabelling the majority. + let bounded_reason = match reason.as_deref() { + None => "unspecified", + Some(value) => bounded_termination_reason(value), }; self.observability .metrics() @@ -892,22 +934,19 @@ impl AppState { } else { "WARN" }, - "job.completed", + // A terminal JobStatus is a status transition, not the + // separate JobCompleted event; naming both `job.completed` + // conflated two distinct records in the log stream. + "job.status.terminal", vec![ - ("event.name".to_string(), "job.completed".to_string()), + ("event.name".to_string(), "job.status.terminal".to_string()), ("conclusion".to_string(), conclusion.to_string()), ("reason".to_string(), bounded_reason.to_string()), ], ); } NdjsonEvent::JobCompleted { status, .. } if status.is_terminal() => { - let conclusion = match status { - preloop_gha_protocol::ExecutionStatus::Success => "success", - preloop_gha_protocol::ExecutionStatus::Failure => "failure", - preloop_gha_protocol::ExecutionStatus::Cancelled => "cancelled", - preloop_gha_protocol::ExecutionStatus::Skipped => "skipped", - _ => "unknown", - }; + let conclusion = execution_conclusion(*status); self.observability .metrics() .lifecycle @@ -1369,3 +1408,57 @@ mod tests { ); } } + +#[cfg(test)] +mod termination_reason_tests { + use super::bounded_termination_reason; + + #[test] + fn exact_codes_pass_through() { + assert_eq!( + bounded_termination_reason("concurrency_cancelled"), + "concurrency_cancelled" + ); + assert_eq!(bounded_termination_reason("timeout"), "timeout"); + } + + #[test] + fn starvation_prose_classifies_to_no_runner() { + // The starvation sweep builds this sentence with the job's runs-on + // labels interpolated. It must classify, not pass through. + let prose = "no runner is registered for `runs-on: self-hosted, Linux, ARM64` and none \ + appeared within 120s, so the job cannot be scheduled"; + assert_eq!(bounded_termination_reason(prose), "no_runner"); + } + + #[test] + fn user_controlled_labels_never_become_the_label() { + // A hostile or merely unusual `runs-on` must not reach the metric. + let prose = "no runner is registered for `runs-on: attacker-controlled-\u{1F4A5}-label` \ + and none appeared within 120s, so the job cannot be scheduled"; + let bounded = bounded_termination_reason(prose); + assert_eq!(bounded, "no_runner"); + assert!(!bounded.contains("attacker")); + } + + #[test] + fn unknown_prose_is_bounded_not_passed_through() { + let bounded = bounded_termination_reason("something entirely new happened with id-99999"); + assert_eq!(bounded, "unrecognized"); + assert!(!bounded.contains("99999")); + } + + #[test] + fn classification_is_a_finite_set() { + // Drive 1,000 distinct prose strings; the label set must stay bounded. + let mut seen = std::collections::BTreeSet::new(); + for i in 0..1000 { + let prose = format!( + "no runner is registered for `runs-on: label-{i}` and none appeared within 120s" + ); + seen.insert(bounded_termination_reason(&prose)); + seen.insert(bounded_termination_reason(&format!("novel reason {i}"))); + } + assert_eq!(seen.len(), 2, "expected exactly no_runner + unrecognized"); + } +} diff --git a/crates/preloop-runner/src/main.rs b/crates/preloop-runner/src/main.rs index a7c5a687..10740709 100644 --- a/crates/preloop-runner/src/main.rs +++ b/crates/preloop-runner/src/main.rs @@ -16,7 +16,8 @@ async fn main() -> Result<()> { // Runner gets structured local logging only — never OTLP export by default. // `PRELOOP_LOG_FORMAT` still controls pretty/json/auto for consistency. - let obs_config = preloop_observability::ObservabilityConfig::from_env(); + let obs_config = preloop_observability::ObservabilityConfig::from_env() + .with_service_version(env!("CARGO_PKG_VERSION")); let (observability, observability_runtime) = preloop_observability::Observability::from_config(obs_config); preloop_observability::ObservabilityRuntime::install_fmt_subscriber(observability.config()); From 5808ba40abfd2345866988850afcc82c8886bdf7 Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 19:02:12 -0400 Subject: [PATCH 21/22] feat(observability): export metrics and traces over OTLP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Logs were the only signal reaching a backend; metrics were Prometheus-pull only and traces did not exist at all — the HTTP middleware built a span and dropped it. Metrics. `MetricsRegistry::collect` snapshots every instrument as OTLP-ready families, so export scrapes the same instruments `/metrics` renders rather than maintaining a second set. Counters become cumulative monotonic sums, gauges become gauges, and the internal histogram becomes an explicit-bucket histogram with the implicit `+Inf` bucket OTLP requires. Every cumulative point carries the process start as `startTimeUnixNano`, without which a backend reads a restart as a counter reset. Traces. Add real W3C Trace Context: an inbound `traceparent` is adopted so a caller's trace continues through the control plane, a malformed one starts a new root rather than failing the request, and all-zero ids are rejected per the spec. Spans carry the matched route template and finite surface, never the raw URI. Only 5xx sets `Error` — marking 4xx would make every unauthenticated probe look like an outage. Health and metrics probes are suppressed from trace export; they would swamp the store and explain nothing. Log records now carry `traceId`/`spanId` as OTLP fields, not attributes, so a backend can pivot log to trace. One worker drains logs and spans from a shared bounded queue and scrapes metrics on the same tick, so all three share one client, one batching cadence, and one fail-open path. Verified against a pinned single-node OpenObserve: traces, logs and metrics streams all populated; an injected traceparent arrived as trace_id=4bf92f3577b34da6a3ce929d0e0e4736 with a fresh span id; histogram points carry AGGREGATION_TEMPORALITY_CUMULATIVE with bounded attributes (http_route, preloop_surface) and service_version 0.2.0; public probes absent from spans. 24 crate tests including traceparent adoption, malformed rejection, id uniqueness, OTLP shapes, and the +Inf bucket invariant. Entire-Checkpoint: 01M0GPJ7JVYWNJMPJYVE5QW4P6 --- crates/preloop-observability/Cargo.toml | 1 + crates/preloop-observability/src/export.rs | 623 ++++++++++++++++-- crates/preloop-observability/src/lib.rs | 33 +- crates/preloop-observability/src/metrics.rs | 239 +++++++ .../preloop-runner-server/src/http_metrics.rs | 71 +- 5 files changed, 895 insertions(+), 72 deletions(-) diff --git a/crates/preloop-observability/Cargo.toml b/crates/preloop-observability/Cargo.toml index d8c694c9..fd724f6a 100644 --- a/crates/preloop-observability/Cargo.toml +++ b/crates/preloop-observability/Cargo.toml @@ -10,6 +10,7 @@ repository = "https://github.com/preloopdev/preloop" anyhow = { workspace = true } chrono = { workspace = true } parking_lot = { workspace = true } +rand = { workspace = true } reqwest = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } diff --git a/crates/preloop-observability/src/export.rs b/crates/preloop-observability/src/export.rs index 67d3b91b..4110678f 100644 --- a/crates/preloop-observability/src/export.rs +++ b/crates/preloop-observability/src/export.rs @@ -61,32 +61,136 @@ fn now_secs() -> u64 { .unwrap_or(0) } -fn now_nanos() -> u128 { +/// Wall-clock nanoseconds since the epoch, for OTLP timestamps. +pub fn now_nanos() -> u128 { SystemTime::now() .duration_since(UNIX_EPOCH) .map(|d| d.as_nanos()) .unwrap_or(0) } +/// W3C Trace Context: 16-byte trace id / 8-byte span id, lowercase hex. +/// +/// Generated locally when a request arrives without a `traceparent`, or +/// adopted from the incoming header so a caller's trace continues through +/// the control plane. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SpanContext { + pub trace_id: String, + pub span_id: String, + pub parent_span_id: Option, +} + +impl SpanContext { + /// New root context with a random trace and span id. + pub fn root() -> Self { + Self { + trace_id: random_hex(16), + span_id: random_hex(8), + parent_span_id: None, + } + } + + /// Child of an incoming `traceparent`, or a new root when absent/invalid. + /// + /// Format: `00-<32 hex trace>-<16 hex span>-<2 hex flags>`. A malformed + /// header starts a new trace rather than failing the request — telemetry + /// must never reject traffic. + pub fn from_traceparent(header: Option<&str>) -> Self { + let Some(raw) = header else { + return Self::root(); + }; + let parts: Vec<&str> = raw.trim().split('-').collect(); + if parts.len() != 4 { + return Self::root(); + } + let (version, trace_id, parent_span_id) = (parts[0], parts[1], parts[2]); + let valid = version.len() == 2 + && trace_id.len() == 32 + && parent_span_id.len() == 16 + && trace_id.chars().all(|c| c.is_ascii_hexdigit()) + && parent_span_id.chars().all(|c| c.is_ascii_hexdigit()) + // All-zero ids are explicitly invalid per the spec. + && trace_id.chars().any(|c| c != '0') + && parent_span_id.chars().any(|c| c != '0'); + if !valid { + return Self::root(); + } + Self { + trace_id: trace_id.to_ascii_lowercase(), + span_id: random_hex(8), + parent_span_id: Some(parent_span_id.to_ascii_lowercase()), + } + } +} + +fn random_hex(bytes: usize) -> String { + use std::fmt::Write; + let mut out = String::with_capacity(bytes * 2); + for _ in 0..bytes { + let byte: u8 = rand::random(); + let _ = write!(out, "{byte:02x}"); + } + out +} + +/// OTLP span status. `Unset` is the default for a successful server span; +/// only an actual error sets `Error`, per the OTLP spec. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SpanStatus { + Unset, + Error, +} + +/// One completed span queued for export. +#[derive(Debug, Clone)] +pub struct SpanRecord { + pub context: SpanContext, + pub name: String, + pub start_nanos: u128, + pub end_nanos: u128, + pub status: SpanStatus, + pub attributes: Vec<(String, String)>, +} + /// One log record queued for export. #[derive(Debug, Clone)] pub struct LogRecord { pub severity: &'static str, pub body: String, pub attributes: Vec<(String, String)>, + /// Correlates this record with a span. OTLP carries these as first-class + /// fields, not attributes, so a backend can pivot log <-> trace. + pub trace_id: Option, + pub span_id: Option, +} + +/// Either signal, multiplexed over one bounded channel so a burst of one +/// cannot starve the other beyond the shared capacity. +#[derive(Debug, Clone)] +pub enum Item { + Log(LogRecord), + Span(SpanRecord), } /// Handle used by the rest of the process to enqueue telemetry. #[derive(Debug, Clone)] pub struct Exporter { - tx: mpsc::Sender, + tx: mpsc::Sender, health: Arc, } impl Exporter { /// Enqueue a log record. Never blocks; drops on a full queue. pub fn log(&self, record: LogRecord) { - if self.tx.try_send(record).is_err() { + if self.tx.try_send(Item::Log(record)).is_err() { + self.health.dropped.fetch_add(1, Ordering::Relaxed); + } + } + + /// Enqueue a completed span. Never blocks; drops on a full queue. + pub fn span(&self, record: SpanRecord) { + if self.tx.try_send(Item::Span(record)).is_err() { self.health.dropped.fetch_add(1, Ordering::Relaxed); } } @@ -98,20 +202,27 @@ impl Exporter { /// Spawn the export worker. Returns `None` when no endpoint is configured, /// so the absent-endpoint path opens no socket at all. +/// +/// One worker drains logs and spans from the shared queue and scrapes the +/// metrics registry on each tick, so all three signals share one batching +/// cadence and one client. pub fn spawn( endpoint: Option<&str>, headers: Option<&str>, service_name: &str, instance_id: &str, service_version: &str, + metrics: Option>, ) -> Option<(Exporter, Arc)> { let endpoint = endpoint?.trim_end_matches('/').to_string(); let header_pairs = parse_headers(headers); - let service_name = service_name.to_string(); - let instance_id = instance_id.to_string(); - let service_version = service_version.to_string(); + let resource = Resource { + service_name: service_name.to_string(), + instance_id: instance_id.to_string(), + service_version: service_version.to_string(), + }; let health = Arc::new(ExportHealth::default()); - let (tx, mut rx) = mpsc::channel::(QUEUE_CAPACITY); + let (tx, mut rx) = mpsc::channel::(QUEUE_CAPACITY); let worker_health = health.clone(); tokio::spawn(async move { @@ -122,35 +233,51 @@ pub fn spawn( Ok(client) => client, Err(error) => { // Sanitized: never the endpoint or headers. - tracing::warn!( - failure = "client_build", - %error, - "telemetry export disabled" - ); + tracing::warn!(failure = "client_build", %error, "telemetry export disabled"); return; } }; - let logs_url = format!("{endpoint}/v1/logs"); - let mut buffer: Vec = Vec::with_capacity(BATCH_MAX); + // Cumulative temporality needs a fixed start for every point, or a + // backend cannot tell a restart from a counter reset. + let start_nanos = now_nanos(); + let urls = Urls { + logs: format!("{endpoint}/v1/logs"), + traces: format!("{endpoint}/v1/traces"), + metrics: format!("{endpoint}/v1/metrics"), + }; + let mut logs: Vec = Vec::with_capacity(BATCH_MAX); + let mut spans: Vec = Vec::with_capacity(BATCH_MAX); let mut ticker = tokio::time::interval(FLUSH_INTERVAL); loop { tokio::select! { maybe = rx.recv() => { match maybe { - Some(record) => { - buffer.push(record); - if buffer.len() >= BATCH_MAX { - flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &service_version, &mut buffer, &worker_health).await; + Some(Item::Log(record)) => { + logs.push(record); + if logs.len() >= BATCH_MAX { + flush_logs(&client, &urls, &header_pairs, &resource, &mut logs, &worker_health).await; + } + } + Some(Item::Span(record)) => { + spans.push(record); + if spans.len() >= BATCH_MAX { + flush_spans(&client, &urls, &header_pairs, &resource, &mut spans, &worker_health).await; } } None => { - flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &service_version, &mut buffer, &worker_health).await; + // Channel closed: final drain, then exit. + flush_logs(&client, &urls, &header_pairs, &resource, &mut logs, &worker_health).await; + flush_spans(&client, &urls, &header_pairs, &resource, &mut spans, &worker_health).await; break; } } } _ = ticker.tick() => { - flush(&client, &logs_url, &header_pairs, &service_name, &instance_id, &service_version, &mut buffer, &worker_health).await; + flush_logs(&client, &urls, &header_pairs, &resource, &mut logs, &worker_health).await; + flush_spans(&client, &urls, &header_pairs, &resource, &mut spans, &worker_health).await; + if let Some(registry) = &metrics { + flush_metrics(&client, &urls, &header_pairs, &resource, registry, start_nanos, &worker_health).await; + } } } } @@ -165,23 +292,31 @@ pub fn spawn( )) } -async fn flush( +#[derive(Debug, Clone)] +struct Urls { + logs: String, + traces: String, + metrics: String, +} + +#[derive(Debug, Clone)] +struct Resource { + service_name: String, + instance_id: String, + service_version: String, +} + +/// POST one payload, recording health. Never propagates an error. +async fn post( client: &reqwest::Client, url: &str, headers: &[(String, String)], - service_name: &str, - instance_id: &str, - service_version: &str, - buffer: &mut Vec, + payload: &Value, + signal: &'static str, + count: u64, health: &Arc, ) { - if buffer.is_empty() { - return; - } - let batch = std::mem::take(buffer); - let count = batch.len() as u64; - let payload = encode_logs(&batch, service_name, instance_id, service_version); - let mut req = client.post(url).json(&payload); + let mut req = client.post(url).json(payload); for (name, value) in headers { req = req.header(name.as_str(), value.as_str()); } @@ -191,6 +326,7 @@ async fn flush( // Status class only — never the body, which can echo credentials. tracing::warn!( failure = "http_status", + signal, status = response.status().as_u16(), "telemetry export failed" ); @@ -199,16 +335,112 @@ async fn flush( Err(_) => { // No error text: reqwest errors embed the URL, which may carry // credentials in userinfo. - tracing::warn!(failure = "transport", "telemetry export failed"); + tracing::warn!(failure = "transport", signal, "telemetry export failed"); health.record_failure(); } } } +async fn flush_logs( + client: &reqwest::Client, + urls: &Urls, + headers: &[(String, String)], + resource: &Resource, + buffer: &mut Vec, + health: &Arc, +) { + if buffer.is_empty() { + return; + } + let batch = std::mem::take(buffer); + let count = batch.len() as u64; + let payload = encode_logs( + &batch, + &resource.service_name, + &resource.instance_id, + &resource.service_version, + ); + post(client, &urls.logs, headers, &payload, "logs", count, health).await; +} + +async fn flush_spans( + client: &reqwest::Client, + urls: &Urls, + headers: &[(String, String)], + resource: &Resource, + buffer: &mut Vec, + health: &Arc, +) { + if buffer.is_empty() { + return; + } + let batch = std::mem::take(buffer); + let count = batch.len() as u64; + let payload = encode_spans( + &batch, + &resource.service_name, + &resource.instance_id, + &resource.service_version, + ); + post( + client, + &urls.traces, + headers, + &payload, + "traces", + count, + health, + ) + .await; +} + +async fn flush_metrics( + client: &reqwest::Client, + urls: &Urls, + headers: &[(String, String)], + resource: &Resource, + registry: &Arc, + start_nanos: u128, + health: &Arc, +) { + let families = registry.collect(); + if families.is_empty() { + return; + } + let count = families.len() as u64; + let payload = encode_metrics( + &families, + &resource.service_name, + &resource.instance_id, + &resource.service_version, + start_nanos, + ); + post( + client, + &urls.metrics, + headers, + &payload, + "metrics", + count, + health, + ) + .await; +} + +const SCOPE_NAME: &str = "preloop-observability"; + fn attr(key: &str, value: &str) -> Value { json!({"key": key, "value": {"stringValue": value}}) } +fn resource_attributes(service_name: &str, instance_id: &str, service_version: &str) -> Vec { + vec![ + attr("service.name", service_name), + attr("service.instance.id", instance_id), + attr("service.version", service_version), + ] +} + fn severity_number(severity: &str) -> u8 { match severity { "TRACE" => 1, @@ -233,25 +465,25 @@ pub fn encode_logs( .map(|record| { let attributes: Vec = record.attributes.iter().map(|(k, v)| attr(k, v)).collect(); - json!({ + let mut obj = json!({ "timeUnixNano": ts, "observedTimeUnixNano": ts, "severityNumber": severity_number(record.severity), "severityText": record.severity, "body": {"stringValue": record.body}, "attributes": attributes, - }) + }); + // OTLP carries correlation as first-class fields, not attributes. + if let (Some(trace_id), Some(span_id)) = (&record.trace_id, &record.span_id) { + obj["traceId"] = json!(trace_id); + obj["spanId"] = json!(span_id); + } + obj }) .collect(); json!({ "resourceLogs": [{ - "resource": { - "attributes": [ - attr("service.name", service_name), - attr("service.instance.id", instance_id), - attr("service.version", service_version), - ] - }, + "resource": {"attributes": resource_attributes(service_name, instance_id, service_version)}, "scopeLogs": [{ "scope": {"name": "preloop-observability"}, "logRecords": records, @@ -279,13 +511,166 @@ fn parse_headers(raw: Option<&str>) -> Vec<(String, String)> { .collect() } +/// Encode a batch into the OTLP/HTTP JSON `ExportTraceServiceRequest` shape. +pub fn encode_spans( + batch: &[SpanRecord], + service_name: &str, + instance_id: &str, + service_version: &str, +) -> Value { + let spans: Vec = batch + .iter() + .map(|record| { + let attributes: Vec = + record.attributes.iter().map(|(k, v)| attr(k, v)).collect(); + let mut obj = json!({ + "traceId": record.context.trace_id, + "spanId": record.context.span_id, + "name": record.name, + // 2 = SPAN_KIND_SERVER: every span we emit today is an + // inbound request handled by the control plane. + "kind": 2, + "startTimeUnixNano": record.start_nanos.to_string(), + "endTimeUnixNano": record.end_nanos.to_string(), + "attributes": attributes, + "status": {"code": match record.status { + SpanStatus::Unset => 0, + SpanStatus::Error => 2, + }}, + }); + if let Some(parent) = &record.context.parent_span_id { + obj["parentSpanId"] = json!(parent); + } + obj + }) + .collect(); + json!({ + "resourceSpans": [{ + "resource": {"attributes": resource_attributes(service_name, instance_id, service_version)}, + "scopeSpans": [{ + "scope": {"name": SCOPE_NAME}, + "spans": spans, + }] + }] + }) +} + +/// Encode collected families into the OTLP/HTTP JSON `ExportMetricsServiceRequest`. +/// +/// All points are cumulative (`aggregationTemporality: 2`) and share the +/// process start as `startTimeUnixNano`, so a backend can distinguish a +/// restart from a counter reset. +pub fn encode_metrics( + families: &[crate::metrics::MetricFamily], + service_name: &str, + instance_id: &str, + service_version: &str, + start_nanos: u128, +) -> Value { + use crate::metrics::MetricPoint; + const CUMULATIVE: u8 = 2; + let now = now_nanos().to_string(); + let start = start_nanos.to_string(); + + let metrics: Vec = families + .iter() + .map(|family| { + let mut metric = json!({"name": family.name, "unit": family.unit}); + match family.points.first() { + Some(MetricPoint::Sum { .. }) => { + let points: Vec = family + .points + .iter() + .filter_map(|point| match point { + MetricPoint::Sum { value, attributes } => Some(json!({ + "asDouble": value, + "startTimeUnixNano": start, + "timeUnixNano": now, + "attributes": encode_attrs(attributes), + })), + _ => None, + }) + .collect(); + metric["sum"] = json!({ + "dataPoints": points, + "aggregationTemporality": CUMULATIVE, + "isMonotonic": true, + }); + } + Some(MetricPoint::Gauge { .. }) => { + let points: Vec = family + .points + .iter() + .filter_map(|point| match point { + MetricPoint::Gauge { value, attributes } => Some(json!({ + "asDouble": value, + "timeUnixNano": now, + "attributes": encode_attrs(attributes), + })), + _ => None, + }) + .collect(); + metric["gauge"] = json!({"dataPoints": points}); + } + Some(MetricPoint::Histogram { .. }) => { + let points: Vec = family + .points + .iter() + .filter_map(|point| match point { + MetricPoint::Histogram { + count, + sum, + bounds, + bucket_counts, + attributes, + } => Some(json!({ + "count": count.to_string(), + "sum": sum, + "explicitBounds": bounds, + "bucketCounts": bucket_counts + .iter() + .map(|c| c.to_string()) + .collect::>(), + "startTimeUnixNano": start, + "timeUnixNano": now, + "attributes": encode_attrs(attributes), + })), + _ => None, + }) + .collect(); + metric["histogram"] = json!({ + "dataPoints": points, + "aggregationTemporality": CUMULATIVE, + }); + } + None => {} + } + metric + }) + .collect(); + + json!({ + "resourceMetrics": [{ + "resource": {"attributes": resource_attributes(service_name, instance_id, service_version)}, + "scopeMetrics": [{ + "scope": {"name": SCOPE_NAME}, + "metrics": metrics, + }] + }] + }) +} + +fn encode_attrs(attributes: &[(String, String)]) -> Vec { + attributes.iter().map(|(k, v)| attr(k, v)).collect() +} + #[cfg(test)] mod tests { use super::*; #[test] fn absent_endpoint_spawns_nothing() { - assert!(spawn(None, None, "preloop", "abc", "9.9.9").is_none()); + assert!(spawn(None, None, "preloop", "abc", "9.9.9", None).is_none()); } #[test] @@ -307,6 +692,8 @@ mod tests { severity: "WARN", body: "pool provisioning failed".to_string(), attributes: vec![("event.name".to_string(), "pool.provision".to_string())], + trace_id: None, + span_id: None, }]; let payload = encode_logs(&batch, "preloop", "inst-1", "9.9.9"); let record = &payload["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]; @@ -323,3 +710,153 @@ mod tests { ); } } + +#[cfg(test)] +mod signal_tests { + use super::*; + use crate::metrics::{MetricFamily, MetricPoint}; + + #[test] + fn traceparent_is_adopted_as_parent() { + let ctx = SpanContext::from_traceparent(Some( + "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", + )); + assert_eq!(ctx.trace_id, "4bf92f3577b34da6a3ce929d0e0e4736"); + assert_eq!(ctx.parent_span_id.as_deref(), Some("00f067aa0ba902b7")); + // A child must get its own span id, never reuse the parent's. + assert_ne!(ctx.span_id, "00f067aa0ba902b7"); + assert_eq!(ctx.span_id.len(), 16); + } + + #[test] + fn malformed_traceparent_starts_a_new_root() { + for bad in [ + "garbage", + "00-tooshort-00f067aa0ba902b7-01", + // All-zero ids are invalid per the spec. + "00-00000000000000000000000000000000-00f067aa0ba902b7-01", + "00-4bf92f3577b34da6a3ce929d0e0e4736-0000000000000000-01", + "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7", + ] { + let ctx = SpanContext::from_traceparent(Some(bad)); + assert!( + ctx.parent_span_id.is_none(), + "{bad} must not adopt a parent" + ); + assert_eq!(ctx.trace_id.len(), 32); + } + } + + #[test] + fn generated_ids_are_unique_and_well_formed() { + let a = SpanContext::root(); + let b = SpanContext::root(); + assert_ne!(a.trace_id, b.trace_id); + assert_eq!(a.trace_id.len(), 32); + assert_eq!(a.span_id.len(), 8 * 2); + assert!(a.trace_id.chars().all(|c| c.is_ascii_hexdigit())); + } + + #[test] + fn spans_encode_to_otlp_shape() { + let ctx = SpanContext::from_traceparent(Some( + "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", + )); + let batch = vec![SpanRecord { + context: ctx, + name: "GET /api/v1/runs/:run_id".to_string(), + start_nanos: 1_000, + end_nanos: 2_000, + status: SpanStatus::Error, + attributes: vec![("http.route".to_string(), "/api/v1/runs/:run_id".to_string())], + }]; + let payload = encode_spans(&batch, "preloop", "inst", "0.2.0"); + let span = &payload["resourceSpans"][0]["scopeSpans"][0]["spans"][0]; + assert_eq!(span["traceId"], "4bf92f3577b34da6a3ce929d0e0e4736"); + assert_eq!(span["parentSpanId"], "00f067aa0ba902b7"); + assert_eq!(span["kind"], 2, "server span"); + assert_eq!(span["status"]["code"], 2, "error"); + assert_eq!(span["startTimeUnixNano"], "1000"); + } + + #[test] + fn logs_carry_trace_correlation_as_fields() { + let batch = vec![LogRecord { + severity: "WARN", + body: "job.status.terminal".to_string(), + attributes: vec![], + trace_id: Some("4bf92f3577b34da6a3ce929d0e0e4736".to_string()), + span_id: Some("00f067aa0ba902b7".to_string()), + }]; + let payload = encode_logs(&batch, "preloop", "inst", "0.2.0"); + let record = &payload["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]; + // OTLP requires these as fields, not attributes, for log<->trace pivot. + assert_eq!(record["traceId"], "4bf92f3577b34da6a3ce929d0e0e4736"); + assert_eq!(record["spanId"], "00f067aa0ba902b7"); + } + + #[test] + fn counters_encode_as_cumulative_monotonic_sums() { + let families = vec![MetricFamily { + name: "preloop.job.completed".to_string(), + unit: "{job}", + points: vec![MetricPoint::Sum { + value: 7.0, + attributes: vec![("preloop.conclusion".to_string(), "failure".to_string())], + }], + }]; + let payload = encode_metrics(&families, "preloop", "inst", "0.2.0", 500); + let metric = &payload["resourceMetrics"][0]["scopeMetrics"][0]["metrics"][0]; + assert_eq!(metric["name"], "preloop.job.completed"); + assert_eq!(metric["sum"]["isMonotonic"], true); + assert_eq!(metric["sum"]["aggregationTemporality"], 2, "cumulative"); + let point = &metric["sum"]["dataPoints"][0]; + assert_eq!(point["asDouble"], 7.0); + assert_eq!( + point["startTimeUnixNano"], "500", + "cumulative points need a fixed start or a restart reads as a reset" + ); + } + + #[test] + fn histograms_encode_with_the_implicit_inf_bucket() { + let families = vec![MetricFamily { + name: "http.server.request.duration".to_string(), + unit: "s", + points: vec![MetricPoint::Histogram { + count: 5, + sum: 0.25, + bounds: vec![0.005, 0.01], + bucket_counts: vec![1, 3, 5], + attributes: vec![("http.route".to_string(), "/healthz".to_string())], + }], + }]; + let payload = encode_metrics(&families, "preloop", "inst", "0.2.0", 0); + let point = &payload["resourceMetrics"][0]["scopeMetrics"][0]["metrics"][0]["histogram"] + ["dataPoints"][0]; + let bounds = point["explicitBounds"].as_array().unwrap(); + let counts = point["bucketCounts"].as_array().unwrap(); + assert_eq!( + counts.len(), + bounds.len() + 1, + "OTLP requires one more bucket count than bounds (+Inf)" + ); + assert_eq!(point["count"], "5"); + } + + #[test] + fn gauges_have_no_temporality_or_start_time() { + let families = vec![MetricFamily { + name: "http.server.active_requests".to_string(), + unit: "{request}", + points: vec![MetricPoint::Gauge { + value: 3.0, + attributes: vec![], + }], + }]; + let payload = encode_metrics(&families, "preloop", "inst", "0.2.0", 0); + let metric = &payload["resourceMetrics"][0]["scopeMetrics"][0]["metrics"][0]; + assert!(metric["gauge"]["dataPoints"][0]["asDouble"] == 3.0); + assert!(metric["gauge"]["aggregationTemporality"].is_null()); + } +} diff --git a/crates/preloop-observability/src/lib.rs b/crates/preloop-observability/src/lib.rs index a82e9c14..589cfdfc 100644 --- a/crates/preloop-observability/src/lib.rs +++ b/crates/preloop-observability/src/lib.rs @@ -443,12 +443,16 @@ impl Observability { pub fn from_config(config: ObservabilityConfig) -> (Self, ObservabilityRuntime) { let is_noop = !config.otlp_enabled; // Absent endpoint spawns nothing at all — no worker, no socket. + // The registry is shared with the worker so metric export scrapes the + // same instruments `/metrics` renders — one source, never two. + let metrics = Arc::new(metrics::MetricsRegistry::default()); let exporter = export::spawn( config.otel_endpoint_raw(), config.otel_headers_raw(), &config.service_name, &config.instance_id, &config.service_version, + Some(metrics.clone()), ) .map(|(exporter, _health)| exporter); let handle = Self { @@ -456,7 +460,7 @@ impl Observability { config: Arc::new(config), heartbeat: TaskHeartbeat::default(), limits: LimitRegistry::default(), - metrics: Arc::new(metrics::MetricsRegistry::default()), + metrics, vm_registry: Arc::new(vm_telemetry::VmTelemetryRegistry::default()), exporter, is_noop, @@ -504,16 +508,43 @@ impl Observability { severity: &'static str, body: impl Into, attributes: Vec<(String, String)>, + ) { + self.export_log_in_span(severity, body, attributes, None); + } + + /// As [`Observability::export_log`], correlated with a span so a backend + /// can pivot from a log line to the request that produced it. + pub fn export_log_in_span( + &self, + severity: &'static str, + body: impl Into, + attributes: Vec<(String, String)>, + context: Option<&export::SpanContext>, ) { if let Some(exporter) = &self.inner.exporter { exporter.log(export::LogRecord { severity, body: body.into(), attributes, + trace_id: context.map(|c| c.trace_id.clone()), + span_id: context.map(|c| c.span_id.clone()), }); } } + /// Enqueue a completed span. No-op when export is disabled. + pub fn export_span(&self, record: export::SpanRecord) { + if let Some(exporter) = &self.inner.exporter { + exporter.span(record); + } + } + + /// Whether spans are worth building. Lets a caller skip id and timestamp + /// work entirely when nothing would consume the result. + pub fn tracing_enabled(&self) -> bool { + self.inner.exporter.is_some() + } + /// Export health for `/api/v1/status` and `preloop.telemetry.export`. pub fn export_health(&self) -> Option<&Arc> { self.inner.exporter.as_ref().map(|e| e.health()) diff --git a/crates/preloop-observability/src/metrics.rs b/crates/preloop-observability/src/metrics.rs index 5943bcc9..02de0788 100644 --- a/crates/preloop-observability/src/metrics.rs +++ b/crates/preloop-observability/src/metrics.rs @@ -566,3 +566,242 @@ mod tests { assert_eq!(m.series_count(), 1, "1000 distinct IDs must be 1 series"); } } + +// --------------------------------------------------------------------------- +// Structured collection for OTLP export +// --------------------------------------------------------------------------- + +/// One data point, already reduced to bounded attributes. +#[derive(Debug, Clone)] +pub enum MetricPoint { + /// Monotonic counter (OTLP `sum`, cumulative, `isMonotonic: true`). + Sum { + value: f64, + attributes: Vec<(String, String)>, + }, + /// Instantaneous value (OTLP `gauge`). + Gauge { + value: f64, + attributes: Vec<(String, String)>, + }, + /// Explicit-bucket histogram (OTLP `histogram`, cumulative). + /// + /// `bucket_counts` is one longer than `bounds`: OTLP requires the + /// implicit `+Inf` bucket to be present as the final entry. + Histogram { + count: u64, + sum: f64, + bounds: Vec, + bucket_counts: Vec, + attributes: Vec<(String, String)>, + }, +} + +#[derive(Debug, Clone)] +pub struct MetricFamily { + pub name: String, + pub unit: &'static str, + pub points: Vec, +} + +impl Histogram { + /// Cumulative bucket counts plus the implicit `+Inf` bucket OTLP requires. + fn otlp_bucket_counts(&self) -> Vec { + let mut counts: Vec = self.buckets.iter().map(|(_, c)| *c).collect(); + counts.push(self.count); + counts + } + + fn bounds(&self) -> Vec { + self.buckets.iter().map(|(le, _)| *le).collect() + } +} + +impl HttpMetrics { + fn collect(&self, out: &mut Vec) { + let durations = self.durations.read(); + if !durations.is_empty() { + out.push(MetricFamily { + name: "http.server.request.duration".to_string(), + unit: "s", + points: durations + .iter() + .map(|(labels, hist)| MetricPoint::Histogram { + count: hist.count, + sum: hist.sum, + bounds: hist.bounds(), + bucket_counts: hist.otlp_bucket_counts(), + attributes: vec![ + ("http.request.method".to_string(), labels.method.clone()), + ("http.route".to_string(), labels.route.clone()), + ("preloop.surface".to_string(), labels.surface.clone()), + ( + "http.response.status_class".to_string(), + labels.status_class.clone(), + ), + ], + }) + .collect(), + }); + } + let active = self.active.read(); + if !active.is_empty() { + out.push(MetricFamily { + name: "http.server.active_requests".to_string(), + unit: "{request}", + points: active + .iter() + .map(|(labels, value)| MetricPoint::Gauge { + value: *value as f64, + attributes: vec![ + ("http.request.method".to_string(), labels.method.clone()), + ("http.route".to_string(), labels.route.clone()), + ("preloop.surface".to_string(), labels.surface.clone()), + ], + }) + .collect(), + }); + } + } +} + +impl StoreMetrics { + fn collect(&self, out: &mut Vec) { + let durations = self.durations.read(); + if !durations.is_empty() { + out.push(MetricFamily { + name: "preloop.store.operation.duration".to_string(), + unit: "s", + points: durations + .iter() + .map(|(labels, hist)| MetricPoint::Histogram { + count: hist.count, + sum: hist.sum, + bounds: hist.bounds(), + bucket_counts: hist.otlp_bucket_counts(), + attributes: vec![ + ("db.system".to_string(), labels.backend.clone()), + ("preloop.operation".to_string(), labels.operation.clone()), + ("preloop.outcome".to_string(), labels.outcome.clone()), + ], + }) + .collect(), + }); + } + let failures = self.consecutive_failures.read(); + if !failures.is_empty() { + out.push(MetricFamily { + name: "preloop.store.consecutive_failures".to_string(), + unit: "{failure}", + points: failures + .iter() + .map(|(backend, value)| MetricPoint::Gauge { + value: *value as f64, + attributes: vec![("db.system".to_string(), backend.clone())], + }) + .collect(), + }); + } + } +} + +impl LifecycleMetrics { + fn collect(&self, out: &mut Vec) { + let completed = self.job_completed.read(); + if !completed.is_empty() { + out.push(MetricFamily { + name: "preloop.job.completed".to_string(), + unit: "{job}", + points: completed + .iter() + .map(|(labels, value)| MetricPoint::Sum { + value: *value as f64, + attributes: vec![ + ("preloop.conclusion".to_string(), labels.conclusion.clone()), + ("preloop.reason".to_string(), labels.reason.clone()), + ], + }) + .collect(), + }); + } + let wait = self.queue_wait.read(); + if !wait.is_empty() { + out.push(MetricFamily { + name: "preloop.job.queue.wait".to_string(), + unit: "s", + points: wait + .iter() + .map(|(labels, hist)| MetricPoint::Histogram { + count: hist.count, + sum: hist.sum, + bounds: hist.bounds(), + bucket_counts: hist.otlp_bucket_counts(), + attributes: vec![("preloop.outcome".to_string(), labels.outcome.clone())], + }) + .collect(), + }); + } + let poll = self.broker_poll.read(); + if !poll.is_empty() { + out.push(MetricFamily { + name: "preloop.broker.poll".to_string(), + unit: "{poll}", + points: poll + .iter() + .map(|(labels, value)| MetricPoint::Sum { + value: *value as f64, + attributes: vec![("preloop.outcome".to_string(), labels.outcome.clone())], + }) + .collect(), + }); + } + let sessions = self.session_transition.read(); + if !sessions.is_empty() { + out.push(MetricFamily { + name: "preloop.runner.session.transition".to_string(), + unit: "{transition}", + points: sessions + .iter() + .map(|(labels, value)| MetricPoint::Sum { + value: *value as f64, + attributes: vec![ + ("preloop.operation".to_string(), labels.operation.clone()), + ("preloop.reason".to_string(), labels.reason.clone()), + ], + }) + .collect(), + }); + } + let concurrency = self.concurrency_decision.read(); + if !concurrency.is_empty() { + out.push(MetricFamily { + name: "preloop.concurrency.decision".to_string(), + unit: "{decision}", + points: concurrency + .iter() + .map(|(labels, value)| MetricPoint::Sum { + value: *value as f64, + attributes: vec![ + ("preloop.queue_mode".to_string(), labels.queue_mode.clone()), + ("preloop.action".to_string(), labels.action.clone()), + ], + }) + .collect(), + }); + } + } +} + +impl MetricsRegistry { + /// Snapshot every instrument as OTLP-ready families. + /// + /// Read-only: takes each sub-registry's read lock in turn and never holds + /// two at once, so a scrape cannot deadlock against a recording caller. + pub fn collect(&self) -> Vec { + let mut families = Vec::new(); + self.http.collect(&mut families); + self.store.collect(&mut families); + self.lifecycle.collect(&mut families); + families + } +} diff --git a/crates/preloop-runner-server/src/http_metrics.rs b/crates/preloop-runner-server/src/http_metrics.rs index 9dd9b28d..2283b8e4 100644 --- a/crates/preloop-runner-server/src/http_metrics.rs +++ b/crates/preloop-runner-server/src/http_metrics.rs @@ -56,6 +56,20 @@ pub async fn http_metrics_middleware( shared.state.observability.metrics().http.inc_active(lbl); } + // Adopt an inbound W3C trace so a caller's trace continues through the + // control plane; otherwise start a root. Health and metrics probes are + // suppressed from trace export per the signal policy — they would swamp + // the trace store and tell an operator nothing. + let traced = shared.state.observability.tracing_enabled() && surface != "public"; + let span_context = traced.then(|| { + preloop_observability::export::SpanContext::from_traceparent( + req.headers() + .get("traceparent") + .and_then(|value| value.to_str().ok()), + ) + }); + let span_start = preloop_observability::export::now_nanos(); + let start = Instant::now(); let res = next.run(req).await; let elapsed = start.elapsed(); @@ -66,38 +80,39 @@ pub async fn http_metrics_middleware( if let Some(lbl) = labels { let mut lbl = lbl; lbl.status_class = sc.clone(); - // Record duration only for non-live_logs + let metrics = shared.state.observability.metrics(); + metrics.http.observe_duration(lbl.clone(), elapsed); + metrics.http.dec_active(&lbl); + } + + if let Some(context) = span_context { + // Attributes are allowlisted, never derived from the raw URI: the + // route is the matched template and the surface is a finite set. + let attributes = vec![ + ("http.request.method".to_string(), method.clone()), + ("http.route".to_string(), route.clone()), + ("preloop.surface".to_string(), surface.clone()), + ("http.response.status_code".to_string(), status.to_string()), + ]; shared .state .observability - .metrics() - .http - .observe_duration(lbl.clone(), elapsed); - shared.state.observability.metrics().http.dec_active(&lbl); + .export_span(preloop_observability::export::SpanRecord { + context, + name: format!("{method} {route}"), + start_nanos: span_start, + end_nanos: preloop_observability::export::now_nanos(), + // Only 5xx is the server's fault; a 4xx is the caller's and + // marking it Error would make every unauthenticated probe + // look like an outage. + status: if status >= 500 { + preloop_observability::export::SpanStatus::Error + } else { + preloop_observability::export::SpanStatus::Unset + }, + attributes, + }); } - // Safe span: method + route template + surface + status, no headers/body/query. - // Use `tracing::info_span!` so it appears in logs when RUST_LOG includes it, - // but filtered at DEBUG by default (poll/renew are DEBUG). - let span = tracing::info_span!( - "http.request", - http.method = %method, - http.route = %route, - http.surface = %surface, - http.status_code = status, - http.status_class = %sc, - otel.kind = "server", - ); - // Attach span to response for trace correlation; the span itself is not - // entered for the handler thread beyond this point (no await while held). - - // Also emit a counter for broker poll outcomes — the control plane's - // poll is a long-poll that returns job|empty|cancel|error. That is - // already counted via the HTTP histogram, but the plan also wants - // `preloop.broker.poll{outcome}` to distinguish empty vs error. - // For now we just log at DEBUG; the dedicated broker counter is wired - // in the broker module itself (Step 4 lifecycle). - let _ = span; - res } From 4faa700fd9516795f996dc61fe9197ea0bb9222b Mon Sep 17 00:00:00 2001 From: Bill Njoroge Date: Thu, 20 Aug 2026 19:17:18 -0400 Subject: [PATCH 22/22] fix(server): keep the reason prose on the log and classify external-host failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Load testing exposed both halves of this. A sustained run produced 21 `reason="unrecognized"` job completions with no way to find out what they were: the previous change claimed the full message stayed on the log record, but only the bounded code was ever attached, so the prose was unrecoverable. Attach it as `reason.detail`. Logs are not a label space, and without it an `unrecognized` classification is a dead end. With the detail visible the path was obvious — a second never-claimable sentence the classifier did not match: no windows runner is registered with this server, so `runs-on: windows-latest` cannot be scheduled built at runtime_scheduling.rs with the platform interpolated. It is a distinct condition from the starvation sweep and gets its own code rather than folding into `no_runner`: the sweep means "no matching runner appeared within the grace window", which more capacity fixes, while this means the server has no runner of that platform class at all and never will until one is registered. Matching is on the invariant phrase, so any interpolated platform classifies. After the fix a mixed load of 40 workflow submits and 640 reads produced only `no_runner` (56) and `no_platform_runner` (8), with zero `unrecognized`, four route templates and two surfaces. Entire-Checkpoint: 01M0GQDWZHCJVWQ5TQB0XEETTB --- crates/preloop-runner-server/src/state.rs | 62 ++++++++++++++++++++--- 1 file changed, 55 insertions(+), 7 deletions(-) diff --git a/crates/preloop-runner-server/src/state.rs b/crates/preloop-runner-server/src/state.rs index a0b10bd1..b62e2e5d 100644 --- a/crates/preloop-runner-server/src/state.rs +++ b/crates/preloop-runner-server/src/state.rs @@ -597,8 +597,19 @@ fn bounded_termination_reason(value: &str) -> &'static str { "startup_orphan" => return "startup_orphan", _ => {} } - // Prose paths — match on the stable leading phrase, never the whole - // string, so an interpolated label cannot change the classification. + // Prose paths — match on the invariant phrase, never the whole string, so + // an interpolated label or platform cannot change the classification. + // + // Two distinct never-claimable conditions, and conflating them would hide + // the difference between "wait or add capacity" and "this will never work + // until you register that platform": + // - the starvation sweep, which fires after a grace window; + // - the external-host check, where the server has no runner of that + // platform class at all (`no {platform} runner is registered with + // this server, so `runs-on: …` cannot be scheduled`). + if value.contains("runner is registered with this server") { + return "no_platform_runner"; + } if value.starts_with("no runner is registered for") { return "no_runner"; } @@ -938,11 +949,22 @@ impl AppState { // separate JobCompleted event; naming both `job.completed` // conflated two distinct records in the log stream. "job.status.terminal", - vec![ - ("event.name".to_string(), "job.status.terminal".to_string()), - ("conclusion".to_string(), conclusion.to_string()), - ("reason".to_string(), bounded_reason.to_string()), - ], + { + let mut attributes = vec![ + ("event.name".to_string(), "job.status.terminal".to_string()), + ("conclusion".to_string(), conclusion.to_string()), + ("reason".to_string(), bounded_reason.to_string()), + ]; + // The bounded code is the metric dimension; the prose + // is what an operator actually needs to act. Logs may + // carry it (they are not a label space), and without + // it an `unrecognized` classification is a dead end — + // you cannot tell which path produced it. + if let Some(detail) = reason.as_deref() { + attributes.push(("reason.detail".to_string(), detail.to_string())); + } + attributes + }, ); } NdjsonEvent::JobCompleted { status, .. } if status.is_terminal() => { @@ -1441,6 +1463,32 @@ mod termination_reason_tests { assert!(!bounded.contains("attacker")); } + #[test] + fn external_host_prose_is_its_own_code() { + // `{platform}` is interpolated, so match the invariant phrase. + for platform in ["windows", "macos", "freebsd-13"] { + let prose = format!( + "no {platform} runner is registered with this server, so \ + `runs-on: {platform}-latest` cannot be scheduled" + ); + assert_eq!( + bounded_termination_reason(&prose), + "no_platform_runner", + "{platform} must classify distinctly from the starvation sweep" + ); + } + } + + #[test] + fn platform_and_starvation_do_not_collide() { + let starved = "no runner is registered for `runs-on: self-hosted, Linux, ARM64` and none \ + appeared within 120s, so the job cannot be scheduled"; + let platform = "no windows runner is registered with this server, so \ + `runs-on: windows-latest` cannot be scheduled"; + assert_eq!(bounded_termination_reason(starved), "no_runner"); + assert_eq!(bounded_termination_reason(platform), "no_platform_runner"); + } + #[test] fn unknown_prose_is_bounded_not_passed_through() { let bounded = bounded_termination_reason("something entirely new happened with id-99999");