From af5dda6418399c7809fd445d5b26c381d4f04d81 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Sun, 6 Sep 2026 00:22:24 +0100 Subject: [PATCH 1/7] docs(openspec): propose the metrics-graph change --- openspec/changes/metrics-graph/.openspec.yaml | 2 + openspec/changes/metrics-graph/design.md | 68 ++++++++ openspec/changes/metrics-graph/proposal.md | 33 ++++ .../metrics-graph/specs/daemon-api/spec.md | 38 +++++ .../specs/engine-activity/spec.md | 38 +++++ .../metrics-graph/specs/fleet-client/spec.md | 148 ++++++++++++++++++ .../specs/remote-metrics-bar-format/spec.md | 107 +++++++++++++ .../metrics-graph/specs/remote-stats/spec.md | 66 ++++++++ openspec/changes/metrics-graph/tasks.md | 42 +++++ 9 files changed, 542 insertions(+) create mode 100644 openspec/changes/metrics-graph/.openspec.yaml create mode 100644 openspec/changes/metrics-graph/design.md create mode 100644 openspec/changes/metrics-graph/proposal.md create mode 100644 openspec/changes/metrics-graph/specs/daemon-api/spec.md create mode 100644 openspec/changes/metrics-graph/specs/engine-activity/spec.md create mode 100644 openspec/changes/metrics-graph/specs/fleet-client/spec.md create mode 100644 openspec/changes/metrics-graph/specs/remote-metrics-bar-format/spec.md create mode 100644 openspec/changes/metrics-graph/specs/remote-stats/spec.md create mode 100644 openspec/changes/metrics-graph/tasks.md diff --git a/openspec/changes/metrics-graph/.openspec.yaml b/openspec/changes/metrics-graph/.openspec.yaml new file mode 100644 index 0000000..1a62d62 --- /dev/null +++ b/openspec/changes/metrics-graph/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-09-06 diff --git a/openspec/changes/metrics-graph/design.md b/openspec/changes/metrics-graph/design.md new file mode 100644 index 0000000..25f82ce --- /dev/null +++ b/openspec/changes/metrics-graph/design.md @@ -0,0 +1,68 @@ +## Context + +All three metrics surfaces (`remote metrics`, `fleet metrics`, the fleet dashboard) render a point-in-time snapshot; no history is kept anywhere, client or server. The daemon already runs a background sampler (`SampleActivity`, `internal/daemon/activity.go`) that ticks at `DefaultSampleInterval` (15s) while an engine runs, but it reads only the engine's token counters and only when a scrape target is known. System stats (CPU/RAM/GPU via `metrics.Collector.System`) are collected on demand inside `Daemon.Metrics()`. The remote stats Lambda relays the daemon's `/v1/metrics` reply over an SSM curl and copies fields into its own reply — and SSM command output is truncated at 4KB. + +## Goals / Non-Goals + +**Goals:** +- A 0–100% sparkline of the last 10 minutes per resource series on all three surfaces, selectable as `bar` (the default), with the previous filled-bar drawing preserved as `gauge`. +- History owned by the daemon so every client — one-shot, watch, dashboard, remote relay — sees the same window from a single read. + +**Non-Goals:** +- Persisting history to disk, or longer than a 10-minute window. +- Charting non-percentile series (token counters stay as plain lines; they have no 0–100% axis). +- A per-node format choice in the dashboard, or charting in the dashboard's detail view. + +## Decisions + +### 1. The daemon owns the history; clients never accumulate + +A ring buffer in the daemon, exposed on `/v1/metrics`, beats client-side accumulation: a one-shot `--format=bar` shows real history on first read, the dashboard needs no per-client buffer, and every client sees the same window. Client accumulation was rejected because a one-shot would degenerate to a single point and each client would show a different window. + +### 2. The sampler takes a system reading each tick + +`SampleActivity` already ticks at 15s while the engine runs. Each tick additionally runs the system collector and appends one reading to the buffer. This is deliberately independent of the counter scrape's gates: system figures come from host commands, not from the engine's metrics endpoint, so a scrape target is not required — only the engine being running (a stopped engine has no utilisation to chart, matching the bar format's existing behaviour). + +A failed system reading records no sample and reports no error — the same "a failed sample is a non-observation" rule the counter sampler follows, and the on-request collection keeps its own error reporting. + +Retention hooks follow the existing engine-boundary hooks in `StartEngine`: the buffer is cleared there alongside `sample.forget()` (one engine's readings never meet the next engine's), and nothing clears it on stop, so the history persists across a stop exactly like the last-active record. + +### 3. Samples store percentages, not raw figures + +Each sample carries the time (unix seconds) and, per series, the 0–100% figure the sparkline plots: CPU utilisation, RAM used/total, and per-GPU utilisation and memory. Storing raw bytes would double the wire size for nothing the graph uses, and the 4KB SSM relay budget makes the difference concrete: 40 samples (10 min at 15s) of percentages is roughly 2–2.5KB; raw memory figures push it toward the truncation limit. The current reading keeps its existing raw shape; only history is percentage-form. + +The Go shape lives in `internal/metrics` (the shared stats dialect): a `History` slice on `Stats`, each sample with `time`, `cpu`, `mem`, and `gpus` (index, util, mem), all omitted when absent — the existing absence-not-zero convention. + +### 4. Rendering: eight block glyphs, max-pooled, last point coloured + +The sparkline maps each value to one of eight Unicode block elements (U+2581–U+2588). Where the window holds more samples than the draw width, each column takes the **maximum** of its pool: utilisation thresholds are about peaks, and a max keeps a red spike visible that an average would smooth away. The final glyph takes the bar format's green/yellow/red colour by threshold; earlier glyphs use the terminal default. + +The full view draws one glyph per sample (up to the 40-column window); the 42-column dashboard tile downsamples to fit, which keeps both formats at one line per series — the tile layout does not change. + +### 5. No-history fallback is the gauge drawing + +Where a daemon reports no history (predates the feature, or an engine with no reading yet), bar format draws the current reading in the gauge's filled style. Consequence worth stating: a fleet mixed with older daemons renders those nodes exactly as it does today, per node, with no client-side version detection. + +### 6. Code rename: the old `renderBar` becomes the gauge renderer + +`bar` now names the sparkline format, so `renderBar`/`renderStatBars` are renamed to their gauge role and the sparkline paths take the `bar` names. `--format` on both metrics commands accepts `bar|gauge|table|json`; the dashboard carries a board-wide format state toggled by `g`, opening in bar. + +### 7. The Lambda relays the field verbatim + +`shared/daemon.ts` and `shared/stats.ts` gain the history types, and the stats Lambda copies the field through unchanged — the same treatment as `lastActiveAt`/`idleSeconds`: the daemon decides what counts, the relay does not reshape it. + +## Risks / Trade-offs + +- [SSM output truncation at 4KB] → samples are percentage-form and the window is fixed at 40; the reply stays well under the limit. If the window or sample shape ever grows, the daemon-side size is the place to cap it, not the client. +- [Unconditional 15s system sampling while an engine runs] → one set of host commands (`nvidia-smi`/`vmstat` or `top`+`vm_stat`) every 15s even when nobody is watching. Small and bounded; the same commands already run on every metrics request, and the dashboard polls those every 2s today. +- [Default output changes for existing users] → `--format=gauge` restores the previous drawing; the release notes name it. +- [Remote environments gain history only after the on-instance daemon is new] → until the next boot/redeploy they render the gauge fallback; nothing breaks. +- [Max-pooling can overstate a spiky series' average level] → acceptable: the graph's job is to show pressure, and the trailing percentage is always the exact latest value. + +## Migration Plan + +Single release; the API change is additive (new field on `/v1/metrics`), and older daemons keep working through the per-node fallback. Rollback is a revert. No data migration. + +## Open Questions + +None — the window (10 min at the 15s cadence), the rename target (`gauge`), the toggle key (`g`), and the stopped-engine behaviour (retained buffer) were settled with the user. diff --git a/openspec/changes/metrics-graph/proposal.md b/openspec/changes/metrics-graph/proposal.md new file mode 100644 index 0000000..50c094e --- /dev/null +++ b/openspec/changes/metrics-graph/proposal.md @@ -0,0 +1,33 @@ +## Why + +`spinloop remote metrics`, `spinloop fleet metrics`, and the fleet dashboard all draw utilisation as an instant block bar. That answers "how full is it right now" but nothing about how the level has been moving, which is what an operator actually asks when they watch an engine. None of the three surfaces can show a 0–100% series over time. + +## What Changes + +- **BREAKING**: `--format=bar` (the default) now draws a sparkline of recent samples per series — `CPU ▁▂▃▅▇▆▅▃▄▅▇ 62%` — instead of an instant filled bar. The previous filled-bar drawing is kept under a new name, `--format=gauge`. +- The daemon retains a rolling 10-minute history of system readings (CPU, RAM, each GPU's utilisation and memory), sampled at the existing 15-second sampler cadence while an engine runs. The history is exposed on `/v1/metrics`, persists across an engine stop, and is cleared when the next engine starts. +- The remote stats Lambda relays the daemon's history in its reply, so `spinloop remote metrics` gets the same view through the control plane. +- The fleet dashboard gains a `g` key that toggles every tile between bar and gauge; tiles open in bar. +- JSON output gains the history field (additive). `docs/openapi.yaml` gains it on the metrics response. +- Where no history is available (an older daemon), the bar format draws the current reading in the gauge style, so a pre-history daemon renders exactly as it does today. + +## Capabilities + +### New Capabilities + +(None — every behaviour change lands in an existing capability.) + +### Modified Capabilities + +- `remote-metrics-bar-format`: `bar` becomes the history sparkline (its drawing, its colour rule, and its behaviour for a stopped engine); the previous drawing is added as the `gauge` format; the history source, window, and no-history fallback are specified. +- `remote-stats`: the format list gains `gauge`; the report carries the daemon's history where the control plane relays it. +- `fleet-client`: `fleet metrics` accepts `gauge`; the dashboard's panels default to the bar format and a `g` key toggles bar and gauge. +- `daemon-api`: the metrics endpoint includes the retained history of system readings. +- `engine-activity`: the background sampler takes a system reading on each tick while an engine runs, feeding the retained history. + +## Impact + +- Go: `internal/metrics` (stats shape), `internal/daemon` (sampler, ring buffer, `/v1/metrics`), `cmd/spinloop` (renderers, `--format` in remote.go and fleet.go, dashboard model and render), `docs/openapi.yaml`, and tests across those packages. +- TypeScript (`remote/`): shared types and the stats Lambda relay the history field; covered by the pnpm suite. +- CLI: `--format=bar` output changes for existing users, and so does the default output of both metrics commands; `--format=gauge` restores the previous drawing. +- Remote environments carry history only once the on-instance daemon is new (next boot or redeploy); until then they render the gauge fallback. diff --git a/openspec/changes/metrics-graph/specs/daemon-api/spec.md b/openspec/changes/metrics-graph/specs/daemon-api/spec.md new file mode 100644 index 0000000..5ed7d93 --- /dev/null +++ b/openspec/changes/metrics-graph/specs/daemon-api/spec.md @@ -0,0 +1,38 @@ +## ADDED Requirements + +### Requirement: Metrics endpoint reports the system-reading history + +The metrics endpoint SHALL include, alongside the current readings, the +history of system readings the daemon retains: one entry per sampler tick +while an engine was running, each carrying its time and that tick's CPU, +memory, and per-GPU utilisation and memory readings. The history SHALL cover +at most the last 10 minutes at the sampler's cadence. It SHALL persist when +the engine stops — the readings taken before the stop remain, so a caller can +see what the engine was doing until it stopped — and SHALL be cleared when +the next engine starts, so one engine's readings are never reported against +another. Where no engine has run in this daemon's life, or no reading has been +taken, the field SHALL be omitted rather than empty. + +#### Scenario: History grows while the engine runs + +- **WHEN** an engine has been running for several sampler ticks and a metrics + request is made +- **THEN** the response includes one history entry per tick, each stamped with + its time, covering up to the last 10 minutes + +#### Scenario: A stopped engine still reports its history + +- **WHEN** a metrics request is made after the engine has been stopped +- **THEN** the response still includes the readings taken before the stop, + though the current running-engine figures are omitted + +#### Scenario: A new engine clears the previous history + +- **WHEN** an engine is stopped and a later engine is started, and a metrics + request is made +- **THEN** the history holds only the later engine's readings + +#### Scenario: No history yet is absent, not empty + +- **WHEN** a metrics request is made on a daemon that has never run an engine +- **THEN** the response carries no history field diff --git a/openspec/changes/metrics-graph/specs/engine-activity/spec.md b/openspec/changes/metrics-graph/specs/engine-activity/spec.md new file mode 100644 index 0000000..cbd9fdc --- /dev/null +++ b/openspec/changes/metrics-graph/specs/engine-activity/spec.md @@ -0,0 +1,38 @@ +## ADDED Requirements + +### Requirement: System readings for the retained history + +While an engine is running, the sampler SHALL take one system reading — the +host's CPU, memory, and GPU figures — on each tick at its own interval, +independently of any request to the control API and independently of whether a +scrape target for the engine's counters is known. Each reading SHALL be +retained in the history the metrics endpoint reports, for at most the last +10 minutes. A failed system reading SHALL record no sample for its tick and +SHALL NOT be reported as an error: the on-request collection keeps its own +error reporting, and a transient sampling failure is neither data nor a +condition worth surfacing on every tick. + +#### Scenario: System readings happen without being asked + +- **WHEN** an engine is running and no client calls the control API +- **THEN** the daemon still takes a system reading on each sampler tick and + retains it + +#### Scenario: System readings do not depend on a scrape target + +- **WHEN** the running engine exposes no metrics endpoint to scrape +- **THEN** the system readings are still taken and retained, since they come + from the host, not from the engine + +#### Scenario: A failed system reading records nothing + +- **WHEN** a system reading fails on a tick because a host command is missing + or fails +- **THEN** no sample is recorded for that tick and no error is reported for + it + +#### Scenario: Reading stops with the engine, retention does not end + +- **WHEN** the engine is stopped +- **THEN** no further system readings are taken, and the readings taken before + the stop remain retained until the next engine starts diff --git a/openspec/changes/metrics-graph/specs/fleet-client/spec.md b/openspec/changes/metrics-graph/specs/fleet-client/spec.md new file mode 100644 index 0000000..f97e510 --- /dev/null +++ b/openspec/changes/metrics-graph/specs/fleet-client/spec.md @@ -0,0 +1,148 @@ +## MODIFIED Requirements + +### Requirement: Fleet metrics + +`spinloop fleet metrics` SHALL query every node's metrics endpoint and render +each node's engine and system metrics using the same bar, gauge, table, and +json formats `spinloop remote metrics` provides, selected by `--format`. +Unreachable nodes SHALL be reported as in status rather than omitted. The +command SHALL support a `--watch`/`-w` mode that refreshes on an interval, +clearing and redrawing the screen in place with no scrollback accumulation, +and exiting cleanly on interrupt. + +#### Scenario: Bar format per node + +- **WHEN** `spinloop fleet metrics` runs without `--format` +- **THEN** each reachable node's metrics render in bar format under its name + +#### Scenario: Gauge format per node + +- **WHEN** `spinloop fleet metrics --format=gauge` runs +- **THEN** each reachable node's resource series render in gauge format under + its name + +#### Scenario: JSON aggregates the fleet + +- **WHEN** `spinloop fleet metrics --format=json` runs +- **THEN** the output is valid JSON keyed or labelled by node, including + unreachable nodes with their error + +#### Scenario: Watch redraws in place + +- **WHEN** `spinloop fleet metrics --watch` runs +- **THEN** each refresh clears the screen and redraws the fleet, and Ctrl+C + exits cleanly + +### Requirement: Dashboard panels show the node's metrics + +Each panel SHALL show, for a node that answered the last completed refresh, the +same facts the bar format of `fleet metrics` renders for that node: its state, +what it serves (runner and model when known), how long since it last did work +(with the same labelling rules as the rest of the fleet surfaces), its resource +usage, and its token and request counters. A panel SHALL draw the node's +resource series in the board's current format — bar by default — from the +history the node's daemon reports, falling back per the bar format's no-history +rule where it reports none. A panel SHALL show the answer of the last completed +refresh for that node — not a mix of refreshes and not a stale bar with a fresh +outcome. + +A panel SHALL degrade gracefully when a node answers with fewer facts (no system +stats, no GPUs, an engine that is not running) rather than failing to render. +A node whose answer is a failure — unreachable, unauthorised, a configuration +error — SHALL show its typed outcome and reason in its panel instead of metrics. + +One node SHALL be selected at a time, and the selected panel SHALL be +distinguishable at a glance from the others. Selection SHALL move according to +the grid the dashboard is currently rendering, not the fleet file's flat +order: the up and down keys SHALL move the selection to the tile directly +above or below it, in the same column of the adjacent row; the left and right +keys SHALL move the selection to the adjacent tile in the same row. None of +the four directions SHALL wrap — a move off the grid's edge SHALL leave the +selection where it was. A resize that changes how many tiles fit per row +SHALL be reflected immediately: the next move follows the new grid, not the +one the dashboard was drawing before the resize. + +#### Scenario: A running node's panel updates + +- **WHEN** a node's engine is running and its counters or utilisation change +- **THEN** the node's panel shows the new figures on a subsequent refresh, + without the operator pressing any key + +#### Scenario: A node that reports fewer facts still renders + +- **WHEN** a node's answer carries no GPU or system statistics +- **THEN** its panel shows the facts it has, with the missing ones absent rather + than an error + +#### Scenario: A failing node's panel shows why + +- **WHEN** a node cannot be reached, or its daemon rejects the client +- **THEN** its panel shows the node's outcome and reason, and the other panels + are unaffected + +#### Scenario: Selection moves and is visible + +- **WHEN** the operator moves the selection with the navigation keys +- **THEN** the selection moves according to the currently rendered grid and + the selected panel is visibly distinct from the others + +#### Scenario: Up and down move by grid row + +- **WHEN** the dashboard renders more than one tile per row and the operator + presses the down key +- **THEN** the selection moves to the tile directly below it, in the same + column of the next row, rather than to the next tile in the fleet file's + order + +#### Scenario: Left and right move within a row + +- **WHEN** the operator presses the right key +- **THEN** the selection moves to the next tile in the same row +- **AND** pressing the left key from there returns the selection to the tile + it started on + +#### Scenario: Movement clamps at the grid's edges + +- **WHEN** the selection is on the top row and the operator presses up, on the + bottom row and presses down, on the first column and presses left, or on the + last column and presses right +- **THEN** the selection does not move and does not wrap to the opposite edge + +#### Scenario: Down clamps on a short last row + +- **WHEN** the selection is in a column that the last, partially-filled grid + row does not reach, and the operator presses down +- **THEN** the selection moves to the last tile that exists in that row + instead of moving past the end of the fleet + +#### Scenario: A resize changes the grid the arrow keys follow + +- **WHEN** the terminal is resized so the dashboard now fits a different + number of tiles per row, and the operator then presses an arrow key +- **THEN** the selection moves according to the new column count, not the one + in effect before the resize + +## ADDED Requirements + +### Requirement: Dashboard format toggle + +The dashboard SHALL provide a key, `g`, that toggles the resource series of +every panel between bar and gauge. The board SHALL open in bar. The toggle +SHALL be board-wide — one format for every panel — rather than per node, and +the key help line SHALL name it. + +#### Scenario: Pressing the key switches every panel + +- **WHEN** the operator presses `g` on the grid +- **THEN** every panel's resource series redraws in the other format, and + pressing `g` again returns them + +#### Scenario: The board opens in bar + +- **WHEN** the dashboard opens +- **THEN** the panels draw the resource series in bar format + +#### Scenario: The key help names the toggle + +- **WHEN** the dashboard draws its key help line +- **THEN** it names `g` as the format toggle diff --git a/openspec/changes/metrics-graph/specs/remote-metrics-bar-format/spec.md b/openspec/changes/metrics-graph/specs/remote-metrics-bar-format/spec.md new file mode 100644 index 0000000..2451847 --- /dev/null +++ b/openspec/changes/metrics-graph/specs/remote-metrics-bar-format/spec.md @@ -0,0 +1,107 @@ +## MODIFIED Requirements + +### Requirement: Bar format output + +The system SHALL support a `--format=bar` option that renders each resource series as a sparkline drawn from the history the on-instance daemon retains: a left-aligned label, one glyph per sample, and the latest value as a right-aligned percentage. The glyphs SHALL be Unicode block elements of one grade per utilisation level, so a series reads as a line of bars across the window. The series drawn SHALL be the same set the gauge format draws: CPU, RAM, and each GPU's utilisation and memory, with the same per-GPU labelling. + +#### Scenario: Bar format displays CPU utilization + +- **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance that has CPU data and a retained history +- **THEN** the output includes a row labelled "CPU" whose glyphs are the sampled CPU utilisation across the window and whose trailing figure is the latest sample's percentage + +#### Scenario: Bar format displays RAM utilization + +- **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance that has memory data +- **THEN** the output includes a row labelled "RAM" whose glyphs are the sampled used/total memory ratio across the window and whose trailing figure is the latest ratio + +#### Scenario: Bar format displays GPU utilization + +- **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance that has GPU data +- **THEN** the output includes rows labelled "GPU util" and "GPU mem" (or "GPU N util"/"GPU N mem" for multiple GPUs), each drawn from the retained history + +#### Scenario: Bar format header line + +- **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance +- **THEN** the first line shows the environment, state, instance type, and model ID separated by double spaces + +### Requirement: Colour thresholds + +The sparkline's latest point SHALL be colour-coded based on utilization: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%. Every earlier point SHALL appear in the terminal's default colour, so the coloured point is the one to read. + +#### Scenario: Green bar for low utilization + +- **WHEN** the latest sample of a series is 70% +- **THEN** the sparkline's final glyph appears in green and the earlier glyphs appear in the terminal's default colour + +#### Scenario: Yellow bar for high utilization + +- **WHEN** the latest sample of a series is 85% +- **THEN** the sparkline's final glyph appears in yellow + +#### Scenario: Red bar for critical utilization + +- **WHEN** the latest sample of a series is 95% +- **THEN** the sparkline's final glyph appears in red + +### Requirement: Bar format with stopped instance + +When the instance is not running, bar format SHALL show the header line with environment, state, instance type, and model, and — where the daemon's retained history survives the stop — the series drawn from it, ending at the stop. The retained history answers "what was this engine doing until it stopped", the same question the last-active figure answers, and the header already carries the state. When no history is available, the format SHALL fall back to the gauge drawing of the current reading per the no-history rule — which for a stopped engine, whose current reading carries no resource figures, means no resource series at all. When a last-active time is known it SHALL still be shown, in the same place it occupies for a running instance. + +#### Scenario: Stopped instance shows header only + +- **WHEN** the user runs `spinloop remote metrics --format=bar` and the instance is stopped with no retained history and no recorded activity +- **THEN** the output shows the header with state "stopped" and no resource series + +#### Scenario: Stopped instance still reports its last activity + +- **WHEN** the user runs `spinloop remote metrics --format=bar`, the instance is stopped, and a last-active time is known +- **THEN** the output shows the header, the last-active line, and the series drawn from the retained history where one exists + +#### Scenario: Stopped instance shows its history + +- **WHEN** the user runs `spinloop remote metrics --format=bar` and the instance's engine has been stopped after running, with a retained history +- **THEN** the output shows the series as sparklines drawn from the readings taken before the stop, ending at the stop + +## ADDED Requirements + +### Requirement: Gauge format + +The system SHALL support a `--format=gauge` option that renders each resource series as a horizontal progress gauge: a left-aligned label, a filled portion using block characters, an unfilled portion using light shade characters, and a right-aligned percentage value. The gauge draws the current reading only — it carries no history. The series drawn SHALL be CPU, RAM, and each GPU's utilisation and memory, with the same labels the bar format uses. The gauge fill SHALL be colour-coded on the bar format's thresholds: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%, with the colour reset after the filled portion so the unfilled characters and percentage appear in the terminal's default colour. + +#### Scenario: Gauge format displays CPU utilization + +- **WHEN** the user runs `spinloop remote metrics --format=gauge` with a running instance that has CPU data +- **THEN** the output includes a gauge labelled "CPU" with filled and unfilled segments proportional to the current utilization + +#### Scenario: Gauge format displays GPU utilisation + +- **WHEN** the user runs `spinloop remote metrics --format=gauge` with a running instance that has GPU data +- **THEN** the output includes gauges labelled "GPU util" and "GPU mem" (or "GPU N util"/"GPU N mem" for multiple GPUs) + +#### Scenario: Gauge colours the fill + +- **WHEN** a gauge's current value is 95% +- **THEN** its filled segment appears in red, and its unfilled segment and percentage appear in the terminal's default colour + +### Requirement: Bar draws the retained history + +Bar format SHALL draw each series from the history the on-instance daemon retains: readings taken at the sampler's cadence while the engine ran, covering at most the last 10 minutes. The sparkline SHALL show every sample the window holds, downsampled to the draw width where the window holds more samples than the width allows; downsampled points SHALL preserve the window's extremes rather than averaging them away. + +The format SHALL be usable in one-shot mode: the history comes from the daemon, not from the command's own polling, so `--format=bar` without `--watch` draws the same window `--watch` would. + +Where the daemon reports no history — a daemon that predates the feature, or an engine with no reading yet — bar format SHALL draw each series from the current reading alone, in the gauge's filled style, so the default format still shows the current level and a pre-history daemon renders exactly as it does today. + +#### Scenario: One-shot bar shows the daemon's window + +- **WHEN** the user runs `spinloop remote metrics --format=bar` without `--watch` against an engine that has been running +- **THEN** the output shows each series as a sparkline covering up to the last 10 minutes of the daemon's retained samples + +#### Scenario: More samples than width are downsampled + +- **WHEN** the retained window holds more samples than the draw width +- **THEN** the sparkline shows one glyph per column of the width, and a spike inside a downsampled range is still visible rather than smoothed away + +#### Scenario: No history falls back to the gauge drawing + +- **WHEN** the user runs `spinloop remote metrics --format=bar` against a daemon that reports no history +- **THEN** each series is drawn from the current reading in the gauge's filled style diff --git a/openspec/changes/metrics-graph/specs/remote-stats/spec.md b/openspec/changes/metrics-graph/specs/remote-stats/spec.md new file mode 100644 index 0000000..199cddd --- /dev/null +++ b/openspec/changes/metrics-graph/specs/remote-stats/spec.md @@ -0,0 +1,66 @@ +## MODIFIED Requirements + +### Requirement: Tabular display + +The stats output SHALL support four formats via the `--format` flag: `bar` (default), `gauge`, `table`, and `json`. The `bar` format SHALL produce a compact display drawing each resource series as a sparkline of the daemon's retained history, with the latest point colour-coded by utilization level. The `gauge` format SHALL produce a compact display with horizontal progress gauges for the current reading, colour-coded by utilization level. The `table` format SHALL produce a tab-separated key-value table, one line per metric, with the key column left-aligned and values right of it. The `json` format SHALL output the response as a JSON object to standard output. Progress and error messages SHALL go to standard error regardless of format. + +#### Scenario: Clean output + +- **WHEN** the command succeeds +- **THEN** standard output contains only the stats data with no progress or debug lines + +#### Scenario: Default format is bar + +- **WHEN** the user runs `spinloop remote metrics` without `--format` +- **THEN** the output is in bar format + +#### Scenario: Table format is explicit + +- **WHEN** the user runs `spinloop remote metrics --format=table` +- **THEN** the output is in table format + +#### Scenario: Bar format is explicit + +- **WHEN** the user runs `spinloop remote metrics --format=bar` +- **THEN** the output is in bar format, drawing each resource series as a sparkline of the daemon's retained history + +#### Scenario: Gauge format is explicit + +- **WHEN** the user runs `spinloop remote metrics --format=gauge` +- **THEN** the output is in gauge format with progress gauges for the current reading + +#### Scenario: JSON format + +- **WHEN** the user runs `spinloop remote metrics --format=json` +- **THEN** the output is valid JSON containing the instance state, runner, model, GPU info, CPU/RAM usage, and token counts + +#### Scenario: JSON format with cost + +- **WHEN** the user runs `spinloop remote metrics --format=json --cost` +- **THEN** the JSON output includes a cost estimate field + +#### Scenario: Invalid format errors + +- **WHEN** the user runs `spinloop remote metrics --format=csv` +- **THEN** the command exits with an error and usage message + +## ADDED Requirements + +### Requirement: History in the report + +When the on-instance daemon's metrics reply carries a history of system readings, the report SHALL carry it through to the command's output: the `json` format SHALL include the readings, and the `bar` format SHALL draw them. Where the daemon's reply carries no history, the report SHALL omit the field and the `bar` format SHALL fall back per the bar format specification. The control plane's relay of the daemon's reply SHALL NOT alter the readings it carries. + +#### Scenario: JSON carries the daemon's history + +- **WHEN** the instance's daemon reports a retained history and the user runs `spinloop remote metrics --format=json` +- **THEN** the JSON output includes the history's readings + +#### Scenario: Bar draws the relayed history + +- **WHEN** the instance's daemon reports a retained history and the user runs `spinloop remote metrics --format=bar` +- **THEN** each resource series is drawn as a sparkline from the readings the control plane relayed + +#### Scenario: A daemon without history degrades + +- **WHEN** the instance runs a daemon whose reply carries no history and the user runs `spinloop remote metrics` +- **THEN** the report omits the history field and bar format draws the current reading in the gauge's filled style diff --git a/openspec/changes/metrics-graph/tasks.md b/openspec/changes/metrics-graph/tasks.md new file mode 100644 index 0000000..907c524 --- /dev/null +++ b/openspec/changes/metrics-graph/tasks.md @@ -0,0 +1,42 @@ +## 1. Daemon history + +- [ ] 1.1 Add the history shape to `internal/metrics`: per-sample time plus CPU, RAM, and per-GPU percentage readings, and a `History` field on `Stats` omitted when empty +- [ ] 1.2 Add the daemon's ring buffer (10-minute window at the sampler cadence) with append, snapshot, and clear operations +- [ ] 1.3 Take a system reading on each sampler tick while an engine runs (independent of a scrape target), record a sample on success, record nothing on failure +- [ ] 1.4 Clear the buffer on engine start alongside the existing counter/reset hooks; leave it untouched on stop +- [ ] 1.5 Include the buffer's contents in `Daemon.Metrics` so `/v1/metrics` reports the history +- [ ] 1.6 Unit tests: buffer wrap and ordering, sample on success / none on failure, clear on start, persistence across stop, exposure on the metrics reply + +## 2. API contract + +- [ ] 2.1 Add the history field to the metrics response in `docs/openapi.yaml` +- [ ] 2.2 Confirm `openapi_test.go` passes with the new field + +## 3. CLI rendering + +- [ ] 3.1 Add the sparkline renderer: eight block glyphs, max-pool downsampling to the draw width, latest point coloured on the 80/90 thresholds, trailing percentage +- [ ] 3.2 Rename the existing filled-bar renderer to the gauge role and route `--format` on both `remote metrics` and `fleet metrics` across `bar`, `gauge`, `table`, and `json` +- [ ] 3.3 Draw the bar format from the daemon's history for the same series and labels the gauge draws, with the gauge-style fallback where a daemon reports no history +- [ ] 3.4 Keep the bar format drawing the retained history for a stopped engine (ending at the stop), with the existing last-active and header behaviour +- [ ] 3.5 Tests: renderer glyphs, downsampling keeps peaks, colour thresholds on the last point, format validation errors, no-history fallback, stopped-engine output + +## 4. Dashboard + +- [ ] 4.1 Add the board-wide format state and the `g` toggle to the dashboard model, opening in bar, with the key help naming it +- [ ] 4.2 Draw each tile's resource series in the board's current format from the node's history, reusing the fallback for nodes whose daemon reports none +- [ ] 4.3 Tests: toggle switches every tile and back, tiles keep their geometry in both formats, nodes without history fall back + +## 5. Remote relay + +- [ ] 5.1 Add the history types to `remote/lambda/shared/daemon.ts` and `shared/stats.ts` +- [ ] 5.2 Copy the history field through in the stats Lambda, unaltered +- [ ] 5.3 TypeScript tests covering the relay of the field and its absence + +## 6. Documentation + +- [ ] 6.1 Update the command reference under `docs/` for the `bar`/`gauge` formats and the dashboard's `g` key + +## 7. Verification + +- [ ] 7.1 `gofmt`, `go vet ./...`, and `go test ./... -cover` with total coverage at or above 80% +- [ ] 7.2 The `remote/` pnpm suite passes From dd5fd0f84bbf76340084652c8901fca1b75efb54 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Sun, 6 Sep 2026 01:33:00 +0100 Subject: [PATCH 2/7] feat: add the bar format, drawing the daemon's retained metrics history The daemon keeps one system reading per sampler tick for the last ten minutes and reports them on /v1/metrics; the buffer clears on the next engine start and a stopped engine keeps its readings to the stop. remote metrics, fleet metrics and the dashboard draw the readings as a bar, one block glyph per reading with the last point coloured on the 80/90 thresholds; a series with no retained readings falls back to the old drawing, renamed gauge, so a daemon that predates the history renders as it did. The stats Lambda relays the history through unaltered, and the dashboard toggles between the two formats board-wide on g. --- cmd/spinloop/dashboard_detail.go | 5 +- cmd/spinloop/dashboard_model.go | 15 +- cmd/spinloop/dashboard_render.go | 29 +- cmd/spinloop/fleet.go | 29 +- cmd/spinloop/fleet_dashboard_test.go | 128 ++++++++- cmd/spinloop/metrics_render.go | 257 ++++++++++++++++- cmd/spinloop/metrics_render_test.go | 316 +++++++++++++++++++++ cmd/spinloop/remote.go | 82 +++--- docs/commands/fleet.md | 20 +- docs/commands/remote.md | 11 + docs/http-api.md | 9 + docs/internals.md | 7 +- docs/openapi.yaml | 50 ++++ internal/daemon/activity.go | 1 + internal/daemon/daemon.go | 10 +- internal/daemon/history.go | 114 ++++++++ internal/daemon/history_test.go | 351 ++++++++++++++++++++++++ internal/daemon/openapi_test.go | 2 + internal/fleet/remote_node.go | 1 + internal/metrics/metrics.go | 41 +++ internal/remote/remote.go | 7 +- openspec/changes/metrics-graph/tasks.md | 44 +-- remote/lambda/shared/daemon.ts | 9 +- remote/lambda/shared/stats.ts | 35 +++ remote/lambda/stats/index.ts | 4 + remote/test/stats-relay.test.ts | 30 ++ remote/test/stats.test.ts | 35 +++ 27 files changed, 1527 insertions(+), 115 deletions(-) create mode 100644 cmd/spinloop/metrics_render_test.go create mode 100644 internal/daemon/history.go create mode 100644 internal/daemon/history_test.go diff --git a/cmd/spinloop/dashboard_detail.go b/cmd/spinloop/dashboard_detail.go index 9139ea3..fb4c62b 100644 --- a/cmd/spinloop/dashboard_detail.go +++ b/cmd/spinloop/dashboard_detail.go @@ -183,11 +183,12 @@ func (m *dashModel) detailSectionHeights() (metrics, log int) { } // detailNodeLines is the metrics section: the same lines the node's tile draws, -// for the node the view is open on. +// for the node the view is open on, in the board's current format at the full +// view's width. func (m *dashModel) detailNodeLines() []string { e := m.entries[m.cursor] lines, _ := dashNodeView(e.name, m.results[m.cursor], m.actions[m.cursor], - dashNow(), dashStaleAfter(e.kind)) + dashNow(), dashStaleAfter(e.kind), m.gauge, barLineW) return lines } diff --git a/cmd/spinloop/dashboard_model.go b/cmd/spinloop/dashboard_model.go index 39d000e..851eaec 100644 --- a/cmd/spinloop/dashboard_model.go +++ b/cmd/spinloop/dashboard_model.go @@ -71,6 +71,13 @@ type dashModel struct { confirm bool // a stop is waiting on its confirmation statusLine string + // gauge is the board's resource-series format: false draws the bar + // format (the sparkline of each node's retained history), true the gauge + // format (the current reading). Board-wide, toggled by g — one format for + // every panel rather than a choice per node. The zero value opens the + // board in bar. + gauge bool + // send feeds a message back into the program from outside the Update // loop — the in-flight progress of a start, which its call reports from // its own goroutine. It is the tea.Program's Send, safe from any @@ -351,6 +358,10 @@ func (m *dashModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) { if len(m.entries) > 0 && m.actions[m.cursor].verb == "" { m.confirm = true } + case "g": + // The board-wide format toggle: every panel redraws in the other + // format, and g again returns them. + m.gauge = !m.gauge case "r": // A manual refresh is due for every node, cloud or local, // whatever their own deadlines say. @@ -658,7 +669,7 @@ func (m dashModel) View() string { tiles := make([]string, len(m.entries)) for i := range m.entries { tiles[i] = dashTile(m.entries[i].name, m.results[i], i == m.cursor, m.actions[i], - now, dashStaleAfter(m.entries[i].kind)) + now, dashStaleAfter(m.entries[i].kind), m.gauge) } rows := dashGridRows(tiles, dashCols(w)) lo := m.scrollRow @@ -688,7 +699,7 @@ func (m dashModel) headerLine(w int) string { // dashGridKeys is the grid's own key help; the detail view's footer shares // footerLine but names its own keys instead (see dashDetailKeys). -const dashGridKeys = "↑↓←→ move s start a abort x stop r refresh q quit" +const dashGridKeys = "↑↓←→ move s start a abort x stop g format r refresh q quit" // footerLine is the frame's bottom line: the given key help, replaced by the // stop confirmation prompt while one is pending, with the status line and a diff --git a/cmd/spinloop/dashboard_render.go b/cmd/spinloop/dashboard_render.go index 027cfc5..df734cc 100644 --- a/cmd/spinloop/dashboard_render.go +++ b/cmd/spinloop/dashboard_render.go @@ -284,7 +284,7 @@ func dashStaleAfter(kind string) time.Duration { // its age wherever it is drawn, and takes the panel to the unknown tier: a // stale reading is not a wrong reading, but drawing it identically to a // current one is. -func dashNodeView(name string, r fleet.NodeResult, a dashAction, now time.Time, staleAfter time.Duration) ([]string, dashHealthTier) { +func dashNodeView(name string, r fleet.NodeResult, a dashAction, now time.Time, staleAfter time.Duration, gauge bool, lineW int) ([]string, dashHealthTier) { age := dashReadingAge(r, now, staleAfter) var b strings.Builder switch { @@ -297,7 +297,7 @@ func dashNodeView(name string, r fleet.NodeResult, a dashAction, now time.Time, if s := r.Metrics.State; s != "" { fmt.Fprintln(&b, dashStateLine(s, r.Metrics)+age) } - dashTileReportBody(&b, r.Metrics, true) + dashTileReportBody(&b, r.Metrics, true, gauge, lineW) } case r.Outcome == "": fmt.Fprintf(&b, "%s\nwaiting for first refresh…\n", name) @@ -308,7 +308,14 @@ func dashNodeView(name string, r fleet.NodeResult, a dashAction, now time.Time, } default: fmt.Fprintf(&b, "%s %s%s\n", name, dashStateLine(r.Metrics.State, r.Metrics), age) - dashTileReportBody(&b, r.Metrics, r.Metrics.State == "running") + // In bar the series also come from the retained history, which + // survives a stop: a stopped node with a window still has series to + // draw, and its current reading simply carries no fallback for them. + resources := r.Metrics.State == "running" + if !gauge && len(r.Metrics.History) > 0 { + resources = true + } + dashTileReportBody(&b, r.Metrics, resources, gauge, lineW) } lines := strings.Split(b.String(), "\n") lines = lines[:len(lines)-1] // the trailing newline splits an extra empty piece @@ -373,8 +380,8 @@ func dashHealthTierFor(r fleet.NodeResult, a dashAction, stale bool) dashHealthT // tile's fixed height and clipped to its fixed width, with the first line // drawn as the header bar — tile-only, not part of dashNodeView, so the detail // view (which draws the same lines full-screen) keeps a plain first line. -func dashTileContent(name string, r fleet.NodeResult, a dashAction, now time.Time, staleAfter time.Duration) string { - lines, tier := dashNodeView(name, r, a, now, staleAfter) +func dashTileContent(name string, r fleet.NodeResult, a dashAction, now time.Time, staleAfter time.Duration, gauge bool) string { + lines, tier := dashNodeView(name, r, a, now, staleAfter, gauge, dashBarLineW) if len(lines) == 0 { lines = []string{""} } @@ -426,19 +433,23 @@ func dashStateLine(state string, m metrics.Stats) string { // answer has it. A settled tile gates the resources block on the node being // running; the in-flight tile draws whatever there is, because a boot half // done has some of the facts and not the rest. -func dashTileReportBody(w io.Writer, m metrics.Stats, resources bool) { +func dashTileReportBody(w io.Writer, m metrics.Stats, resources bool, gauge bool, lineW int) { if line := dashTileServingLine(m); line != "" { fmt.Fprintln(w, line) } renderLastActiveIndented(w, m.LastActiveAt, m.IdleSeconds) if resources { - renderStatBars(w, m.CPU, m.Memory, m.GPUs) + if gauge { + renderStatGauges(w, m.CPU, m.Memory, m.GPUs) + } else { + renderStatBars(w, m.CPU, m.Memory, m.GPUs, m.History, lineW) + } renderTokenLines(w, m.Tokens) } } // dashTile frames one panel; the selected one carries a lit border. -func dashTile(name string, r fleet.NodeResult, selected bool, a dashAction, now time.Time, staleAfter time.Duration) string { +func dashTile(name string, r fleet.NodeResult, selected bool, a dashAction, now time.Time, staleAfter time.Duration, gauge bool) string { style := lipgloss.NewStyle(). Width(dashTileW).Height(dashTileH). Border(lipgloss.RoundedBorder()) @@ -447,7 +458,7 @@ func dashTile(name string, r fleet.NodeResult, selected bool, a dashAction, now } else { style = style.BorderForeground(lipgloss.Color("240")) } - return style.Render(dashTileContent(name, r, a, now, staleAfter)) + return style.Render(dashTileContent(name, r, a, now, staleAfter, gauge)) } // dashGridRows lays tiles out left to right, top to bottom, in fleet-file diff --git a/cmd/spinloop/fleet.go b/cmd/spinloop/fleet.go index 3719ad3..c1c4de9 100644 --- a/cmd/spinloop/fleet.go +++ b/cmd/spinloop/fleet.go @@ -118,8 +118,8 @@ func fleetMetricsCmd() *cobra.Command { SilenceUsage: true, RunE: func(c *cobra.Command, _ []string) error { resolve(c) - if format != "bar" && format != "table" && format != "json" { - return fmt.Errorf("--format must be \"bar\", \"table\", or \"json\", got %q", format) + if err := validateMetricsFormat(format); err != nil { + return err } cfg, err := fleet.Resolve(path) if err != nil { @@ -134,7 +134,7 @@ func fleetMetricsCmd() *cobra.Command { } fs := c.Flags() fs.StringVarP(&path, "fleet", "f", "", fleetFileUsage) - fs.StringVar(&format, "format", "bar", "output format: bar (default), table or json") + fs.StringVar(&format, "format", "bar", "output format: bar (default), gauge, table or json") fs.BoolVarP(&watch, "watch", "w", false, "redraw the fleet every 60 seconds") c.ValidArgsFunction = noPositionals compRegister(c, "fleet", compFiles) @@ -205,13 +205,24 @@ func renderFleetMetrics(w io.Writer, results []fleet.NodeResult, format string) // before theirs: a node whose engine has stopped still has a useful // answer to "when did this last do anything?". renderLastActiveIndented(w, stats.LastActiveAt, stats.IdleSeconds) - if stats.State != "running" { - continue - } - if format == "bar" { - renderStatBars(w, stats.CPU, stats.Memory, stats.GPUs) + switch format { + case "bar": + // No state gate, for the same reason the remote bar format has + // none: a stopped node's retained history says what its engine + // was doing until it stopped, and a stopped node's current + // reading carries no figures for it to fall back on. + renderStatBars(w, stats.CPU, stats.Memory, stats.GPUs, stats.History, barLineW) renderTokenLines(w, stats.Tokens) - } else { + case "gauge": + if stats.State != "running" { + continue + } + renderStatGauges(w, stats.CPU, stats.Memory, stats.GPUs) + renderTokenLines(w, stats.Tokens) + default: + if stats.State != "running" { + continue + } renderTokenLines(w, stats.Tokens) renderGPUTable(w, stats.GPUs) renderCPUMemTable(w, stats.CPU, stats.Memory) diff --git a/cmd/spinloop/fleet_dashboard_test.go b/cmd/spinloop/fleet_dashboard_test.go index 7e77d9e..06b24f6 100644 --- a/cmd/spinloop/fleet_dashboard_test.go +++ b/cmd/spinloop/fleet_dashboard_test.go @@ -288,11 +288,13 @@ func dashFixNow(t *testing.T, at time.Time) { t.Cleanup(func() { dashNow = time.Now }) } -// dashTestTile draws one tile at the board's current clock, with the staleness -// bound of a local node — the tile tests supply readings with no time on them, -// which are never called stale, so the bound is not what any of them is about. +// dashTestTile draws one tile at the board's current clock, in the board's +// default format (bar), with the staleness bound of a local node — the tile +// tests supply readings with no time on them, which are never called stale, so +// the bound is not what any of them is about. A test that wants the gauge +// format draws with dashTile directly. func dashTestTile(name string, r fleet.NodeResult, selected bool, a dashAction) string { - return dashTile(name, r, selected, a, dashNow(), dashStaleAfter(fleet.KindDaemon)) + return dashTile(name, r, selected, a, dashNow(), dashStaleAfter(fleet.KindDaemon), false) } // dashExpectedHeader is the tile's header bar as a test spells it out: the @@ -607,7 +609,7 @@ func TestDashNodeViewEveryPhaseAgainstEveryReading(t *testing.T) { for _, rd := range readings { t.Run(ph.name+" over "+rd.name, func(t *testing.T) { a := dashAction{verb: "start", since: now.Add(-30 * time.Second), phase: ph.phase} - lines, tier := dashNodeView("n", rd.r, a, now, staleAfter) + lines, tier := dashNodeView("n", rd.r, a, now, staleAfter, false, barLineW) joined := strings.Join(lines, "\n") // The action's own account leads, and the node's report // follows it where the reading has one to give. @@ -633,7 +635,7 @@ func TestDashNodeViewEveryPhaseAgainstEveryReading(t *testing.T) { } // Nothing about the action is shown once it settles, and the // reading alone then decides the tier. - settledLines, settledTier := dashNodeView("n", rd.r, dashAction{}, now, staleAfter) + settledLines, settledTier := dashNodeView("n", rd.r, dashAction{}, now, staleAfter, false, barLineW) if settledTier != rd.settled { t.Errorf("settled tier = %v, want %v", settledTier, rd.settled) } @@ -658,7 +660,7 @@ func TestDashTileStaleReadingShowsItsAgeAndRecovers(t *testing.T) { Metrics: metrics.Stats{State: "running", Ready: "ready"}, At: now.Add(-4 * time.Minute)} staleAfter := dashStaleAfter(fleet.KindRemote) // three minutes, on the minute cadence - got := dashTile("dev-1", r, false, dashAction{}, now, staleAfter) + got := dashTile("dev-1", r, false, dashAction{}, now, staleAfter, false) want := dashTileExpected([]string{ dashExpectedHeader("dev-1 running · 4m 0s ago", dashUnknown), "", "", "", "", "", "", "", "", "", "", "", @@ -672,7 +674,7 @@ func TestDashTileStaleReadingShowsItsAgeAndRecovers(t *testing.T) { dashExpectedHeader("dev-1 running", dashHealthy), "", "", "", "", "", "", "", "", "", "", "", }) - if got := dashTile("dev-1", r, false, dashAction{}, now, staleAfter); got != wantFresh { + if got := dashTile("dev-1", r, false, dashAction{}, now, staleAfter, false); got != wantFresh { t.Errorf("recovered tile mismatch:\ngot:\n%q\nwant:\n%q", got, wantFresh) } } @@ -3076,3 +3078,113 @@ func TestDashProgramDetailViewLogAndBack(t *testing.T) { t.Fatalf("starts=%d, want 1", node.starts) } } + +// A node with retained readings, for the format tests: the same engine a +// byte-stability test would draw, with a short history behind its current +// reading. +func dashHistoryNode() fleet.NodeResult { + return fleet.NodeResult{ + Name: "up", Outcome: fleet.OutcomeOK, + Metrics: metrics.Stats{ + State: "running", Runner: "llamacpp", ModelID: "org/qwen:q4", + UptimeSeconds: 7200, LastActiveAt: "2026-08-21T10:00:00Z", IdleSeconds: 12, + CPU: &metrics.CpuStat{Utilization: 42}, + Memory: &metrics.MemoryStat{Total: 1000, Used: 300}, + GPUs: []metrics.GpuStat{{Index: 0, Name: "H100", Utilization: 61, MemoryUsed: 80, MemoryTotal: 160}}, + Tokens: &metrics.TokenStats{Running: 2, PromptTokens: 4096, GenerationTokens: 1024, Requests: 17}, + History: []metrics.HistorySample{ + {Time: 1786276800, CPU: ptrPct(10), Mem: ptrPct(20), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 50, Mem: ptrPct(40)}}}, + {Time: 1786276815, CPU: ptrPct(20), Mem: ptrPct(30), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 61, Mem: ptrPct(50)}}}, + }, + }, + } +} + +func dashTileAt(gauge bool) string { + return dashTile("up", dashHistoryNode(), false, dashAction{}, dashNow(), dashStaleAfter(fleet.KindDaemon), gauge) +} + +// The g key toggles the board-wide format: every panel redraws in the other +// format, and g again returns them. The formats differ only where a series +// has history to draw — a node without it looks the same either way. +func TestDashModelFormatToggle(t *testing.T) { + lipgloss.SetColorProfile(termenv.Ascii) + dashFixNow(t, dashTestClock) + r := dashHistoryNode() + m := &dashModel{ + entries: []dashEntry{{name: "up"}}, + results: []fleet.NodeResult{r}, + actions: []dashAction{{}}, + width: 120, height: 40, + } + + bar := dashTileAt(false) + gauge := dashTileAt(true) + if bar == gauge { + t.Fatal("the two formats drew identical tiles for a node with history") + } + // The bar format draws the retained readings: a fresh window, so leading + // blank columns, ending on the last retained sample — the current reading + // is the next point, not in the history yet. + cpuLine := " CPU " + strings.Repeat(" ", 23) + "▁" + ansiGreen + "▂" + ansiReset + " 20%" + if !strings.Contains(bar, cpuLine) { + t.Errorf("bar tile did not draw the history, want %q in:\n%s", cpuLine, bar) + } + // The gauge format draws the current reading filled, as before the change. + if !strings.Contains(gauge, dashBar("CPU", 42)) || strings.Contains(gauge, "▁") { + t.Errorf("gauge tile: %q", gauge) + } + + // One press flips the flag, the second press returns it. + next, cmd := m.Update(dashKey("g")) + if cmd != nil { + t.Fatal("the format toggle returned a cmd") + } + m = next.(*dashModel) + if !m.gauge { + t.Fatal("g did not switch the board to gauge") + } + next, _ = m.Update(dashKey("g")) + m = next.(*dashModel) + if m.gauge { + t.Fatal("a second g did not return the board to bar") + } +} + +// Both formats keep the tile's geometry: the frame is the same width and +// height whatever the body draws, so the board's grid never shifts. +func TestDashTileGeometryHoldsInBothFormats(t *testing.T) { + lipgloss.SetColorProfile(termenv.Ascii) + dashFixNow(t, dashTestClock) + for _, gauge := range []bool{false, true} { + lines := strings.Split(dashTileAt(gauge), "\n") + if len(lines) != dashTileH+2 { + t.Errorf("gauge=%v: %d lines, want %d", gauge, len(lines), dashTileH+2) + continue + } + for i, line := range lines { + if w := lipgloss.Width(line); w != dashTileW+2 { + t.Errorf("gauge=%v line %d: width %d, want %d", gauge, i, w, dashTileW+2) + } + } + } +} + +// A node the daemon has not filled the history for — old daemons, a series +// the engine never reported — falls back to the gauge drawing in the bar +// format, so it renders exactly as it did before the change. +func TestDashTileBarFallsBackToGaugeWithoutHistory(t *testing.T) { + lipgloss.SetColorProfile(termenv.Ascii) + dashFixNow(t, dashTestClock) + r := dashHistoryNode() + r.Metrics.History = nil + tile := func(gauge bool) string { + return dashTile("up", r, false, dashAction{}, dashNow(), dashStaleAfter(fleet.KindDaemon), gauge) + } + if got, want := tile(false), tile(true); got != want { + t.Errorf("a history-less node differs between formats:\nbar:\n%q\ngauge:\n%q", got, want) + } + if !strings.Contains(tile(false), dashBar("CPU", 42)) { + t.Errorf("the fallback did not draw the old gauges: %q", tile(false)) + } +} diff --git a/cmd/spinloop/metrics_render.go b/cmd/spinloop/metrics_render.go index e7959c8..9c73d3d 100644 --- a/cmd/spinloop/metrics_render.go +++ b/cmd/spinloop/metrics_render.go @@ -50,30 +50,263 @@ func renderLastActiveKeyValue(w io.Writer, lastActiveAt string, idleSeconds int) } } -// renderStatBars draws the resource bars: CPU, RAM, then each GPU's -// utilisation and memory. Used by the bar format on both sides. -func renderStatBars(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat) { +// validateMetricsFormat rejects a --format value the metrics commands do not +// understand, naming the ones they do. Both `remote metrics` and +// `fleet metrics` run it before doing any work. +func validateMetricsFormat(format string) error { + switch format { + case "bar", "gauge", "table", "json": + return nil + } + return fmt.Errorf("--format must be \"bar\", \"gauge\", \"table\", or \"json\", got %q", format) +} + +// barLineW is the sparkline's draw width in the full view: the width of the +// window at the default sampler cadence, so a steady engine fills it and a +// fresh engine's still-filling window reads as leading space. +const barLineW = 40 + +// dashBarLineW is the sparkline's draw width inside a dashboard tile: the +// tile's width minus the label column and the widest trailing figure, so a +// full row fits the tile exactly and the clip never takes the percentage. +const dashBarLineW = 25 + +// barGlyphs is the eight block elements the sparkline draws with, lightest to +// heaviest: a series' value maps to the one whose fill height is nearest. A +// rune slice, not a string — the elements are multibyte, so byte indexing +// would not land on glyph boundaries. +var barGlyphs = []rune("▁▂▃▄▅▆▇█") + +// barGlyph picks the block element for a 0-100% value. +func barGlyph(pct float64) rune { + i := int(pct / 100.0 * float64(len(barGlyphs))) + if i > len(barGlyphs)-1 { + i = len(barGlyphs) - 1 + } + return barGlyphs[i] +} + +// poolMax reduces a series to at most width values, one per column, each the +// maximum of the samples pooled into its column. A sparkline is about +// pressure and the thresholds are about peaks, so a pool keeps its highest +// reading rather than averaging one away — an average would flatten the red +// spike the series exists to show. +func poolMax(values []float64, width int) []float64 { + if len(values) <= width { + return values + } + out := make([]float64, width) + per := float64(len(values)) / float64(width) + for c := range out { + lo := int(float64(c) * per) + hi := int(float64(c+1) * per) + if hi > len(values) { + hi = len(values) + } + m := values[lo] + for i := lo + 1; i < hi; i++ { + if values[i] > m { + m = values[i] + } + } + out[c] = m + } + return out +} + +// renderSparkline draws one resource series across width columns, one glyph +// per sample, newest on the right. The window's leading columns are blank +// while it still fills, the final glyph takes the state colour on the +// gauge's 80/90 thresholds, and the trailing figure is the latest sample's +// percentage — the exact value the last glyph approximates. +func renderSparkline(w io.Writer, label string, samples []float64, width int) { + pooled := poolMax(samples, width) + last := pooled[len(pooled)-1] + colour := ansiGreen + if last > 90 { + colour = ansiRed + } else if last >= 80 { + colour = ansiYellow + } + fmt.Fprintf(w, " %-9s ", label) + for i := 0; i < width-len(pooled); i++ { + fmt.Fprint(w, " ") + } + for i, v := range pooled { + if i == len(pooled)-1 { + fmt.Fprintf(w, "%s%c%s", colour, barGlyph(v), ansiReset) + } else { + fmt.Fprintf(w, "%c", barGlyph(v)) + } + } + fmt.Fprintf(w, " %.0f%%\n", last) +} + +// renderGauge draws one resource series as a horizontal progress gauge: the +// filled portion in the state colour, the unfilled portion in light shade, +// the percentage in the terminal's default colour. It draws the current +// reading only — it carries no history. +func renderGauge(w io.Writer, label string, pct float64) { + const width = 25 + colour := ansiGreen + if pct > 90 { + colour = ansiRed + } else if pct >= 80 { + colour = ansiYellow + } + filled := int(pct / 100.0 * float64(width)) + if filled > width { + filled = width + } + empty := width - filled + fmt.Fprintf(w, " %-9s ", label) + fmt.Fprintf(w, "%s", colour) + for i := 0; i < filled; i++ { + fmt.Fprint(w, "█") + } + fmt.Fprintf(w, "%s", ansiReset) + for i := 0; i < empty; i++ { + fmt.Fprint(w, "░") + } + fmt.Fprintf(w, " %.0f%%\n", pct) +} + +// barSeries is one resource series the bar and gauge formats draw: the label +// the gauge uses for it, the retained percentage readings oldest first, and +// the current reading the gauge fallback draws where the history holds none. +type barSeries struct { + label string + history []float64 + current *float64 +} + +// barSeriesList works out which series the reading and the history carry, in +// the order the gauge draws them: CPU, RAM, then each GPU's utilisation and +// memory. The GPU set is the union of the two sources in the current +// reading's order first — a stopped engine's current reading names no GPU, +// but its retained readings still name the ones it ran on, and the series +// those readings carry are the point of keeping them. +func barSeriesList(cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, history []metrics.HistorySample) []barSeries { + var out []barSeries + add := func(label string, current *float64, pick func(s metrics.HistorySample) *float64) { + var vals []float64 + for _, s := range history { + if v := pick(s); v != nil { + vals = append(vals, *v) + } + } + if current == nil && len(vals) == 0 { + return + } + out = append(out, barSeries{label: label, history: vals, current: current}) + } if cpu != nil { - renderBar(w, "CPU", cpu.Utilization) + v := cpu.Utilization + add("CPU", &v, func(s metrics.HistorySample) *float64 { return s.CPU }) + } else { + add("CPU", nil, func(s metrics.HistorySample) *float64 { return s.CPU }) } if mem != nil { pct := 0.0 if mem.Total > 0 { pct = float64(mem.Used) / float64(mem.Total) * 100 } - renderBar(w, "RAM", pct) + add("RAM", &pct, func(s metrics.HistorySample) *float64 { return s.Mem }) + } else { + add("RAM", nil, func(s metrics.HistorySample) *float64 { return s.Mem }) } + seen := map[int]bool{} + var idxs []int for _, g := range gpus { + if !seen[g.Index] { + seen[g.Index] = true + idxs = append(idxs, g.Index) + } + } + for _, s := range history { + for _, g := range s.GPUs { + if !seen[g.Index] { + seen[g.Index] = true + idxs = append(idxs, g.Index) + } + } + } + for _, idx := range idxs { prefix := "GPU" - if len(gpus) > 1 { - prefix = fmt.Sprintf("GPU %d", g.Index) + if len(idxs) > 1 { + prefix = fmt.Sprintf("GPU %d", idx) + } + add(prefix+" util", currentGPUUtil(gpus, idx), func(s metrics.HistorySample) *float64 { + for _, g := range s.GPUs { + if g.Index == idx { + v := float64(g.Util) + return &v + } + } + return nil + }) + add(prefix+" mem", currentGPUMem(gpus, idx), func(s metrics.HistorySample) *float64 { + for _, g := range s.GPUs { + if g.Index == idx { + return g.Mem + } + } + return nil + }) + } + return out +} + +// currentGPUUtil is the GPU's utilisation from the current reading, nil where +// the reading names no such GPU — the stopped engine's case. +func currentGPUUtil(gpus []metrics.GpuStat, idx int) *float64 { + for _, g := range gpus { + if g.Index == idx { + v := float64(g.Utilization) + return &v } - renderBar(w, prefix+" util", float64(g.Utilization)) - memPct := 0.0 - if g.MemoryTotal > 0 { - memPct = float64(g.MemoryUsed) / float64(g.MemoryTotal) * 100 + } + return nil +} + +// currentGPUMem is the GPU's memory ratio from the current reading, 0 where +// the GPU reports no total — the gauge's own rule for that case. +func currentGPUMem(gpus []metrics.GpuStat, idx int) *float64 { + for _, g := range gpus { + if g.Index == idx { + pct := 0.0 + if g.MemoryTotal > 0 { + pct = float64(g.MemoryUsed) / float64(g.MemoryTotal) * 100 + } + return &pct + } + } + return nil +} + +// renderStatBars draws the resource series in the bar format: each series as +// a sparkline of the retained history, or — where the history holds none for +// the series — as the gauge drawing of the current reading, so a daemon that +// predates the history renders exactly as it did before it. Used by the bar +// format on every surface; lineW is the draw width, barLineW for the full +// view and dashBarLineW for a dashboard tile. +func renderStatBars(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, history []metrics.HistorySample, lineW int) { + for _, s := range barSeriesList(cpu, mem, gpus, history) { + if len(s.history) > 0 { + renderSparkline(w, s.label, s.history, lineW) + } else { + renderGauge(w, s.label, *s.current) + } + } +} + +// renderStatGauges draws the resource series in the gauge format: the current +// reading only, whatever the history holds. +func renderStatGauges(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat) { + for _, s := range barSeriesList(cpu, mem, gpus, nil) { + if s.current != nil { + renderGauge(w, s.label, *s.current) } - renderBar(w, prefix+" mem", memPct) } } diff --git a/cmd/spinloop/metrics_render_test.go b/cmd/spinloop/metrics_render_test.go new file mode 100644 index 0000000..463b5c3 --- /dev/null +++ b/cmd/spinloop/metrics_render_test.go @@ -0,0 +1,316 @@ +package main + +import ( + "bytes" + "strings" + "testing" + + "github.com/spinloop-ai/spinloop/internal/metrics" + "github.com/spinloop-ai/spinloop/internal/remote" +) + +func ptrPct(v float64) *float64 { return &v } + +func TestBarGlyph(t *testing.T) { + cases := []struct { + pct float64 + want rune + }{ + {0, '▁'}, {12.4, '▁'}, {12.5, '▂'}, {30, '▃'}, {50, '▅'}, + {75, '▇'}, {87.4, '▇'}, {87.5, '█'}, {100, '█'}, + } + for _, c := range cases { + if got := barGlyph(c.pct); got != c.want { + t.Errorf("barGlyph(%v) = %c, want %c", c.pct, got, c.want) + } + } +} + +func TestPoolMaxKeepsPeaks(t *testing.T) { + // A pool keeps its highest reading, so the spike a threshold is about + // survives the downsampling instead of being averaged away. + got := poolMax([]float64{1, 9, 2, 8, 3, 7}, 3) + if len(got) != 3 || got[0] != 9 || got[1] != 8 || got[2] != 7 { + t.Errorf("poolMax = %v, want [9 8 7]", got) + } + // A series no wider than the draw is passed through untouched. + in := []float64{1, 2, 3} + if got := poolMax(in, 5); len(got) != 3 || got[0] != 1 || got[1] != 2 || got[2] != 3 { + t.Errorf("an unwidened series changed: %v", got) + } +} + +func TestRenderSparkline(t *testing.T) { + var b bytes.Buffer + renderSparkline(&b, "CPU", []float64{20, 30}, 40) + // The trailing figure is the latest sample's percentage — the exact value + // the last glyph approximates — not the series' first. + want := " CPU " + strings.Repeat(" ", 38) + "▂" + ansiGreen + "▃" + ansiReset + " 30%\n" + if got := b.String(); got != want { + t.Errorf("sparkline = %q, want %q", got, want) + } +} + +// Only the final glyph takes the state colour, on the gauge's 80/90 +// thresholds, whatever the rest of the window did — the window here holds a +// red spike earlier that must stay uncoloured. +func TestRenderSparklineColoursOnlyTheLastPoint(t *testing.T) { + cases := []struct { + last float64 + want string + }{ + {79.9, ansiGreen + "▇" + ansiReset + " 80%\n"}, + {85, ansiYellow + "▇" + ansiReset + " 85%\n"}, + {95, ansiRed + "█" + ansiReset + " 95%\n"}, + } + for _, c := range cases { + var b bytes.Buffer + renderSparkline(&b, "CPU", []float64{5, 95, c.last}, 40) + if !strings.HasSuffix(b.String(), c.want) { + t.Errorf("last point %v: %q, want suffix %q", c.last, b.String(), c.want) + } + // Every earlier glyph is uncoloured: the escapes appear once, around + // the final glyph only. + if n := strings.Count(b.String(), "\033["); n != 2 { + t.Errorf("last point %v: %d escape sequences, want 2", c.last, n) + } + } +} + +func TestRenderGauge(t *testing.T) { + var b bytes.Buffer + renderGauge(&b, "CPU", 42) + want := " CPU " + ansiGreen + strings.Repeat("█", 10) + ansiReset + strings.Repeat("░", 15) + " 42%\n" + if got := b.String(); got != want { + t.Errorf("gauge = %q, want %q", got, want) + } +} + +func TestValidateMetricsFormat(t *testing.T) { + for _, f := range []string{"bar", "gauge", "table", "json"} { + if err := validateMetricsFormat(f); err != nil { + t.Errorf("%q rejected: %v", f, err) + } + } + err := validateMetricsFormat("csv") + if err == nil || !strings.Contains(err.Error(), `"csv"`) { + t.Errorf("csv: %v, want the bad value named", err) + } +} + +// A daemon that predates the history — or a series it never reported — draws +// exactly as the old bar format did, byte for byte. +func TestRenderStatBarsFallsBackToGaugeWithoutHistory(t *testing.T) { + cpu := &metrics.CpuStat{Utilization: 42} + mem := &metrics.MemoryStat{Total: 1000, Used: 300} + gpus := []metrics.GpuStat{{Index: 0, Name: "H100", Utilization: 61, MemoryUsed: 80, MemoryTotal: 160}} + + var got, want bytes.Buffer + renderStatBars(&got, cpu, mem, gpus, nil, barLineW) + renderStatGauges(&want, cpu, mem, gpus) + if got.String() != want.String() { + t.Errorf("no-history bar differs from the old drawing:\ngot:\n%q\nwant:\n%q", got.String(), want.String()) + } + if !strings.Contains(got.String(), "░") { + t.Errorf("the fallback drew no gauge: %q", got.String()) + } +} + +func TestRenderStatBarsDrawsHistory(t *testing.T) { + history := []metrics.HistorySample{ + {Time: 1, CPU: ptrPct(10), Mem: ptrPct(20)}, + {Time: 2, CPU: ptrPct(20), Mem: ptrPct(30)}, + {Time: 3, CPU: ptrPct(95), Mem: ptrPct(40)}, + } + var b bytes.Buffer + renderStatBars(&b, nil, nil, nil, history, barLineW) + lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") + if len(lines) != 2 { + t.Fatalf("drew %d lines, want CPU and RAM: %q", len(lines), b.String()) + } + if !strings.Contains(lines[0], ansiRed+"█"+ansiReset+" 95%") { + t.Errorf("CPU line lost its spike or its red last point: %q", lines[0]) + } + if !strings.Contains(lines[1], ansiGreen+"▄"+ansiReset+" 40%") { + t.Errorf("RAM line: %q", lines[1]) + } +} + +// The fallback is per series: a series with retained readings draws them, a +// series without falls back to its current reading on the same screen. +func TestRenderStatBarsFallsBackPerSeries(t *testing.T) { + history := []metrics.HistorySample{ + {Time: 1, CPU: ptrPct(10)}, + {Time: 2, CPU: ptrPct(20)}, + } + var b bytes.Buffer + renderStatBars(&b, nil, &metrics.MemoryStat{Total: 1000, Used: 300}, nil, history, barLineW) + lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") + if len(lines) != 2 { + t.Fatalf("drew %d lines, want 2: %q", len(lines), b.String()) + } + if strings.Contains(lines[0], "░") || !strings.Contains(lines[0], " 20%") { + t.Errorf("CPU did not draw its history: %q", lines[0]) + } + if !strings.Contains(lines[1], "░") || !strings.Contains(lines[1], " 30%") { + t.Errorf("RAM did not fall back to the gauge: %q", lines[1]) + } +} + +// A stopped engine carries no current figures, so the bar format draws the +// retained readings alone — including the GPU series the engine ran on, which +// the current reading no longer names. +func TestRenderStatBarsStoppedEngineDrawsHistoryAlone(t *testing.T) { + history := []metrics.HistorySample{ + {Time: 1, CPU: ptrPct(10), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 50, Mem: ptrPct(50)}}}, + {Time: 2, CPU: ptrPct(20), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 60, Mem: ptrPct(60)}}}, + } + var b bytes.Buffer + renderStatBars(&b, nil, nil, nil, history, barLineW) + lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") + if len(lines) != 3 { + t.Fatalf("drew %d lines, want CPU, GPU util and GPU mem: %q", len(lines), b.String()) + } + if !strings.HasPrefix(lines[0], " CPU ") { + t.Errorf("first line: %q", lines[0]) + } + // One GPU: the plain "GPU" prefix, as the gauge used it. + if !strings.Contains(lines[1], "GPU util") || !strings.Contains(lines[1], " 60%") { + t.Errorf("GPU util line: %q", lines[1]) + } + if !strings.Contains(lines[2], "GPU mem") || !strings.Contains(lines[2], " 60%") { + t.Errorf("GPU mem line: %q", lines[2]) + } + + // Two GPUs: the index goes in the label. + history = []metrics.HistorySample{ + {Time: 1, CPU: ptrPct(10), GPUs: []metrics.HistoryGPU{ + {Index: 0, Util: 10, Mem: ptrPct(10)}, + {Index: 1, Util: 90, Mem: ptrPct(20)}, + }}, + } + b.Reset() + renderStatBars(&b, nil, nil, nil, history, barLineW) + out := b.String() + if !strings.Contains(out, "GPU 0 util") || !strings.Contains(out, "GPU 1 util") || + !strings.Contains(out, "GPU 0 mem") || !strings.Contains(out, "GPU 1 mem") { + t.Errorf("two GPUs: %q", out) + } +} + +// The gauge format draws the current reading only, whatever the history +// holds — and a stopped engine's reading carries nothing, so it draws none. +func TestRenderStatGaugesIgnoresHistory(t *testing.T) { + cpu := &metrics.CpuStat{Utilization: 50} + var b bytes.Buffer + renderStatGauges(&b, cpu, nil, nil) + if !strings.Contains(b.String(), " 50%") || strings.Contains(b.String(), "▁") { + t.Errorf("gauge drew history or the wrong value: %q", b.String()) + } + b.Reset() + renderStatGauges(&b, nil, nil, nil) + if b.String() != "" { + t.Errorf("a stopped engine drew gauges: %q", b.String()) + } +} + +func TestFormatMetricsBarStoppedWithHistory(t *testing.T) { + resp := &remote.StatsResponse{ + Environment: "prod", State: "stopped", ModelID: "org/qwen:q4", + LastActiveAt: "2026-08-21T10:00:00Z", IdleSeconds: 12, + History: []metrics.HistorySample{ + {Time: 1, CPU: ptrPct(10)}, + {Time: 2, CPU: ptrPct(20)}, + }, + } + var b bytes.Buffer + if err := formatMetricsBar(resp, remote.Config{}, &b); err != nil { + t.Fatal(err) + } + want := "prod stopped org/qwen:q4\n" + + " last active 12s ago\n" + + " CPU " + strings.Repeat(" ", 38) + "▁" + ansiGreen + "▂" + ansiReset + " 20%\n" + if got := b.String(); got != want { + t.Errorf("stopped bar = %q, want %q", got, want) + } + + // The gauge format draws no series for a stopped endpoint: the header and + // the last-active line, and nothing after. + b.Reset() + if err := formatMetricsGauge(resp, remote.Config{}, &b); err != nil { + t.Fatal(err) + } + want = "prod stopped org/qwen:q4\n" + + " last active 12s ago\n" + if got := b.String(); got != want { + t.Errorf("stopped gauge = %q, want %q", got, want) + } +} + +func TestFormatMetricsBarRunning(t *testing.T) { + resp := &remote.StatsResponse{ + Environment: "prod", State: "running", InstanceType: "g5.xlarge", + ModelID: "org/qwen:q4", Version: "0.4.3", + LastActiveAt: "2026-08-21T10:00:00Z", IdleSeconds: 3, + CPU: &metrics.CpuStat{Utilization: 62}, + Memory: &metrics.MemoryStat{Total: 1000, Used: 300}, + GPUs: []metrics.GpuStat{{Index: 0, Name: "H100", Utilization: 61, MemoryUsed: 80, MemoryTotal: 160}}, + Tokens: &remote.TokenStats{Running: 2, PromptTokens: 4096, GenerationTokens: 1024, Requests: 17}, + History: []metrics.HistorySample{ + {Time: 1, CPU: ptrPct(10), Mem: ptrPct(20), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 50, Mem: ptrPct(50)}}}, + {Time: 2, CPU: ptrPct(20), Mem: ptrPct(30), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 61, Mem: ptrPct(50)}}}, + }, + } + var b bytes.Buffer + if err := formatMetricsBar(resp, remote.Config{}, &b); err != nil { + t.Fatal(err) + } + got := b.String() + if !strings.HasPrefix(got, "prod running g5.xlarge org/qwen:q4 0.4.3\n") { + t.Errorf("header: %q", got) + } + if !strings.Contains(got, " last active 3s ago\n") { + t.Errorf("last active line missing: %q", got) + } + // Every series drew a sparkline from the history: no gauge in the output. + if strings.Contains(got, "░") { + t.Errorf("a series fell back to the gauge although history holds it: %q", got) + } + for _, label := range []string{"CPU", "RAM", "GPU util", "GPU mem"} { + if !strings.Contains(got, label) { + t.Errorf("series %q missing: %q", label, got) + } + } + if !strings.Contains(got, " running: 2\n") || !strings.Contains(got, " requests: 17\n") { + t.Errorf("token block missing: %q", got) + } +} + +func TestFormatMetricsJSONCarriesHistory(t *testing.T) { + resp := &remote.StatsResponse{ + Environment: "prod", State: "running", + CPU: &metrics.CpuStat{Utilization: 62}, + History: []metrics.HistorySample{ + {Time: 1786276800, CPU: ptrPct(62), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 61, Mem: ptrPct(50)}}}, + }, + } + var b bytes.Buffer + if err := formatMetricsJSON(resp, false, remote.Config{}, &b); err != nil { + t.Fatal(err) + } + got := b.String() + for _, want := range []string{`"history"`, `"t":`, `"c":`, `"g":`, `"i":`, `"u":`} { + if !strings.Contains(got, want) { + t.Errorf("json %s: missing %s", want, got) + } + } + // Absent history stays absent, as on the daemon. + b.Reset() + resp.History = nil + if err := formatMetricsJSON(resp, false, remote.Config{}, &b); err != nil { + t.Fatal(err) + } + if strings.Contains(b.String(), "history") { + t.Errorf("absent history serialised: %s", b.String()) + } +} diff --git a/cmd/spinloop/remote.go b/cmd/spinloop/remote.go index b12e2be..804c6fa 100644 --- a/cmd/spinloop/remote.go +++ b/cmd/spinloop/remote.go @@ -772,10 +772,12 @@ func runRemoteStatus(args []string) error { } // cmdRemoteMetrics queries the stats Lambda for instance metrics: token usage, -// GPU, CPU, and RAM utilization. With --format=json it outputs JSON; the -// default is a key-value table. With --cost, it looks up the on-demand price -// for the instance type from the AWS Price List API. With --watch it polls -// every 60 seconds until interrupted. +// GPU, CPU, and RAM utilization. The default bar format draws each series as +// a sparkline of the daemon's retained history; --format=gauge draws the +// current reading as progress gauges instead. With --format=json it outputs +// JSON. With --cost, it looks up the on-demand price for the instance type +// from the AWS Price List API. With --watch it polls every 60 seconds until +// interrupted. func remoteMetricsCmd() *cobra.Command { var ( withCost bool @@ -796,15 +798,15 @@ func remoteMetricsCmd() *cobra.Command { } fs := c.Flags() fs.BoolVar(&withCost, "cost", false, "include cost estimate from AWS Price List API") - fs.StringVar(&format, "format", "bar", "output format: bar (default), table or json") + fs.StringVar(&format, "format", "bar", "output format: bar (default), gauge, table or json") fs.BoolVarP(&watch, "watch", "w", false, "poll metrics every 60 seconds") return c } // runRemoteMetrics is the body of `spinloop remote metrics`. func runRemoteMetrics(args []string, withCost bool, format string, watch bool) error { - if format != "table" && format != "json" && format != "bar" { - return fmt.Errorf("--format must be \"table\", \"bar\", or \"json\", got %q", format) + if err := validateMetricsFormat(format); err != nil { + return err } cfg, err := resolveRemoteConfig(spinloopArg(args)) @@ -824,10 +826,12 @@ func runMetricsOnce(ctx context.Context, cfg remote.Config, format string, withC return err } - if format == "json" { + switch format { + case "json": return formatMetricsJSON(resp, withCost, cfg, w) - } - if format == "bar" { + case "gauge": + return formatMetricsGauge(resp, cfg, w) + case "bar": return formatMetricsBar(resp, cfg, w) } return formatMetricsTable(ctx, resp, withCost, cfg, w) @@ -955,7 +959,9 @@ func formatMetricsJSON(resp *remote.StatsResponse, withCost bool, cfg remote.Con return nil } -func formatMetricsBar(resp *remote.StatsResponse, cfg remote.Config, w io.Writer) error { +// formatMetricsHeader draws the line both compact formats open with: the +// environment, state, instance type, and model, double-spaced. +func formatMetricsHeader(resp *remote.StatsResponse, w io.Writer) { fmt.Fprintf(w, "%s %s", resp.Environment, resp.State) if resp.InstanceType != "" { fmt.Fprintf(w, " %s", resp.InstanceType) @@ -967,47 +973,45 @@ func formatMetricsBar(resp *remote.StatsResponse, cfg remote.Config, w io.Writer fmt.Fprintf(w, " %s", resp.Version) } fmt.Fprintln(w) +} + +func formatMetricsBar(resp *remote.StatsResponse, cfg remote.Config, w io.Writer) error { + formatMetricsHeader(resp, w) + + // Before the series: when the endpoint last did work is worth showing for + // whatever state it is in, and the retained history is too — a stopped + // endpoint's readings up to the stop answer what it was doing until it + // stopped. A stopped endpoint's current reading carries no resource + // figures, so the series it draws come from the history alone, or not at + // all where the daemon predates it. + renderLastActiveIndented(w, resp.LastActiveAt, resp.IdleSeconds) + + renderStatBars(w, resp.CPU, resp.Memory, resp.GPUs, resp.History, barLineW) + renderTokenLines(w, resp.Tokens) + renderCollectionErrors(os.Stderr, resp.Errors) - // Before the early return: a stopped endpoint draws no bars, but when it - // last did work is exactly what a stopped endpoint is worth asking about. + return nil +} + +func formatMetricsGauge(resp *remote.StatsResponse, cfg remote.Config, w io.Writer) error { + formatMetricsHeader(resp, w) + + // Before the early return: a stopped endpoint draws no gauges, but when + // it last did work is exactly what a stopped endpoint is worth asking + // about. renderLastActiveIndented(w, resp.LastActiveAt, resp.IdleSeconds) if resp.State != "running" { return nil } - renderStatBars(w, resp.CPU, resp.Memory, resp.GPUs) + renderStatGauges(w, resp.CPU, resp.Memory, resp.GPUs) renderTokenLines(w, resp.Tokens) renderCollectionErrors(os.Stderr, resp.Errors) return nil } -func renderBar(w io.Writer, label string, pct float64) { - const width = 25 - colour := "\033[92m" - if pct > 90 { - colour = "\033[31m" - } else if pct >= 80 { - colour = "\033[33m" - } - filled := int(pct / 100.0 * float64(width)) - if filled > width { - filled = width - } - empty := width - filled - fmt.Fprintf(w, " %-9s ", label) - fmt.Fprintf(w, "%s", colour) - for i := 0; i < filled; i++ { - fmt.Fprint(w, "█") - } - fmt.Fprintf(w, "\033[0m") - for i := 0; i < empty; i++ { - fmt.Fprint(w, "░") - } - fmt.Fprintf(w, " %.0f%%\n", pct) -} - func formatDuration(seconds int) string { d := time.Duration(seconds) * time.Second h := int(d.Hours()) diff --git a/docs/commands/fleet.md b/docs/commands/fleet.md index 95de66f..36d6103 100644 --- a/docs/commands/fleet.md +++ b/docs/commands/fleet.md @@ -248,9 +248,13 @@ has been started at all. ## Metrics `spinloop fleet metrics` renders each node's engine and system metrics in the -same `bar` (default), `table`, and `json` formats as +same `bar` (default), `gauge`, `table`, and `json` formats as [`spinloop remote metrics`](remote.md) — they share the renderers, so a node in -your fleet and a cloud endpoint look the same. +your fleet and a cloud endpoint look the same. `bar` draws each series as a +sparkline of the node's daemon's retained history; a node whose daemon reports +no history falls back to the gauge drawing of its current reading, so a fleet +mixed with older daemons renders each node the best way it can. A stopped node +keeps its readings, so its sparkline runs to the stop. Each node's block carries the same `last active` figure the status table shows, for the reasons given above, and on the same terms: absent until the @@ -279,10 +283,11 @@ silently missing whatever was down: `spinloop fleet dashboard` is that same board as a live view: one tile per node, repainted in place, each drawing exactly what `fleet metrics`' bar format prints for the node — state and uptime, what it serves, the CPU/GPU/RAM -bars, the token counters — so the view and the one-shot command never word a -number differently. A node that is down is a tile that says why, and a node -whose token reference resolves to nothing holds that reason for the life of -the view: +sparklines, the token counters — so the view and the one-shot command never +word a number differently. `g` toggles every tile between the sparklines and +the gauge drawing of the current reading; the board opens in bar. A node that +is down is a tile that says why, and a node whose token reference resolves to +nothing holds that reason for the life of the view: ```sh spinloop fleet dashboard # ./fleet.yaml @@ -295,6 +300,7 @@ spinloop fleet dashboard --fleet f.yaml # another fleet file | `PgUp`/`PgDn` | Page the grid when there are more nodes than fit | | `Enter` | Open a full-screen view of the selected node | | `r` | Force a refresh of every node, now | +| `g` | Toggle every tile's resource series between bar (sparklines of the retained history) and gauge (the current reading) | | `s` | Start the selected node — without confirmation | | `a` | Abandon a start in flight on the selected node — the wait ends, the node is free again (a stop in flight is not abortable) | | `x` | Stop the selected node — it asks first (`y` sends, `n` or `esc` cancel) | @@ -513,7 +519,7 @@ deploy`](remote.md), applied per node. | `--all` | `start`/`stop`/`deploy`: act on every node (or every `kind: remote` node, for `deploy`) instead of named ones | | `--node ` | `route` only: report this node rather than choosing one | | `--prefer` | `route` only: rank by `idle` or `active`, overriding the file | -| `--format` | `metrics`: `bar` (default), `table`, or `json`; `logs`: `text` (default) or `json` | +| `--format` | `metrics`: `bar` (default), `gauge`, `table`, or `json`; `logs`: `text` (default) or `json` | | `-w`, `--watch` | `metrics` only: redraw on an interval until interrupted | | `-f`, `--follow` | `logs` only: keep printing new output until interrupted | | `--limit` | `logs` only: lines of backlog per node (default 200) | diff --git a/docs/commands/remote.md b/docs/commands/remote.md index 2678d3a..0527ba3 100644 --- a/docs/commands/remote.md +++ b/docs/commands/remote.md @@ -158,6 +158,17 @@ spinloop remote metrics # what is it doing — tokens, GPU, CPU spinloop remote metrics -w # the same, redrawn every 60 seconds ``` +`metrics` draws its resource series as **bar** format by default: a sparkline +of the last 10 minutes per series, taken by the on-instance daemon at its +sampler's cadence while the engine ran, with only the latest point coloured — +green at or below 80%, yellow to 90%, red above. `--format=gauge` draws the +current reading as filled progress gauges instead, `--format=table` as a +key-value table, and `--format=json` as the raw reply. A daemon that predates +the history, or an engine with no reading yet, falls back to the gauge drawing +of the current reading, so the default output degrades rather than goes blank. +A stopped engine keeps its readings: the sparkline runs to the stop, ending +at it. + Both report **`last active`** — how long since the endpoint's engine last did any work. It comes from the activity the on-instance daemon tracks, so it is one answer decided on the box rather than something each command re-derives diff --git a/docs/http-api.md b/docs/http-api.md index e8ec9b0..11726b9 100644 --- a/docs/http-api.md +++ b/docs/http-api.md @@ -58,8 +58,17 @@ Stops the engine. Returns the current metrics: - Token usage counters (from the engine's Prometheus `/metrics` endpoint) - Host system metrics (GPU, CPU, RAM) +- `history`, the daemon's retained system readings — one per sampler tick while an engine ran, each a 0–100% figure per series (`t` time, `c` CPU, `m` memory, `g` per-GPU utilisation and memory), covering at most the last 10 minutes - `lastActiveAt` and `idleSeconds`, the same pair `/v1/status` reports +The history survives a stop — the readings up to the stop say what the engine +was doing until it stopped — and clears when the next engine starts, so one +engine's readings are never reported against another. It is omitted where no +reading has been taken. The field names are one letter each on purpose: the +readings also ride the cloud relay over SSM, whose command output truncates at +4KB, and the window's samples must fit that budget alongside the current +reading. + The activity pair comes from the same record `/v1/status` reads, so the two endpoints cannot disagree. Unlike the counters and system figures, it is reported whatever the engine's state: a stopped engine returns no tokens and diff --git a/docs/internals.md b/docs/internals.md index a763672..f1dc2b4 100644 --- a/docs/internals.md +++ b/docs/internals.md @@ -38,10 +38,15 @@ A few Bubble Tea/lipgloss specifics that are easy to break by "simplifying": - One function, `dashNodeView`, produces both a panel's lines and its health tier, from the reading, the action, the current time, and how old a reading of that node may be. Nothing in it reads a clock, so every pairing of a start's phase against a reading can be enumerated in a test. - A tile's first line is a header bar drawn in raw ANSI — the body is one plain string under a single lipgloss style, so per-character colour cannot be lipgloss's. The board's own title bar (`dashTitleBar`) uses lipgloss instead, and the two share one surface index (`barSurface`) because they are set through different mechanisms and would otherwise drift. - A grid row joins the *corresponding lines* of the tiles it places, not the tile blocks — joining whole blocks glues the second tile's top border to the first tile's bottom border and shifts its body down a line. -- A tile's content is exactly the lines `fleet metrics` bar format prints (`renderStatBars`/`renderTokenLines` are shared, not reimplemented), so the panel and `fleet metrics` can never disagree on a number. +- A tile's content is exactly the lines `fleet metrics` prints for the node in the board's current format (`renderStatBars`/`renderStatGauges`/`renderTokenLines` are shared, not reimplemented), so the panel and `fleet metrics` can never disagree on a number. The format is board-wide, in `dashModel.gauge`, toggled by `g`, opening in bar; the tile draws the sparkline at `dashBarLineW`, chosen so a full row (label, glyphs, trailing percentage) fits the tile's width exactly. Behavior (panel contents, refresh cadence, start/stop/abort semantics, the detail view) is specified in `openspec/specs/fleet-client/spec.md`. +## The metrics history + +- The bar format's data lives in the daemon, not the client: `systemHistory` (`internal/daemon/history.go`) is appended on each sampler tick while an engine runs, survives a stop, and clears on the next start alongside the counter baseline's `sample.forget()`. Every client — one-shot, watch, dashboard, the cloud relay — draws the same window from the one read. +- The history samples' JSON field names are one letter each (`t`/`c`/`m`/`g`, `i`/`u`/`m`) because the reply crosses SSM on the cloud relay, and SSM command output truncates at 4KB. The window is 10 minutes at the 15s cadence (40 samples) and is capped at `historyLimit` samples regardless of cadence — the catch-up ticks run at 1s, and an engine with no scrape target never leaves that cadence. If the window or the sample shape grows, cap the size in the daemon, not the client: a truncated reply is corrupt JSON, and `parseDaemonMetrics` turns that into "daemon unreachable" for the whole metrics call. + ## Adapter schema references - opencode config schema: https://opencode.ai/docs/config/. The catalogue follows it: `amazon-bedrock` is the Bedrock provider id, custom providers (`ollama`, `llamacpp`, `openai-compatible`) carry an `npm` package plus `options.baseURL`. The key is written as opencode's `{env:VAR}` substitution rather than the resolved secret, so no secret lands on disk; `spinloop harness` passes the keys it can resolve to the agent it launches, which is what makes a config spinloop wrote usable without exporting anything by hand. diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 180c13f..d15b272 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -396,6 +396,17 @@ components: $ref: "#/components/schemas/CpuStat" memory: $ref: "#/components/schemas/MemoryStat" + history: + type: array + description: | + The retained system readings, oldest first — one per sampler tick + while an engine ran, covering at most the last 10 minutes. They + survive a stop (the readings up to the stop say what the engine + was doing until it stopped) and clear when the next engine + starts. Absent where no reading has been taken, on the same + absence-not-zero terms as the rest of the reply. + items: + $ref: "#/components/schemas/HistorySample" errors: type: array description: Collection failures. An absent source is omitted rather than reported here. @@ -479,6 +490,45 @@ components: used: type: integer + HistorySample: + type: object + description: | + One retained reading of the host's figures, as the bar format plots + it: a 0-100% figure per series rather than the raw one. The field + names are one letter each because the readings ride the remote relay + over SSM, whose command output truncates at 4KB — forty samples of + the window must fit that budget alongside the current reading. + required: [t] + properties: + t: + type: integer + description: When the reading was taken, unix seconds. + c: + type: number + description: Whole-host CPU utilisation, percent. + m: + type: number + description: System memory used over total, percent. + g: + type: array + items: + $ref: "#/components/schemas/HistoryGPU" + + HistoryGPU: + type: object + description: One GPU's figures in a retained reading, one-letter fields as its parent. + required: [i, u] + properties: + i: + type: integer + description: The GPU's index. + u: + type: integer + description: Utilisation, percent. + m: + type: number + description: Memory used over total, percent. Absent where the GPU reports no total. + DeployConfig: type: object description: | diff --git a/internal/daemon/activity.go b/internal/daemon/activity.go index 84438ae..ab7d9bf 100644 --- a/internal/daemon/activity.go +++ b/internal/daemon/activity.go @@ -93,6 +93,7 @@ func (d *Daemon) SampleActivity(ctx context.Context) { for { d.sampleOnce(ctx) d.checkReadyOnce(ctx) + d.systemSampleOnce(ctx) // Until a reading has landed there is nothing for /v1/metrics to // report, so wait a short interval rather than the full one. That is // the window just after an engine starts, when someone is most likely diff --git a/internal/daemon/daemon.go b/internal/daemon/daemon.go index 817d032..544ddaa 100644 --- a/internal/daemon/daemon.go +++ b/internal/daemon/daemon.go @@ -65,6 +65,7 @@ type Daemon struct { act activity sample engineSample ready readiness + hist systemHistory mu sync.Mutex runner string @@ -264,10 +265,11 @@ func (d *Daemon) StartEngine() error { // existed — the race the control plane used to close with a last-wake // timestamp of its own. d.act.markActive(d.now()) - // The previous engine's counters must not be reported against this one, - // for the same reason its counter baseline is dropped. + // The previous engine's counters and system readings must not be reported + // against this one, for the same reason its counter baseline is dropped. d.sample.forget() d.ready.forget() + d.hist.clear() return nil } @@ -434,5 +436,9 @@ func (d *Daemon) Metrics(ctx context.Context) metrics.Stats { // the scrape, so a poll reports the activity its own reading just // established rather than the record as it stood one call ago. stats.LastActiveAt, stats.IdleSeconds = d.activity() + // The same survival rule: the readings up to the stop say what the engine + // was doing until it stopped, so they are reported while the running- + // engine figures above are absent. + stats.History = d.hist.snapshot() return stats } diff --git a/internal/daemon/history.go b/internal/daemon/history.go new file mode 100644 index 0000000..1bd237b --- /dev/null +++ b/internal/daemon/history.go @@ -0,0 +1,114 @@ +package daemon + +import ( + "context" + "sync" + "time" + + "github.com/spinloop-ai/spinloop/internal/metrics" +) + +// historyWindow is how far back the retained system readings reach. It is +// the span the bar format draws, so it bounds the buffer whatever the +// sampler's cadence — a faster cadence simply holds more samples inside the +// same window. +const historyWindow = 10 * time.Minute + +// historyLimit is the most samples the buffer holds. The time window is the +// rule the drawing follows; this is the guard for the remote relay, where the +// metrics reply crosses SSM and its output truncates at 4KB. At the default +// 15s cadence the limit never bites — 10 minutes holds exactly 40 samples — +// but a sampler running faster than that (the catch-up cadence just after a +// start) would otherwise fill the reply past the truncation. +const historyLimit = 40 + +// systemHistory is the daemon's retained readings of the host's CPU, memory +// and GPU figures, taken by the sampler while an engine runs. It survives a +// stop — the readings up to the stop answer what the engine was doing until +// it stopped — and is cleared when the next engine starts, so one engine's +// readings are never reported against another. +type systemHistory struct { + mu sync.Mutex + samples []metrics.HistorySample +} + +// add records one reading, dropping what has aged out of the window. The +// sample's own time is the reference, so a reading is never kept longer than +// the window past a later one. +func (h *systemHistory) add(s metrics.HistorySample) { + h.mu.Lock() + defer h.mu.Unlock() + cutoff := time.Unix(s.Time, 0).Add(-historyWindow) + i := 0 + for i < len(h.samples) && time.Unix(h.samples[i].Time, 0).Before(cutoff) { + i++ + } + h.samples = append(h.samples[i:], s) + if len(h.samples) > historyLimit { + h.samples = h.samples[len(h.samples)-historyLimit:] + } +} + +// snapshot reports the retained readings, oldest first; nil when there are +// none, so the metrics reply omits the field rather than showing an empty +// window for a daemon that has never run an engine. +func (h *systemHistory) snapshot() []metrics.HistorySample { + h.mu.Lock() + defer h.mu.Unlock() + if len(h.samples) == 0 { + return nil + } + out := make([]metrics.HistorySample, len(h.samples)) + copy(out, h.samples) + return out +} + +// clear drops every reading. StartEngine calls it beside the counter +// baseline's drop, for the same reason: the previous engine's figures must +// not be reported against the next one. +func (h *systemHistory) clear() { + h.mu.Lock() + defer h.mu.Unlock() + h.samples = nil +} + +// systemSampleOnce takes one reading of the host's figures, for the retained +// history. Unlike the counter scrape it needs no scrape target — the figures +// come from host commands, not from the engine — so an engine with no metrics +// endpoint still yields a window to draw. It runs only while an engine is +// running: a stopped engine has no utilization to chart, and the readings +// taken before the stop remain until the next start. A reading that yields no +// figure at all records nothing and reports nothing: a failed sample is a +// non-observation here as in the activity record, and the on-request +// collection keeps its own error reporting. +func (d *Daemon) systemSampleOnce(ctx context.Context) { + if state, _, _ := d.Sup.Status(); state != StateRunning { + return + } + if d.Collector == nil { + return + } + var stats metrics.Stats + d.Collector.System(ctx, &stats) + sample := metrics.HistorySample{Time: d.now().Unix()} + if stats.CPU != nil { + v := stats.CPU.Utilization + sample.CPU = &v + } + if stats.Memory != nil && stats.Memory.Total > 0 { + v := float64(stats.Memory.Used) / float64(stats.Memory.Total) * 100 + sample.Mem = &v + } + for _, g := range stats.GPUs { + hg := metrics.HistoryGPU{Index: g.Index, Util: g.Utilization} + if g.MemoryTotal > 0 { + v := float64(g.MemoryUsed) / float64(g.MemoryTotal) * 100 + hg.Mem = &v + } + sample.GPUs = append(sample.GPUs, hg) + } + if sample.CPU == nil && sample.Mem == nil && len(sample.GPUs) == 0 { + return + } + d.hist.add(sample) +} diff --git a/internal/daemon/history_test.go b/internal/daemon/history_test.go new file mode 100644 index 0000000..504137e --- /dev/null +++ b/internal/daemon/history_test.go @@ -0,0 +1,351 @@ +//go:build !windows + +package daemon + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "math" + "os/exec" + "strings" + "testing" + "time" + + "github.com/spinloop-ai/spinloop/internal/metrics" + "github.com/spinloop-ai/spinloop/internal/remote" +) + +func f64(v float64) *float64 { return &v } + +// linuxCollector is a Collector stub reading a Linux host with two GPUs: the +// vmstat, free and nvidia-smi outputs in the shapes the parsers know. The +// expected figures: CPU 30%, memory 13.01% used, GPU 0 at 12% utilisation and +// 17.78% memory, GPU 1 at 97% and 88.89%. +func linuxCollector() *metrics.Collector { + return &metrics.Collector{ + GOOS: "linux", + Run: func(ctx context.Context, name string, args ...string) (string, error) { + switch name { + case "vmstat": + // The parser reads the last line's columns: r b swpd free buff + // cache si so bi bo in cs us sy id wa st — id at column 14. + return "procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----\n" + + " r b swpd free buff cache si so bi bo in cs us sy id wa st\n" + + " 1 0 0 947184 84224 590452 0 0 31 17 210 350 3 1 95 1 0\n" + + " 2 0 0 947184 84224 590452 0 0 0 0 180 300 20 5 70 5 0\n", nil + case "free": + return " total used free\n" + + "Mem: 33020416512 4294967296 12884901888\n", nil + case "nvidia-smi": + return "0, NVIDIA L40S, 12, 8192, 46080, 42\n" + + "1, NVIDIA L40S, 97, 40960, 46080, 71\n", nil + } + return "", errors.New("unexpected command " + name) + }, + } +} + +func almostEqual(a, b float64) bool { return math.Abs(a-b) < 0.01 } + +func TestSystemHistoryWindowAndLimit(t *testing.T) { + var h systemHistory + if got := h.snapshot(); got != nil { + t.Fatalf("a fresh buffer snapshots %d samples, want none", len(got)) + } + + add := func(at time.Time) { h.add(metrics.HistorySample{Time: at.Unix(), CPU: f64(10)}) } + add(baseTime) + add(baseTime.Add(15 * time.Second)) + if got := h.snapshot(); len(got) != 2 || got[0].Time != baseTime.Unix() || + got[1].Time != baseTime.Add(15*time.Second).Unix() { + t.Fatalf("snapshot = %+v, want oldest first", got) + } + + // What has aged out of the window is dropped by the next append, not by + // the snapshot: retention is a property of the buffer, not of a read. + add(baseTime.Add(11 * time.Minute)) + if got := h.snapshot(); len(got) != 1 || + got[0].Time != baseTime.Add(11*time.Minute).Unix() { + t.Errorf("after a jump past the window: %+v, want only the newest", got) + } + + // The limit holds the buffer however fast the sampler runs, keeping the + // newest samples. + h.clear() + for i := 0; i < historyLimit+10; i++ { + h.add(metrics.HistorySample{Time: baseTime.Add(time.Duration(i) * time.Second).Unix(), CPU: f64(10)}) + } + got := h.snapshot() + if len(got) != historyLimit { + t.Fatalf("a fast sampler left %d samples, want the limit of %d", len(got), historyLimit) + } + if got[len(got)-1].Time != baseTime.Add(time.Duration(historyLimit+9)*time.Second).Unix() { + t.Errorf("the limit dropped the newest sample: last = %+v", got[len(got)-1]) + } + + // A snapshot is a copy: a held reply does not move under later appends. + held := h.snapshot() + h.add(metrics.HistorySample{Time: baseTime.Add(2 * time.Hour).Unix(), CPU: f64(10)}) + if len(held) != historyLimit { + t.Errorf("a held snapshot changed from %d to %d samples", historyLimit, len(held)) + } + + h.clear() + if got := h.snapshot(); got != nil { + t.Errorf("clear left %d samples", len(got)) + } +} + +func TestSystemSampleOnce(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +while true; do sleep 0.05; done`) + d.Now = func() time.Time { return baseTime } + d.Collector = linuxCollector() + + // Nothing running: no reading is taken, however many times it is asked. + d.systemSampleOnce(context.Background()) + if got := d.hist.snapshot(); len(got) != 0 { + t.Fatalf("a reading was taken with no engine running: %+v", got) + } + + if err := d.Push(remote.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + defer d.Sup.Stop() + waitForState(t, d.Sup, StateRunning) + + d.systemSampleOnce(context.Background()) + got := d.hist.snapshot() + if len(got) != 1 { + t.Fatalf("after one tick: %d samples, want 1", len(got)) + } + s := got[0] + if s.Time != baseTime.Unix() { + t.Errorf("sample time = %d, want %d", s.Time, baseTime.Unix()) + } + if s.CPU == nil || !almostEqual(*s.CPU, 30) { + t.Errorf("sample cpu = %v, want 30", s.CPU) + } + if s.Mem == nil || !almostEqual(*s.Mem, 13.008) { + t.Errorf("sample mem = %v, want ~13.01", s.Mem) + } + if len(s.GPUs) != 2 { + t.Fatalf("sample gpus = %+v, want two", s.GPUs) + } + if s.GPUs[0].Index != 0 || s.GPUs[0].Util != 12 || s.GPUs[0].Mem == nil || + !almostEqual(*s.GPUs[0].Mem, 17.778) { + t.Errorf("gpu 0 = %+v", s.GPUs[0]) + } + if s.GPUs[1].Index != 1 || s.GPUs[1].Util != 97 || s.GPUs[1].Mem == nil || + !almostEqual(*s.GPUs[1].Mem, 88.889) { + t.Errorf("gpu 1 = %+v", s.GPUs[1]) + } + + // A second tick at a later time appends, oldest first. + d.Now = func() time.Time { return baseTime.Add(15 * time.Second) } + d.systemSampleOnce(context.Background()) + if got := d.hist.snapshot(); len(got) != 2 || got[1].Time != baseTime.Add(15*time.Second).Unix() { + t.Errorf("after a second tick: %+v, want the reading appended", got) + } +} + +// A failed reading is a non-observation: nothing is recorded and nothing is +// reported, because the on-request collection keeps its own error reporting +// and a transient sampling failure is not a condition worth surfacing per tick. +func TestSystemSampleOnceRecordsNothingOnFailure(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +while true; do sleep 0.05; done`) + d.Now = func() time.Time { return baseTime } + // Every host command is missing: the absent-source case. + d.Collector = &metrics.Collector{ + GOOS: "linux", + Run: func(ctx context.Context, name string, args ...string) (string, error) { return "", exec.ErrNotFound }, + } + if err := d.Push(remote.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + defer d.Sup.Stop() + waitForState(t, d.Sup, StateRunning) + + d.systemSampleOnce(context.Background()) + if got := d.hist.snapshot(); len(got) != 0 { + t.Errorf("a failed reading recorded a sample: %+v", got) + } + // And a reading that fails part-way — the GPU source there, the rest + // absent — records what it has rather than all or nothing. + d.Collector = &metrics.Collector{ + GOOS: "linux", + Run: func(ctx context.Context, name string, args ...string) (string, error) { + if name == "nvidia-smi" { + return "", exec.ErrNotFound + } + return linuxCollector().Run(ctx, name, args...) + }, + } + d.systemSampleOnce(context.Background()) + got := d.hist.snapshot() + if len(got) != 1 || got[0].CPU == nil || got[0].Mem == nil || len(got[0].GPUs) != 0 { + t.Errorf("a partial reading was not recorded as partial: %+v", got) + } +} + +func TestSystemHistoryClearsOnStartAndSurvivesAStop(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +while true; do sleep 0.05; done`) + now := baseTime + d.Now = func() time.Time { return now } + d.Collector = linuxCollector() + if err := d.Push(remote.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + + // A reading taken before the start must not be reported against the + // engine that starts: the clear goes through StartEngine, not the test. + d.hist.add(metrics.HistorySample{Time: now.Add(-time.Hour).Unix(), CPU: f64(99)}) + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + defer d.Sup.Stop() + waitForState(t, d.Sup, StateRunning) + if got := d.hist.snapshot(); len(got) != 0 { + t.Fatalf("the previous engine's reading survived the start: %+v", got) + } + + now = now.Add(2 * time.Minute) + d.systemSampleOnce(context.Background()) + if got := d.hist.snapshot(); len(got) != 1 { + t.Fatalf("after a tick: %d samples, want 1", len(got)) + } + + // The stop does not clear: the readings up to the stop say what the engine + // was doing until it stopped. + if err := d.Sup.Stop(); err != nil { + t.Fatal(err) + } + waitForState(t, d.Sup, StateStopped) + if got := d.hist.snapshot(); len(got) != 1 || got[0].Time != now.Unix() { + t.Errorf("a stop cleared the history: %+v", got) + } + + // The next engine clears it. + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + waitForState(t, d.Sup, StateRunning) + if got := d.hist.snapshot(); len(got) != 0 { + t.Errorf("the first engine's readings survived into the second: %+v", got) + } +} + +// TestMetricsExposesHistory covers what /v1/metrics says: the retained +// readings alongside the current ones, absent where none has been taken, and +// surviving a stop while the running-engine figures go. +func TestMetricsExposesHistory(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +while true; do sleep 0.05; done`) + d.Now = func() time.Time { return baseTime } + d.Collector = linuxCollector() + + // A daemon that has never run an engine says so by omission, not by an + // empty window. + stats := d.Metrics(context.Background()) + if stats.History != nil { + t.Errorf("a daemon that has served nothing reported history: %+v", stats.History) + } + if body, _ := json.Marshal(stats); bytes.Contains(body, []byte("history")) { + t.Errorf("absent history still serialised: %s", body) + } + + if err := d.Push(remote.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + defer d.Sup.Stop() + waitForState(t, d.Sup, StateRunning) + + d.systemSampleOnce(context.Background()) + stats = d.Metrics(context.Background()) + if len(stats.History) != 1 || stats.History[0].CPU == nil || !almostEqual(*stats.History[0].CPU, 30) { + t.Fatalf("metrics carried no usable history: %+v", stats.History) + } + + // The reply the control plane actually reads carries the compact field + // names, one letter each. + if body, _ := json.Marshal(stats); !strings.Contains(string(body), `"t":`) || + !strings.Contains(string(body), `"g":`) { + t.Errorf("the history did not serialise with its one-letter fields: %s", body) + } + + // Stopping drops the running-engine figures and keeps the history. + if err := d.Sup.Stop(); err != nil { + t.Fatal(err) + } + waitForState(t, d.Sup, StateStopped) + stats = d.Metrics(context.Background()) + if stats.CPU != nil || stats.Memory != nil || len(stats.GPUs) > 0 || stats.Tokens != nil { + t.Errorf("a stopped engine reported running-engine figures: %+v", stats) + } + if len(stats.History) != 1 { + t.Errorf("a stopped engine lost its history: %+v", stats.History) + } +} + +// The loop is what the deployment runs: while an engine runs, one system +// reading per tick, with no scrape target involved — and the readings stop +// when the engine stops, while the retention ends only at the next start. +func TestSamplerTakesSystemReadingsEachTick(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +while true; do sleep 0.05; done`) + clock := &fakeClock{t: baseTime} + d.Now = clock.now + d.Collector = linuxCollector() + // No scrape target is set: the system readings must not depend on one. + old := catchUpInterval + catchUpInterval = 5 * time.Millisecond + defer func() { catchUpInterval = old }() + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go d.SampleActivity(ctx) + + if err := d.Push(remote.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + defer d.Sup.Stop() + waitForState(t, d.Sup, StateRunning) + + deadline := time.Now().Add(3 * time.Second) + for time.Now().Before(deadline) { + if len(d.hist.snapshot()) >= 2 { + break + } + time.Sleep(5 * time.Millisecond) + } + if got := len(d.hist.snapshot()); got < 2 { + t.Fatalf("the sampler took %d system readings in 3s, want at least 2", got) + } + + // The stop ends the sampling but not the retention. + if err := d.Sup.Stop(); err != nil { + t.Fatal(err) + } + waitForState(t, d.Sup, StateStopped) + n := len(d.hist.snapshot()) + time.Sleep(50 * time.Millisecond) + if got := len(d.hist.snapshot()); got != n { + t.Errorf("a stopped engine kept accumulating readings: %d -> %d", n, got) + } +} diff --git a/internal/daemon/openapi_test.go b/internal/daemon/openapi_test.go index e45f173..609aacc 100644 --- a/internal/daemon/openapi_test.go +++ b/internal/daemon/openapi_test.go @@ -47,6 +47,8 @@ func schemaFor() map[string]any { "GpuStat": metrics.GpuStat{}, "CpuStat": metrics.CpuStat{}, "MemoryStat": metrics.MemoryStat{}, + "HistorySample": metrics.HistorySample{}, + "HistoryGPU": metrics.HistoryGPU{}, "DeployConfig": remote.DeployConfig{}, } } diff --git a/internal/fleet/remote_node.go b/internal/fleet/remote_node.go index b20f395..7501c3a 100644 --- a/internal/fleet/remote_node.go +++ b/internal/fleet/remote_node.go @@ -161,6 +161,7 @@ func statsFromRemote(resp remote.StatsResponse) metrics.Stats { GPUs: resp.GPUs, CPU: resp.CPU, Memory: resp.Memory, + History: resp.History, Errors: resp.Errors, LastActiveAt: resp.LastActiveAt, IdleSeconds: resp.IdleSeconds, diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index d483139..be14a88 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -32,6 +32,15 @@ type Stats struct { // idle value: read LastActiveAt to decide whether there is anything to // report, never IdleSeconds. // + // History is the retained system readings, oldest first: one entry per + // sampler tick while an engine ran, each carrying 0-100% figures rather + // than raw ones, because that is the axis the bar format draws and raw + // figures would double the wire size for nothing the graph uses. Absent + // where no reading has been taken, and retained across a stop — the + // readings up to the stop answer what the engine was doing until it + // stopped — so, unlike the running-engine figures above, it can be + // present while they are absent. + History []HistorySample `json:"history,omitempty"` // Unlike the figures above, these describe the engine whatever its state — // a stopped engine still reports when it last worked. LastActiveAt string `json:"lastActiveAt,omitempty"` @@ -74,3 +83,35 @@ type MemoryStat struct { Total int64 `json:"total"` Used int64 `json:"used"` } + +// HistorySample is one point of the retained system-reading history: when the +// reading was taken and the 0-100% figures the bar format plots for it. Every +// figure is optional on the same terms as Stats — a sample taken on a host +// with no source for one simply omits it — so a partial host yields a history +// of partial samples rather than an error. +// +// The JSON field names are one letter each because the history rides the +// remote relay over SSM, whose command output truncates at 4KB: forty +// samples of the window must fit that budget alongside the current reading, +// and at full names they would not. +type HistorySample struct { + // Time is when the reading was taken, unix seconds. + Time int64 `json:"t"` + // CPU is whole-host CPU utilization, percent. + CPU *float64 `json:"c,omitempty"` + // Mem is system memory used over total, percent. + Mem *float64 `json:"m,omitempty"` + // GPUs is each GPU's figures, percent. + GPUs []HistoryGPU `json:"g,omitempty"` +} + +// HistoryGPU is one GPU's figures in a history sample, one-letter fields as +// its parent. +type HistoryGPU struct { + Index int `json:"i"` + // Util is utilization, percent. + Util int `json:"u"` + // Mem is memory used over total, percent. Absent where the GPU reports + // no total. + Mem *float64 `json:"m,omitempty"` +} diff --git a/internal/remote/remote.go b/internal/remote/remote.go index 217e199..1da21cb 100644 --- a/internal/remote/remote.go +++ b/internal/remote/remote.go @@ -729,7 +729,12 @@ type StatsResponse struct { GPUs []GpuStat `json:"gpus"` CPU *CpuStat `json:"cpu"` Memory *MemoryStat `json:"memory"` - Errors []string `json:"errors"` + // History relays the daemon's retained system readings verbatim — the data + // the bar format draws. Nil for a daemon that has never run an engine, or + // a daemon that predates the field; the drawing falls back to the gauge + // for a series with no readings, per series. + History []metrics.HistorySample `json:"history,omitempty"` + Errors []string `json:"errors"` // LastActiveAt and IdleSeconds relay the on-instance daemon's answer to // "has this engine been working?", verbatim. Empty when the daemon was // unreachable, when no engine has run, or when the control plane predates diff --git a/openspec/changes/metrics-graph/tasks.md b/openspec/changes/metrics-graph/tasks.md index 907c524..57a300a 100644 --- a/openspec/changes/metrics-graph/tasks.md +++ b/openspec/changes/metrics-graph/tasks.md @@ -1,42 +1,42 @@ ## 1. Daemon history -- [ ] 1.1 Add the history shape to `internal/metrics`: per-sample time plus CPU, RAM, and per-GPU percentage readings, and a `History` field on `Stats` omitted when empty -- [ ] 1.2 Add the daemon's ring buffer (10-minute window at the sampler cadence) with append, snapshot, and clear operations -- [ ] 1.3 Take a system reading on each sampler tick while an engine runs (independent of a scrape target), record a sample on success, record nothing on failure -- [ ] 1.4 Clear the buffer on engine start alongside the existing counter/reset hooks; leave it untouched on stop -- [ ] 1.5 Include the buffer's contents in `Daemon.Metrics` so `/v1/metrics` reports the history -- [ ] 1.6 Unit tests: buffer wrap and ordering, sample on success / none on failure, clear on start, persistence across stop, exposure on the metrics reply +- [x] 1.1 Add the history shape to `internal/metrics`: per-sample time plus CPU, RAM, and per-GPU percentage readings, and a `History` field on `Stats` omitted when empty +- [x] 1.2 Add the daemon's ring buffer (10-minute window at the sampler cadence) with append, snapshot, and clear operations +- [x] 1.3 Take a system reading on each sampler tick while an engine runs (independent of a scrape target), record a sample on success, record nothing on failure +- [x] 1.4 Clear the buffer on engine start alongside the existing counter/reset hooks; leave it untouched on stop +- [x] 1.5 Include the buffer's contents in `Daemon.Metrics` so `/v1/metrics` reports the history +- [x] 1.6 Unit tests: buffer wrap and ordering, sample on success / none on failure, clear on start, persistence across stop, exposure on the metrics reply ## 2. API contract -- [ ] 2.1 Add the history field to the metrics response in `docs/openapi.yaml` -- [ ] 2.2 Confirm `openapi_test.go` passes with the new field +- [x] 2.1 Add the history field to the metrics response in `docs/openapi.yaml` +- [x] 2.2 Confirm `openapi_test.go` passes with the new field ## 3. CLI rendering -- [ ] 3.1 Add the sparkline renderer: eight block glyphs, max-pool downsampling to the draw width, latest point coloured on the 80/90 thresholds, trailing percentage -- [ ] 3.2 Rename the existing filled-bar renderer to the gauge role and route `--format` on both `remote metrics` and `fleet metrics` across `bar`, `gauge`, `table`, and `json` -- [ ] 3.3 Draw the bar format from the daemon's history for the same series and labels the gauge draws, with the gauge-style fallback where a daemon reports no history -- [ ] 3.4 Keep the bar format drawing the retained history for a stopped engine (ending at the stop), with the existing last-active and header behaviour -- [ ] 3.5 Tests: renderer glyphs, downsampling keeps peaks, colour thresholds on the last point, format validation errors, no-history fallback, stopped-engine output +- [x] 3.1 Add the sparkline renderer: eight block glyphs, max-pool downsampling to the draw width, latest point coloured on the 80/90 thresholds, trailing percentage +- [x] 3.2 Rename the existing filled-bar renderer to the gauge role and route `--format` on both `remote metrics` and `fleet metrics` across `bar`, `gauge`, `table`, and `json` +- [x] 3.3 Draw the bar format from the daemon's history for the same series and labels the gauge draws, with the gauge-style fallback where a daemon reports no history +- [x] 3.4 Keep the bar format drawing the retained history for a stopped engine (ending at the stop), with the existing last-active and header behaviour +- [x] 3.5 Tests: renderer glyphs, downsampling keeps peaks, colour thresholds on the last point, format validation errors, no-history fallback, stopped-engine output ## 4. Dashboard -- [ ] 4.1 Add the board-wide format state and the `g` toggle to the dashboard model, opening in bar, with the key help naming it -- [ ] 4.2 Draw each tile's resource series in the board's current format from the node's history, reusing the fallback for nodes whose daemon reports none -- [ ] 4.3 Tests: toggle switches every tile and back, tiles keep their geometry in both formats, nodes without history fall back +- [x] 4.1 Add the board-wide format state and the `g` toggle to the dashboard model, opening in bar, with the key help naming it +- [x] 4.2 Draw each tile's resource series in the board's current format from the node's history, reusing the fallback for nodes whose daemon reports none +- [x] 4.3 Tests: toggle switches every tile and back, tiles keep their geometry in both formats, nodes without history fall back ## 5. Remote relay -- [ ] 5.1 Add the history types to `remote/lambda/shared/daemon.ts` and `shared/stats.ts` -- [ ] 5.2 Copy the history field through in the stats Lambda, unaltered -- [ ] 5.3 TypeScript tests covering the relay of the field and its absence +- [x] 5.1 Add the history types to `remote/lambda/shared/daemon.ts` and `shared/stats.ts` +- [x] 5.2 Copy the history field through in the stats Lambda, unaltered +- [x] 5.3 TypeScript tests covering the relay of the field and its absence ## 6. Documentation -- [ ] 6.1 Update the command reference under `docs/` for the `bar`/`gauge` formats and the dashboard's `g` key +- [x] 6.1 Update the command reference under `docs/` for the `bar`/`gauge` formats and the dashboard's `g` key ## 7. Verification -- [ ] 7.1 `gofmt`, `go vet ./...`, and `go test ./... -cover` with total coverage at or above 80% -- [ ] 7.2 The `remote/` pnpm suite passes +- [x] 7.1 `gofmt`, `go vet ./...`, and `go test ./... -cover` with total coverage at or above 80% +- [x] 7.2 The `remote/` pnpm suite passes diff --git a/remote/lambda/shared/daemon.ts b/remote/lambda/shared/daemon.ts index 4560aa7..3f3a72d 100644 --- a/remote/lambda/shared/daemon.ts +++ b/remote/lambda/shared/daemon.ts @@ -6,7 +6,7 @@ * (spinloop's internal/metrics); the Lambdas only relay its JSON. */ -import type { CpuStat, GpuStat, MemoryStat, TokenStats } from './stats'; +import type { CpuStat, GpuStat, HistorySample, MemoryStat, TokenStats } from './stats'; /** Where the daemon listens on the instance. Loopback: only SSM reaches it. */ export const DAEMON_API = 'http://127.0.0.1:4242'; @@ -50,6 +50,13 @@ export interface DaemonMetrics { gpus?: GpuStat[]; cpu?: CpuStat; memory?: MemoryStat; + /** + * The daemon's retained system readings, relayed verbatim — the data the + * bar format draws. Absent for a daemon that predates the field or has + * never run an engine; they survive a stop, so the field can be present + * while the running-engine figures above are not. + */ + history?: HistorySample[]; errors?: string[]; /** * The same activity pair `/v1/status` reports, from the same record on the diff --git a/remote/lambda/shared/stats.ts b/remote/lambda/shared/stats.ts index 18ef295..2b6de34 100644 --- a/remote/lambda/shared/stats.ts +++ b/remote/lambda/shared/stats.ts @@ -39,6 +39,33 @@ export interface TokenStats { requests: number; } +/** + * One GPU's figures in a retained system reading, relayed verbatim from the + * daemon (spinloop's metrics.HistoryGPU). One-letter fields, like its parent: + * the readings cross SSM, whose command output truncates at 4KB, so the + * window's forty samples must fit that budget alongside the current reading. + */ +export interface HistoryGPU { + /** The GPU's index. */ + i: number; + /** Utilisation, percent. */ + u: number; + /** Memory used over total, percent. Absent where the GPU reports no total. */ + m?: number; +} + +/** One retained system reading, as the bar format plots it: percent per series. */ +export interface HistorySample { + /** When the reading was taken, unix seconds. */ + t: number; + /** Whole-host CPU utilisation, percent. */ + c?: number; + /** System memory used over total, percent. */ + m?: number; + /** Each GPU's figures, percent. */ + g?: HistoryGPU[]; +} + export interface StatsResult { /** Environment name. */ environment: string; @@ -62,6 +89,14 @@ export interface StatsResult { cpu?: CpuStat; /** System memory stats. */ memory?: MemoryStat; + /** + * The daemon's retained system readings, oldest first — one per sampler + * tick while an engine ran, covering at most the last 10 minutes. They + * survive a stop and clear when the next engine starts. Absent for a daemon + * that predates the field or has never run an engine: the bar format falls + * back to the gauge for a series with no readings. + */ + history?: HistorySample[]; /** Any errors encountered while collecting metrics. */ errors?: string[]; /** When the engine last did any work, RFC 3339, as the daemon reports it. */ diff --git a/remote/lambda/stats/index.ts b/remote/lambda/stats/index.ts index 57ffdc0..15f7eee 100644 --- a/remote/lambda/stats/index.ts +++ b/remote/lambda/stats/index.ts @@ -99,6 +99,10 @@ export async function handler(event: LambdaFunctionURLEvent): Promise { }); }); +describe('stats relays the daemon’s retained history', () => { + it('carries it through unchanged', async () => { + const history = [ + { t: 1785000000, c: 12.5, m: 37.5, g: [{ i: 0, u: 88, m: 51.3 }] }, + { t: 1785000015, c: 62, m: 37.5, g: [{ i: 0, u: 95 }] }, + ]; + stubDaemon(JSON.stringify({ state: 'running', cpu: { utilization: 62 }, history })); + + const body = bodyOf(await handler(statsEvent)); + expect(body.history).toEqual(history); + expect(body.cpu).toBeDefined(); + }); + + it('leaves it absent when the daemon predates the field', async () => { + stubDaemon(JSON.stringify({ state: 'running', cpu: { utilization: 5 } })); + + const body = bodyOf(await handler(statsEvent)); + expect(body).not.toHaveProperty('history'); + expect(body.cpu).toBeDefined(); + }); + + it('leaves it absent when the daemon is unreachable', async () => { + runShellCommand.mockResolvedValue({ status: 'Success', stdout: `${DAEMON_UNREACHABLE}\n` }); + + const body = bodyOf(await handler(statsEvent)); + expect(body).not.toHaveProperty('history'); + expect(body.errors).toContain('daemon: unreachable or unrecognisable metrics reply'); + }); +}); + describe('stats relays the daemon’s version', () => { it('carries it through unchanged', async () => { stubDaemon( diff --git a/remote/test/stats.test.ts b/remote/test/stats.test.ts index ab7a62f..c08061f 100644 --- a/remote/test/stats.test.ts +++ b/remote/test/stats.test.ts @@ -56,10 +56,45 @@ describe('parseDaemonMetrics', () => { expect(parsed).not.toBeNull(); expect(parsed!.tokens).toBeUndefined(); expect(parsed!.gpus).toBeUndefined(); + expect(parsed!.history).toBeUndefined(); expect(parsed!.lastActiveAt).toBeUndefined(); expect(parsed!.idleSeconds).toBeUndefined(); }); + it('parses the retained history the daemon reports', () => { + const parsed = parseDaemonMetrics( + JSON.stringify({ + state: 'running', + cpu: { utilization: 62 }, + history: [ + { t: 1785000000, c: 12.5, m: 37.5, g: [{ i: 0, u: 88, m: 51.3 }] }, + { t: 1785000015, c: 62, m: 37.5, g: [{ i: 0, u: 95 }] }, + ], + }), + ); + expect(parsed).not.toBeNull(); + expect(parsed!.history).toEqual([ + { t: 1785000000, c: 12.5, m: 37.5, g: [{ i: 0, u: 88, m: 51.3 }] }, + { t: 1785000015, c: 62, m: 37.5, g: [{ i: 0, u: 95 }] }, + ]); + }); + + it('parses a stopped engine whose history survives the stop', () => { + // The readings up to the stop are the point of the retention: they arrive + // without any of the running-engine figures beside them. + const parsed = parseDaemonMetrics( + JSON.stringify({ + state: 'stopped', + history: [{ t: 1785000015, c: 62, m: 37.5, g: [{ i: 0, u: 95 }] }], + lastActiveAt: '2026-08-09T12:00:00Z', + idleSeconds: 600, + }), + ); + expect(parsed).not.toBeNull(); + expect(parsed!.cpu).toBeUndefined(); + expect(parsed!.history).toHaveLength(1); + }); + it('parses a stopped engine that still reports when it last worked', () => { // The record survives a stop, so this pair arrives without any of the // running-engine figures beside it. From 4ab698bf88c80b099669acadedf56d54fa83e6eb Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Sun, 6 Sep 2026 01:45:45 +0100 Subject: [PATCH 3/7] test(daemon): stop the sampler before restoring the catch-up cadence The loop reads the cadence on every tick while it has no reading to report, and a daemon with no scrape target never gets one, so the restore ran against a running sampler. Cancel and wait for the sampler to exit first. --- internal/daemon/history_test.go | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/internal/daemon/history_test.go b/internal/daemon/history_test.go index 504137e..b13b6d0 100644 --- a/internal/daemon/history_test.go +++ b/internal/daemon/history_test.go @@ -312,11 +312,21 @@ while true; do sleep 0.05; done`) // No scrape target is set: the system readings must not depend on one. old := catchUpInterval catchUpInterval = 5 * time.Millisecond - defer func() { catchUpInterval = old }() ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - go d.SampleActivity(ctx) + done := make(chan struct{}) + go func() { + d.SampleActivity(ctx) + close(done) + }() + // The loop reads the catch-up cadence on every tick while it has no + // reading to report — with no scrape target, always — so the sampler + // must have exited before the cadence is restored. + defer func() { + cancel() + <-done + catchUpInterval = old + }() if err := d.Push(remote.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { t.Fatal(err) From 5f1f5a7b28297dbcdfec9f4ecba6311690482efd Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Sun, 6 Sep 2026 02:14:06 +0100 Subject: [PATCH 4/7] fix: keep the newest sample in the sparkline's final column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit At the tile's draw width a 29- or 30-sample window splits unevenly, and the split's rounding can leave the newest reading outside the final pool — so the line and its trailing figure drew one sample behind. The final column now takes the series to its end, which the arithmetic otherwise guarantees. --- cmd/spinloop/metrics_render.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/cmd/spinloop/metrics_render.go b/cmd/spinloop/metrics_render.go index 9c73d3d..fee496e 100644 --- a/cmd/spinloop/metrics_render.go +++ b/cmd/spinloop/metrics_render.go @@ -100,7 +100,10 @@ func poolMax(values []float64, width int) []float64 { for c := range out { lo := int(float64(c) * per) hi := int(float64(c+1) * per) - if hi > len(values) { + if c == width-1 { + // The split's rounding can leave the newest sample outside the + // final column, and the trailing figure is that sample's value — + // so the last column takes it whatever the arithmetic says. hi = len(values) } m := values[lo] From 0bb07076353b433a4d9104aa27ed994fd494cd23 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Sun, 6 Sep 2026 02:14:06 +0100 Subject: [PATCH 5/7] test: cover the fleet entry point, the stats relay, and the renderer's edge cases fleet metrics now draws a node's retained history, a stopped node's history alone, and the per-series gauge fallback, through the command itself; statsFromRemote is checked to relay the history unaltered and to keep an absence an absence. The renderer's tests add the 29-sample-at-tile-width regression, a GPU present in only some samples, a gauge value beyond 100, and a memory reading with no total. --- cmd/spinloop/fleet_test.go | 79 +++++++++++++++++++++++++++++ cmd/spinloop/metrics_render_test.go | 57 +++++++++++++++++++++ internal/fleet/remote_node_test.go | 13 +++++ 3 files changed, 149 insertions(+) diff --git a/cmd/spinloop/fleet_test.go b/cmd/spinloop/fleet_test.go index bd5c3be..45a44a8 100644 --- a/cmd/spinloop/fleet_test.go +++ b/cmd/spinloop/fleet_test.go @@ -209,6 +209,85 @@ func TestCmdFleetMetricsRejectsBadFormat(t *testing.T) { } } +// The fleet's entry point draws what the daemons report: a running node's +// retained readings as bars with the per-series gauge fallback, a stopped +// node's history on its own, and the gauge format the current reading only. +func TestCmdFleetMetricsDrawsHistory(t *testing.T) { + t.Setenv("SPINLOOP_CONFIG_DIR", t.TempDir()) + running := metricsOnlyDaemon(t, map[string]any{ + "state": "running", + "modelId": "org/qwen", + "cpu": map[string]any{"utilization": 50.0}, + "memory": map[string]any{"total": 1000, "used": 400}, + // The daemon's compact one-letter sample fields, as /v1/metrics sends + // them. CPU has retained readings; RAM does not. + "history": []map[string]any{ + {"t": 1786276800, "c": 30.0}, + {"t": 1786276815, "c": 50.0}, + }, + }) + t.Cleanup(running.Close) + stopped := metricsOnlyDaemon(t, map[string]any{ + "state": "stopped", "modelId": "org/qwen", + "lastActiveAt": "2026-08-21T10:00:00Z", "idleSeconds": 12, + "history": []map[string]any{ + {"t": 1786276800, "c": 30.0, "g": []map[string]any{{"i": 0, "u": 50, "m": 50.0}}}, + {"t": 1786276815, "c": 50.0, "g": []map[string]any{{"i": 0, "u": 60, "m": 50.0}}}, + }, + }) + t.Cleanup(stopped.Close) + upHost, upPort := hostPort(t, running) + downHost, downPort := hostPort(t, stopped) + writeFleetFile(t, fmt.Sprintf( + "nodes:\n - name: up\n host: %s\n port: %d\n - name: halted\n host: %s\n port: %d\n", + upHost, upPort, downHost, downPort)) + + out := captureStdout(t, func() { + if err := cmdFleet([]string{"metrics"}); err != nil { + t.Error(err) + } + }) + // up: the CPU series drew its history, RAM fell back to the gauge on the + // same screen. + if !strings.Contains(out, " 50%") || !strings.Contains(out, "▃") { + t.Errorf("up's CPU did not draw its history:\n%s", out) + } + if !strings.Contains(out, "░") { + t.Errorf("up's RAM did not fall back to the gauge:\n%s", out) + } + // halted: a stopped node's bar draws the retained readings alone, + // including the GPU series the current reading no longer names. + if !strings.Contains(out, "halted stopped org/qwen") || !strings.Contains(out, "GPU util") { + t.Errorf("halted's history not drawn:\n%s", out) + } + + // The gauge format draws the current reading only: up's CPU gauge shows + // 50 with no sparkline, and halted draws nothing after its header. + out = captureStdout(t, func() { + if err := cmdFleet([]string{"metrics", "--format=gauge"}); err != nil { + t.Error(err) + } + }) + if !strings.Contains(out, " 50%") || strings.Contains(out, "▃") { + t.Errorf("gauge drew history or missed the current reading:\n%s", out) + } + halted := out[strings.Index(out, "halted"):] + if strings.Contains(halted, "GPU util") || strings.Contains(halted, " 60%") { + t.Errorf("a stopped node drew series in the gauge format:\n%s", halted) + } +} + +// metricsOnlyDaemon serves a daemon control API that answers /v1/metrics with +// the given body and nothing else. +func metricsOnlyDaemon(t *testing.T, body map[string]any) *httptest.Server { + t.Helper() + mux := http.NewServeMux() + mux.HandleFunc("GET /v1/metrics", func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(body) + }) + return httptest.NewServer(mux) +} + func TestCmdFleetStartStopDriveOneNode(t *testing.T) { twoNodeFleet(t, "idle") diff --git a/cmd/spinloop/metrics_render_test.go b/cmd/spinloop/metrics_render_test.go index 463b5c3..ecf2a97 100644 --- a/cmd/spinloop/metrics_render_test.go +++ b/cmd/spinloop/metrics_render_test.go @@ -38,6 +38,23 @@ func TestPoolMaxKeepsPeaks(t *testing.T) { if got := poolMax(in, 5); len(got) != 3 || got[0] != 1 || got[1] != 2 || got[2] != 3 { t.Errorf("an unwidened series changed: %v", got) } + // An uneven split pools the same values, one column each. + got = poolMax([]float64{1, 5, 2, 9, 3, 4, 8}, 3) + if len(got) != 3 || got[0] != 5 || got[1] != 9 || got[2] != 8 { + t.Errorf("uneven pool = %v, want [5 9 8]", got) + } + // The newest sample lands in the final column: at the tile's width a + // 29-sample window splits unevenly, and the split's rounding leaves the + // last reading out of that column — and the trailing figure is the last + // reading's value, so it must be in the pool. The values rise, so the + // last column's maximum is its newest sample only if the sample is there. + vals := make([]float64, 29) + for i := range vals { + vals[i] = float64(i + 1) + } + if got := poolMax(vals, 25); got[24] != 29 { + t.Errorf("the newest sample dropped out of the last pool: %v, want 29", got[24]) + } } func TestRenderSparkline(t *testing.T) { @@ -84,6 +101,13 @@ func TestRenderGauge(t *testing.T) { if got := b.String(); got != want { t.Errorf("gauge = %q, want %q", got, want) } + // A value beyond 100 fills the gauge rather than spilling past it. + b.Reset() + renderGauge(&b, "CPU", 150) + want = " CPU " + ansiRed + strings.Repeat("█", 25) + ansiReset + " 150%\n" + if got := b.String(); got != want { + t.Errorf("out-of-range gauge = %q, want %q", got, want) + } } func TestValidateMetricsFormat(t *testing.T) { @@ -198,6 +222,39 @@ func TestRenderStatBarsStoppedEngineDrawsHistoryAlone(t *testing.T) { } } +// A memory reading with no total reports 0, not a division by it. +func TestBarSeriesListMemoryWithoutTotal(t *testing.T) { + var b bytes.Buffer + renderStatBars(&b, nil, &metrics.MemoryStat{Total: 0, Used: 100}, nil, nil, barLineW) + if !strings.Contains(b.String(), " 0%") || strings.Contains(b.String(), "NaN") { + t.Errorf("a memory reading with no total: %q", b.String()) + } +} + +// A GPU that appears in only some samples: each of its series draws the +// samples that carry it, and the pick yields nothing for the rest. +func TestRenderStatBarsGPUInOnlySomeSamples(t *testing.T) { + history := []metrics.HistorySample{ + {Time: 1, GPUs: []metrics.HistoryGPU{{Index: 0, Util: 10, Mem: ptrPct(20)}}}, + {Time: 2, GPUs: []metrics.HistoryGPU{{Index: 1, Util: 90, Mem: ptrPct(80)}}}, + } + var b bytes.Buffer + renderStatBars(&b, nil, nil, nil, history, barLineW) + out := b.String() + // Both GPUs are named in the union, so all four series draw. + for _, want := range []string{"GPU 0 util", "GPU 1 util", "GPU 0 mem", "GPU 1 mem"} { + if !strings.Contains(out, want) { + t.Fatalf("series %q not drawn: %q", want, out) + } + } + // Each series drew exactly the one sample that carried it. + for _, want := range []string{" 10%", " 90%", " 20%", " 80%"} { + if !strings.Contains(out, want) { + t.Errorf("series missing its sample's value %s: %q", want, out) + } + } +} + // The gauge format draws the current reading only, whatever the history // holds — and a stopped engine's reading carries nothing, so it draws none. func TestRenderStatGaugesIgnoresHistory(t *testing.T) { diff --git a/internal/fleet/remote_node_test.go b/internal/fleet/remote_node_test.go index 9ffea62..6d8ec0f 100644 --- a/internal/fleet/remote_node_test.go +++ b/internal/fleet/remote_node_test.go @@ -77,9 +77,12 @@ func TestStatusFromRemote(t *testing.T) { func TestStatsFromRemote(t *testing.T) { tokens := &metrics.TokenStats{Running: 2, PromptTokens: 5, GenerationTokens: 7, Requests: 3} + cpuPct := 30.0 + history := []metrics.HistorySample{{Time: 1, CPU: &cpuPct, GPUs: []metrics.HistoryGPU{{Index: 0, Util: 61, Mem: &cpuPct}}}} got := statsFromRemote(remote.StatsResponse{ State: "running", Runner: "llamacpp", ModelID: "org/m", UptimeSeconds: 10, Tokens: tokens, LastActiveAt: "2026-01-02T00:00:00Z", IdleSeconds: 5, Version: "1.2.3", + History: history, }) if got.State != "running" || got.Runner != "llamacpp" || got.ModelID != "org/m" || got.UptimeSeconds != 10 { t.Errorf("statsFromRemote = %+v", got) @@ -90,6 +93,16 @@ func TestStatsFromRemote(t *testing.T) { if got.IdleSeconds != 5 || got.LastActiveAt == "" { t.Errorf("activity not carried over: %+v", got) } + // The daemon's retained readings relay through unaltered: the fleet does + // not own them, so it copies them rather than rebuilding them. + if len(got.History) != 1 || got.History[0].Time != 1 || got.History[0].CPU == nil || + *got.History[0].CPU != 30 || len(got.History[0].GPUs) != 1 || got.History[0].GPUs[0].Util != 61 { + t.Errorf("history not carried over: %+v", got.History) + } + // And a reply without them stays without them. + if got := statsFromRemote(remote.StatsResponse{State: "running"}); got.History != nil { + t.Errorf("an absent history became present: %+v", got.History) + } } func TestLogsFromRemote(t *testing.T) { From 7ff6024b4a25da0169f23c12153b62f0440a2d8d Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Sun, 6 Sep 2026 03:23:12 +0100 Subject: [PATCH 6/7] docs(openspec): sync the metrics-graph specs into the main specs --- openspec/specs/daemon-api/spec.md | 37 ++++++++ openspec/specs/engine-activity/spec.md | 37 ++++++++ openspec/specs/fleet-client/spec.md | 50 +++++++++-- .../specs/remote-metrics-bar-format/spec.md | 86 ++++++++++++++----- openspec/specs/remote-stats/spec.md | 30 ++++++- 5 files changed, 206 insertions(+), 34 deletions(-) diff --git a/openspec/specs/daemon-api/spec.md b/openspec/specs/daemon-api/spec.md index f68f813..fcf878b 100644 --- a/openspec/specs/daemon-api/spec.md +++ b/openspec/specs/daemon-api/spec.md @@ -448,3 +448,40 @@ mistaken for a stuck one. - **WHEN** a status or metrics request is made while no engine is running - **THEN** the response carries no readiness field + +### Requirement: Metrics endpoint reports the system-reading history + +The metrics endpoint SHALL include, alongside the current readings, the +history of system readings the daemon retains: one entry per sampler tick +while an engine was running, each carrying its time and that tick's CPU, +memory, and per-GPU utilisation and memory readings. The history SHALL cover +at most the last 10 minutes at the sampler's cadence. It SHALL persist when +the engine stops — the readings taken before the stop remain, so a caller can +see what the engine was doing until it stopped — and SHALL be cleared when +the next engine starts, so one engine's readings are never reported against +another. Where no engine has run in this daemon's life, or no reading has been +taken, the field SHALL be omitted rather than empty. + +#### Scenario: History grows while the engine runs + +- **WHEN** an engine has been running for several sampler ticks and a metrics + request is made +- **THEN** the response includes one history entry per tick, each stamped with + its time, covering up to the last 10 minutes + +#### Scenario: A stopped engine still reports its history + +- **WHEN** a metrics request is made after the engine has been stopped +- **THEN** the response still includes the readings taken before the stop, + though the current running-engine figures are omitted + +#### Scenario: A new engine clears the previous history + +- **WHEN** an engine is stopped and a later engine is started, and a metrics + request is made +- **THEN** the history holds only the later engine's readings + +#### Scenario: No history yet is absent, not empty + +- **WHEN** a metrics request is made on a daemon that has never run an engine +- **THEN** the response carries no history field diff --git a/openspec/specs/engine-activity/spec.md b/openspec/specs/engine-activity/spec.md index 1d9e4f6..291f09e 100644 --- a/openspec/specs/engine-activity/spec.md +++ b/openspec/specs/engine-activity/spec.md @@ -118,3 +118,40 @@ start time as activity. - **WHEN** the daemon has been running but no engine has ever been started - **THEN** no last-active time is reported + +### Requirement: System readings for the retained history + +While an engine is running, the sampler SHALL take one system reading — the +host's CPU, memory, and GPU figures — on each tick at its own interval, +independently of any request to the control API and independently of whether a +scrape target for the engine's counters is known. Each reading SHALL be +retained in the history the metrics endpoint reports, for at most the last +10 minutes. A failed system reading SHALL record no sample for its tick and +SHALL NOT be reported as an error: the on-request collection keeps its own +error reporting, and a transient sampling failure is neither data nor a +condition worth surfacing on every tick. + +#### Scenario: System readings happen without being asked + +- **WHEN** an engine is running and no client calls the control API +- **THEN** the daemon still takes a system reading on each sampler tick and + retains it + +#### Scenario: System readings do not depend on a scrape target + +- **WHEN** the running engine exposes no metrics endpoint to scrape +- **THEN** the system readings are still taken and retained, since they come + from the host, not from the engine + +#### Scenario: A failed system reading records nothing + +- **WHEN** a system reading fails on a tick because a host command is missing + or fails +- **THEN** no sample is recorded for that tick and no error is reported for + it + +#### Scenario: Reading stops with the engine, retention does not end + +- **WHEN** the engine is stopped +- **THEN** no further system readings are taken, and the readings taken before + the stop remain retained until the next engine starts diff --git a/openspec/specs/fleet-client/spec.md b/openspec/specs/fleet-client/spec.md index 288b843..d6759b6 100644 --- a/openspec/specs/fleet-client/spec.md +++ b/openspec/specs/fleet-client/spec.md @@ -74,18 +74,24 @@ render, and the command SHALL succeed. ### Requirement: Fleet metrics `spinloop fleet metrics` SHALL query every node's metrics endpoint and render -each node's engine and system metrics using the same bar, table, and json -formats `spinloop remote metrics` provides, selected by `--format`. Unreachable -nodes SHALL be reported as in status rather than omitted. The command SHALL -support a `--watch`/`-w` mode that refreshes on an interval, clearing and -redrawing the screen in place with no scrollback accumulation, and exiting -cleanly on interrupt. +each node's engine and system metrics using the same bar, gauge, table, and +json formats `spinloop remote metrics` provides, selected by `--format`. +Unreachable nodes SHALL be reported as in status rather than omitted. The +command SHALL support a `--watch`/`-w` mode that refreshes on an interval, +clearing and redrawing the screen in place with no scrollback accumulation, +and exiting cleanly on interrupt. #### Scenario: Bar format per node - **WHEN** `spinloop fleet metrics` runs without `--format` - **THEN** each reachable node's metrics render in bar format under its name +#### Scenario: Gauge format per node + +- **WHEN** `spinloop fleet metrics --format=gauge` runs +- **THEN** each reachable node's resource series render in gauge format under + its name + #### Scenario: JSON aggregates the fleet - **WHEN** `spinloop fleet metrics --format=json` runs @@ -597,9 +603,12 @@ Each panel SHALL show, for a node that answered the last completed refresh, the same facts the bar format of `fleet metrics` renders for that node: its state, what it serves (runner and model when known), how long since it last did work (with the same labelling rules as the rest of the fleet surfaces), its resource -usage, and its token and request counters. A panel SHALL show the answer of the -last completed refresh for that node — not a mix of refreshes and not a stale -bar with a fresh outcome. +usage, and its token and request counters. A panel SHALL draw the node's +resource series in the board's current format — bar by default — from the +history the node's daemon reports, falling back per the bar format's no-history +rule where it reports none. A panel SHALL show the answer of the last completed +refresh for that node — not a mix of refreshes and not a stale bar with a fresh +outcome. A panel SHALL degrade gracefully when a node answers with fewer facts (no system stats, no GPUs, an engine that is not running) rather than failing to render. @@ -1285,3 +1294,26 @@ be left as it is. - **THEN** the selected panel's border colour changes as it does today, and every panel's status glyph colour is unaffected by which panel is selected +### Requirement: Dashboard format toggle + +The dashboard SHALL provide a key, `g`, that toggles the resource series of +every panel between bar and gauge. The board SHALL open in bar. The toggle +SHALL be board-wide — one format for every panel — rather than per node, and +the key help line SHALL name it. + +#### Scenario: Pressing the key switches every panel + +- **WHEN** the operator presses `g` on the grid +- **THEN** every panel's resource series redraws in the other format, and + pressing `g` again returns them + +#### Scenario: The board opens in bar + +- **WHEN** the dashboard opens +- **THEN** the panels draw the resource series in bar format + +#### Scenario: The key help names the toggle + +- **WHEN** the dashboard draws its key help line +- **THEN** it names `g` as the format toggle + diff --git a/openspec/specs/remote-metrics-bar-format/spec.md b/openspec/specs/remote-metrics-bar-format/spec.md index 6758bb3..01f3a2b 100644 --- a/openspec/specs/remote-metrics-bar-format/spec.md +++ b/openspec/specs/remote-metrics-bar-format/spec.md @@ -6,22 +6,22 @@ Define the bar graph output format for `spinloop remote metrics` with colour-cod ## Requirements ### Requirement: Bar format output -The system SHALL support a `--format=bar` option that renders resource metrics as horizontal progress bars. Each bar SHALL consist of a left-aligned label, a filled portion using block characters, an unfilled portion using light shade characters, and a right-aligned percentage value. +The system SHALL support a `--format=bar` option that renders each resource series as a sparkline drawn from the history the on-instance daemon retains: a left-aligned label, one glyph per sample, and the latest value as a right-aligned percentage. The glyphs SHALL be Unicode block elements of one grade per utilisation level, so a series reads as a line of bars across the window. The series drawn SHALL be the same set the gauge format draws: CPU, RAM, and each GPU's utilisation and memory, with the same per-GPU labelling. #### Scenario: Bar format displays CPU utilization -- **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance that has CPU data -- **THEN** the output includes a bar labelled "CPU" with filled and unfilled segments proportional to the utilization percentage +- **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance that has CPU data and a retained history +- **THEN** the output includes a row labelled "CPU" whose glyphs are the sampled CPU utilisation across the window and whose trailing figure is the latest sample's percentage #### Scenario: Bar format displays RAM utilization - **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance that has memory data -- **THEN** the output includes a bar labelled "RAM" with filled and unfilled segments proportional to the used/total memory ratio +- **THEN** the output includes a row labelled "RAM" whose glyphs are the sampled used/total memory ratio across the window and whose trailing figure is the latest ratio #### Scenario: Bar format displays GPU utilization - **WHEN** the user runs `spinloop remote metrics --format=bar` with a running instance that has GPU data -- **THEN** the output includes bars labelled "GPU util" and "GPU mem" (or "GPU N util"/"GPU N mem" for multiple GPUs) with filled and unfilled segments proportional to their respective utilization +- **THEN** the output includes rows labelled "GPU util" and "GPU mem" (or "GPU N util"/"GPU N mem" for multiple GPUs), each drawn from the retained history #### Scenario: Bar format header line @@ -30,22 +30,22 @@ The system SHALL support a `--format=bar` option that renders resource metrics a ### Requirement: Colour thresholds -The bar fill SHALL be colour-coded based on utilization: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%. The colour SHALL be reset after the filled portion so the unfilled characters and percentage appear in the terminal's default colour. +The sparkline's latest point SHALL be colour-coded based on utilization: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%. Every earlier point SHALL appear in the terminal's default colour, so the coloured point is the one to read. #### Scenario: Green bar for low utilization -- **WHEN** a metric value is 70% -- **THEN** the bar fill appears in green +- **WHEN** the latest sample of a series is 70% +- **THEN** the sparkline's final glyph appears in green and the earlier glyphs appear in the terminal's default colour #### Scenario: Yellow bar for high utilization -- **WHEN** a metric value is 85% -- **THEN** the bar fill appears in yellow +- **WHEN** the latest sample of a series is 85% +- **THEN** the sparkline's final glyph appears in yellow #### Scenario: Red bar for critical utilization -- **WHEN** a metric value is 95% -- **THEN** the bar fill appears in red +- **WHEN** the latest sample of a series is 95% +- **THEN** the sparkline's final glyph appears in red ### Requirement: Bar format is default @@ -58,22 +58,22 @@ The system SHALL use bar format as the default output when no `--format` flag is ### Requirement: Bar format with stopped instance -When the instance is not running, bar format SHALL show the header line with environment, state, instance type, and model — but SHALL NOT display resource bars. When a last-active time is known it SHALL still be shown, in the same -place it occupies for a running instance: the resource bars describe a running -engine and have nothing to say about a stopped one, but when work last -happened is exactly what a stopped endpoint is worth asking about. +When the instance is not running, bar format SHALL show the header line with environment, state, instance type, and model, and — where the daemon's retained history survives the stop — the series drawn from it, ending at the stop. The retained history answers "what was this engine doing until it stopped", the same question the last-active figure answers, and the header already carries the state. When no history is available, the format SHALL fall back to the gauge drawing of the current reading per the no-history rule — which for a stopped engine, whose current reading carries no resource figures, means no resource series at all. When a last-active time is known it SHALL still be shown, in the same place it occupies for a running instance. #### Scenario: Stopped instance shows header only -- **WHEN** the user runs `spinloop remote metrics --format=bar` and the instance is stopped with no recorded activity -- **THEN** the output shows the header with state "stopped" and no resource bars +- **WHEN** the user runs `spinloop remote metrics --format=bar` and the instance is stopped with no retained history and no recorded activity +- **THEN** the output shows the header with state "stopped" and no resource series #### Scenario: Stopped instance still reports its last activity -- **WHEN** the user runs `spinloop remote metrics --format=bar`, the instance is - stopped, and a last-active time is known -- **THEN** the output shows the header, the last-active line, and no resource - bars +- **WHEN** the user runs `spinloop remote metrics --format=bar`, the instance is stopped, and a last-active time is known +- **THEN** the output shows the header, the last-active line, and the series drawn from the retained history where one exists + +#### Scenario: Stopped instance shows its history + +- **WHEN** the user runs `spinloop remote metrics --format=bar` and the instance's engine has been stopped after running, with a retained history +- **THEN** the output shows the series as sparklines drawn from the readings taken before the stop, ending at the stop ### Requirement: Last-active line in bar format @@ -98,3 +98,45 @@ than shown empty or zeroed. - **WHEN** bar format renders an endpoint with no known last-active time - **THEN** the output goes straight from the header to the resource bars +### Requirement: Gauge format + +The system SHALL support a `--format=gauge` option that renders each resource series as a horizontal progress gauge: a left-aligned label, a filled portion using block characters, an unfilled portion using light shade characters, and a right-aligned percentage value. The gauge draws the current reading only — it carries no history. The series drawn SHALL be CPU, RAM, and each GPU's utilisation and memory, with the same labels the bar format uses. The gauge fill SHALL be colour-coded on the bar format's thresholds: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%, with the colour reset after the filled portion so the unfilled characters and percentage appear in the terminal's default colour. + +#### Scenario: Gauge format displays CPU utilization + +- **WHEN** the user runs `spinloop remote metrics --format=gauge` with a running instance that has CPU data +- **THEN** the output includes a gauge labelled "CPU" with filled and unfilled segments proportional to the current utilization + +#### Scenario: Gauge format displays GPU utilisation + +- **WHEN** the user runs `spinloop remote metrics --format=gauge` with a running instance that has GPU data +- **THEN** the output includes gauges labelled "GPU util" and "GPU mem" (or "GPU N util"/"GPU N mem" for multiple GPUs) + +#### Scenario: Gauge colours the fill + +- **WHEN** a gauge's current value is 95% +- **THEN** its filled segment appears in red, and its unfilled segment and percentage appear in the terminal's default colour + +### Requirement: Bar draws the retained history + +Bar format SHALL draw each series from the history the on-instance daemon retains: readings taken at the sampler's cadence while the engine ran, covering at most the last 10 minutes. The sparkline SHALL show every sample the window holds, downsampled to the draw width where the window holds more samples than the width allows; downsampled points SHALL preserve the window's extremes rather than averaging them away. + +The format SHALL be usable in one-shot mode: the history comes from the daemon, not from the command's own polling, so `--format=bar` without `--watch` draws the same window `--watch` would. + +Where the daemon reports no history — a daemon that predates the feature, or an engine with no reading yet — bar format SHALL draw each series from the current reading alone, in the gauge's filled style, so the default format still shows the current level and a pre-history daemon renders exactly as it does today. + +#### Scenario: One-shot bar shows the daemon's window + +- **WHEN** the user runs `spinloop remote metrics --format=bar` without `--watch` against an engine that has been running +- **THEN** the output shows each series as a sparkline covering up to the last 10 minutes of the daemon's retained samples + +#### Scenario: More samples than width are downsampled + +- **WHEN** the retained window holds more samples than the draw width +- **THEN** the sparkline shows one glyph per column of the width, and a spike inside a downsampled range is still visible rather than smoothed away + +#### Scenario: No history falls back to the gauge drawing + +- **WHEN** the user runs `spinloop remote metrics --format=bar` against a daemon that reports no history +- **THEN** each series is drawn from the current reading in the gauge's filled style + diff --git a/openspec/specs/remote-stats/spec.md b/openspec/specs/remote-stats/spec.md index ce8a6b3..eb5d33d 100644 --- a/openspec/specs/remote-stats/spec.md +++ b/openspec/specs/remote-stats/spec.md @@ -49,7 +49,7 @@ When the user passes `--cost`, the stats report SHALL include an estimated on-de ### Requirement: Tabular display -The stats output SHALL support three formats via the `--format` flag: `bar` (default), `table`, and `json`. The `bar` format SHALL produce a compact display with horizontal progress bars for resource metrics, colour-coded by utilization level. The `table` format SHALL produce a tab-separated key-value table, one line per metric, with the key column left-aligned and values right of it. The `json` format SHALL output the response as a JSON object to standard output. Progress and error messages SHALL go to standard error regardless of format. +The stats output SHALL support four formats via the `--format` flag: `bar` (default), `gauge`, `table`, and `json`. The `bar` format SHALL produce a compact display drawing each resource series as a sparkline of the daemon's retained history, with the latest point colour-coded by utilization level. The `gauge` format SHALL produce a compact display with horizontal progress gauges for the current reading, colour-coded by utilization level. The `table` format SHALL produce a tab-separated key-value table, one line per metric, with the key column left-aligned and values right of it. The `json` format SHALL output the response as a JSON object to standard output. Progress and error messages SHALL go to standard error regardless of format. #### Scenario: Clean output @@ -69,7 +69,12 @@ The stats output SHALL support three formats via the `--format` flag: `bar` (def #### Scenario: Bar format is explicit - **WHEN** the user runs `spinloop remote metrics --format=bar` -- **THEN** the output is in bar format with progress bars for resource metrics +- **THEN** the output is in bar format, drawing each resource series as a sparkline of the daemon's retained history + +#### Scenario: Gauge format is explicit + +- **WHEN** the user runs `spinloop remote metrics --format=gauge` +- **THEN** the output is in gauge format with progress gauges for the current reading #### Scenario: JSON format @@ -78,7 +83,7 @@ The stats output SHALL support three formats via the `--format` flag: `bar` (def #### Scenario: JSON format with cost -- **WHEN** the user runs `spinloop remote metrics --format=json --cost` with a running instance +- **WHEN** the user runs `spinloop remote metrics --format=json --cost` - **THEN** the JSON output includes a cost estimate field #### Scenario: Invalid format errors @@ -159,3 +164,22 @@ show one implying the endpoint has been quiet since it started. - **THEN** the report shows no last-active figure, and the rest of the report renders as it does today +### Requirement: History in the report + +When the on-instance daemon's metrics reply carries a history of system readings, the report SHALL carry it through to the command's output: the `json` format SHALL include the readings, and the `bar` format SHALL draw them. Where the daemon's reply carries no history, the report SHALL omit the field and the `bar` format SHALL fall back per the bar format specification. The control plane's relay of the daemon's reply SHALL NOT alter the readings it carries. + +#### Scenario: JSON carries the daemon's history + +- **WHEN** the instance's daemon reports a retained history and the user runs `spinloop remote metrics --format=json` +- **THEN** the JSON output includes the history's readings + +#### Scenario: Bar draws the relayed history + +- **WHEN** the instance's daemon reports a retained history and the user runs `spinloop remote metrics --format=bar` +- **THEN** each resource series is drawn as a sparkline from the readings the control plane relayed + +#### Scenario: A daemon without history degrades + +- **WHEN** the instance runs a daemon whose reply carries no history and the user runs `spinloop remote metrics` +- **THEN** the report omits the history field and bar format draws the current reading in the gauge's filled style + From 8bfe4f0481ddefa930e34c0f0cea79f2c0c811e7 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Sun, 6 Sep 2026 03:23:28 +0100 Subject: [PATCH 7/7] docs(openspec): archive the metrics-graph change --- .../2026-09-06-metrics-graph}/.openspec.yaml | 0 .../{metrics-graph => archive/2026-09-06-metrics-graph}/design.md | 0 .../2026-09-06-metrics-graph}/proposal.md | 0 .../2026-09-06-metrics-graph}/specs/daemon-api/spec.md | 0 .../2026-09-06-metrics-graph}/specs/engine-activity/spec.md | 0 .../2026-09-06-metrics-graph}/specs/fleet-client/spec.md | 0 .../specs/remote-metrics-bar-format/spec.md | 0 .../2026-09-06-metrics-graph}/specs/remote-stats/spec.md | 0 .../{metrics-graph => archive/2026-09-06-metrics-graph}/tasks.md | 0 9 files changed, 0 insertions(+), 0 deletions(-) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/.openspec.yaml (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/design.md (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/proposal.md (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/specs/daemon-api/spec.md (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/specs/engine-activity/spec.md (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/specs/fleet-client/spec.md (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/specs/remote-metrics-bar-format/spec.md (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/specs/remote-stats/spec.md (100%) rename openspec/changes/{metrics-graph => archive/2026-09-06-metrics-graph}/tasks.md (100%) diff --git a/openspec/changes/metrics-graph/.openspec.yaml b/openspec/changes/archive/2026-09-06-metrics-graph/.openspec.yaml similarity index 100% rename from openspec/changes/metrics-graph/.openspec.yaml rename to openspec/changes/archive/2026-09-06-metrics-graph/.openspec.yaml diff --git a/openspec/changes/metrics-graph/design.md b/openspec/changes/archive/2026-09-06-metrics-graph/design.md similarity index 100% rename from openspec/changes/metrics-graph/design.md rename to openspec/changes/archive/2026-09-06-metrics-graph/design.md diff --git a/openspec/changes/metrics-graph/proposal.md b/openspec/changes/archive/2026-09-06-metrics-graph/proposal.md similarity index 100% rename from openspec/changes/metrics-graph/proposal.md rename to openspec/changes/archive/2026-09-06-metrics-graph/proposal.md diff --git a/openspec/changes/metrics-graph/specs/daemon-api/spec.md b/openspec/changes/archive/2026-09-06-metrics-graph/specs/daemon-api/spec.md similarity index 100% rename from openspec/changes/metrics-graph/specs/daemon-api/spec.md rename to openspec/changes/archive/2026-09-06-metrics-graph/specs/daemon-api/spec.md diff --git a/openspec/changes/metrics-graph/specs/engine-activity/spec.md b/openspec/changes/archive/2026-09-06-metrics-graph/specs/engine-activity/spec.md similarity index 100% rename from openspec/changes/metrics-graph/specs/engine-activity/spec.md rename to openspec/changes/archive/2026-09-06-metrics-graph/specs/engine-activity/spec.md diff --git a/openspec/changes/metrics-graph/specs/fleet-client/spec.md b/openspec/changes/archive/2026-09-06-metrics-graph/specs/fleet-client/spec.md similarity index 100% rename from openspec/changes/metrics-graph/specs/fleet-client/spec.md rename to openspec/changes/archive/2026-09-06-metrics-graph/specs/fleet-client/spec.md diff --git a/openspec/changes/metrics-graph/specs/remote-metrics-bar-format/spec.md b/openspec/changes/archive/2026-09-06-metrics-graph/specs/remote-metrics-bar-format/spec.md similarity index 100% rename from openspec/changes/metrics-graph/specs/remote-metrics-bar-format/spec.md rename to openspec/changes/archive/2026-09-06-metrics-graph/specs/remote-metrics-bar-format/spec.md diff --git a/openspec/changes/metrics-graph/specs/remote-stats/spec.md b/openspec/changes/archive/2026-09-06-metrics-graph/specs/remote-stats/spec.md similarity index 100% rename from openspec/changes/metrics-graph/specs/remote-stats/spec.md rename to openspec/changes/archive/2026-09-06-metrics-graph/specs/remote-stats/spec.md diff --git a/openspec/changes/metrics-graph/tasks.md b/openspec/changes/archive/2026-09-06-metrics-graph/tasks.md similarity index 100% rename from openspec/changes/metrics-graph/tasks.md rename to openspec/changes/archive/2026-09-06-metrics-graph/tasks.md