Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/real-life-validation-plan.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Real-Life Telemetry Validation Plan

Last updated: 2026-06-19
Last updated: 2026-06-24

This plan defines the evidence required before open-logs can be considered a robust universal telemetry data substrate.

Expand All @@ -23,6 +23,8 @@ For each scenario:
These runs are useful evidence, but they do not replace the larger targets below:

- `bun run validate:real-life -- --keep` retained `/tmp/open-logs-real-life-lab-NJZ95y/real-life-validation-report.json`, proving one isolated token-secured server, CLI project creation, remote CLI live watch, cross-process CLI log/span writes, HTTP exception/metric writes, one captured `logs run` test command with Linux procfs resource and process-tree summaries, one raw-backed indexed `process.resource.peak_rss` metric event mapped to that run, one raw-backed indexed process-tree event mapped to that run, one isolated artifact-producing build command with generated JavaScript and source-map outputs, two raw-backed indexed `artifact` events, two SQLite `artifacts` rows with relative paths and SHA-256 hashes, one SQLite `source_maps` row, one SQLite `source_map_sources` row, source-map JavaScript linkage, source content represented only by a hash in collector-owned evidence, source-map content canaries absent from retained collector-owned report/export/raw/SQLite evidence, forbidden source-map raw/projection container keys absent from retained scans, one real Bun JUnit reporter command captured through `logs run`, one raw-backed indexed `category=test_report` event with relative path, parsed JUnit counts, SHA-256 hash, one live SQLite `test_reports` projection row, zero SQLite `test_cases` rows for the passing report, authenticated API test-report list/get access, local CLI test-report list/get access, real MCP stdio test-report search/get access, event export with raw envelopes, `Last-Event-ID` catch-up for a pre-existing post-anchor event, a live follow-up event after catch-up, and `doctor segments` with 60 checked raw events and 0 unindexed raw events.
- `bun run validate:real-life -- --keep` retained `/tmp/open-logs-real-life-lab-6uRbnp/real-life-validation-report.json` after adding a process-level crash-recovery drill: a child producer appended raw log evidence and exited with code 42 before SQLite indexing, `doctor segments` detected exactly 1 unindexed raw event, `doctor rebuild-index` reindexed 61 raw-backed events, verification returned to `ok` with 0 unindexed raw events, the crash log projection was reconstructed from raw evidence, final export included the crash event raw envelope, and final `doctor segments` reported 61 checked raw events with 0 unindexed raw events.
- `bun run validate:real-life -- --keep` retained `/tmp/open-logs-real-life-lab-ku8uZn/real-life-validation-report.json` after adding a true collector-restart/high-volume drill: the lab stopped the live collector process, queued 120 Pino structured records through the SDK transport against the down collector, observed a failed flush, persisted 120 records to the SDK file spool, restarted the collector on the same port and data dir, loaded and replayed all 120 spooled records, cleared the spool file, verified 120 matching SQLite log records with raw segment pointers and SHA-256 record hashes, sampled replayed raw envelopes in export verification, and final `doctor segments` reported 181 checked raw events with 0 unindexed raw events.
- `bun run validate:stress -- --keep` retained `/tmp/open-logs-high-volume-stress-AtCHHl/high-volume-ingest-stress-report.json`, proving 10 producer processes, 5,000 mixed event records, 81 segment rows/files, no missing expected IDs, no duplicate event IDs, full raw pointer reconstruction before and after rebuild, SQLite integrity and foreign-key checks, `doctor segments`, and `doctor rebuild-index`.
- `bun run validate:streams -- --keep` retained `/tmp/open-logs-stream-load-q2aFBk/stream-load-validation-report.json`, proving direct generic SSE delivery, `Last-Event-ID` buffer-miss SQLite catch-up with explicit `buffer_miss_sqlite_catchup`, forced slow-subscriber overflow with explicit `subscriber_queue_overflow`, bounded multi-consumer API SSE fanout with 8 consumers receiving 80 burst events each, remote `logs watch --server`, local `logs watch --events`, real MCP `event_watch` over stdio including raw envelopes and missing-cursor overflow, and `doctor segments` with 141 checked raw events and 0 unindexed raw events.
- `bun run validate:dashboard-stream -- --keep` retained `/tmp/open-logs-dashboard-stream-lab-f1Ji0B/dashboard-stream-validation-report.json`, proving real Chromium sessions against the built dashboard served by both the token-secured source API server and `dist/server/index.js` from an extracted npm package, unauthenticated stream blocking, authorized fetch-backed SSE after dashboard token entry, rendered live event records, dashboard paused state with the paused event not rendered before resume, pause/resume reconnect with query `last_event_id`, catch-up of an event written while paused, post-resume live delivery, and `doctor segments` with 6 checked raw events and 0 unindexed raw events.
Expand Down
Loading