Skip to content

Add crash and restart evidence to real-life validation lab#3

Open
andrei-hasna wants to merge 2 commits into
mainfrom
fix/real-life-validation-lab-evidence
Open

Add crash and restart evidence to real-life validation lab#3
andrei-hasna wants to merge 2 commits into
mainfrom
fix/real-life-validation-lab-evidence

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Extend validate:real-life with a process-level crash-recovery drill.
  • Add a true collector-restart/high-volume drill that stops the live collector, queues 120 Pino structured SDK records while the collector is down, persists them to the SDK file spool, restarts the collector on the same port/data dir, replays the spool, and verifies SQLite/raw pointers.
  • Record both retained validation runs in the real-life validation plan.

Evidence

Crash-recovery retained run:

/tmp/open-logs-real-life-lab-6uRbnp/real-life-validation-report.json

That run proves:

  • child producer exited 42 after raw append before indexing
  • doctor segments detected exactly 1 unindexed raw event
  • doctor rebuild-index reindexed 61 raw-backed events
  • verification returned ok with 0 unindexed raw events
  • crash log projection was reconstructed from raw evidence
  • final export included the crash event raw envelope
  • final doctor segments reported 61 checked raw events and 0 unindexed raw events

Collector-restart/high-volume retained run:

/tmp/open-logs-real-life-lab-ku8uZn/real-life-validation-report.json

That run proves:

  • live collector process was stopped before the producer flush
  • 120 Pino structured records queued through the SDK transport while the collector was down
  • failed flush persisted 120 records to the SDK file spool
  • restarted collector loaded and replayed all 120 spooled records
  • spool file was cleared after replay
  • SQLite contained 120 matching log projections with raw segment pointers and SHA-256 record hashes
  • final export sampled replayed raw envelopes
  • final doctor segments reported 181 checked raw events and 0 unindexed raw events

Validation

  • bun run validate:real-life -- --keep
  • bun test src/lib/sdk-client.test.ts
  • bun run lint
  • bun run build
  • git diff --check

Known existing guard failure outside this slice:

  • ./node_modules/.bin/tsc --noEmit --pretty false currently fails in pre-existing src/lib/redaction.* type errors; this PR did not touch those files.

@andrei-hasna andrei-hasna changed the title Add crash-recovery evidence to real-life validation lab Add crash and restart evidence to real-life validation lab Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant