Skip to content

Storage layer - #2

Merged
nccapo merged 4 commits into
masterfrom
storage-layer
Mar 5, 2026
Merged

Storage layer#2
nccapo merged 4 commits into
masterfrom
storage-layer

Conversation

@nccapo

@nccapo nccapo commented Mar 5, 2026

Copy link
Copy Markdown
Contributor

Here's a summary of what's new

memstore.go

  • MemStore struct with sync.RWMutex protecting five maps: events, groups, fingerprint index ("projectID:fingerprint" → groupID), crashes, projects, and a DSN
    key index
  • Deep copy helpers for all domain types — callers can't mutate stored state
  • Generic paginate[T] helper (limit=0 means no cap)
  • ListGroups sorts by last_seen/first_seen/count descending; ListEvents and ListPodCrashes sort newest-first
  • IncrementGroupCount and UpdateGroupStatus mutate in place (no re-save needed)

memstore_test.go — 39 tests across every method:

  • Save + Get happy paths for all four entity types
  • ErrNotFound for missing IDs and wrong-project isolation
  • Copy semantics verified (mutating returned structs doesn't corrupt store)
  • ListEvents: project filter, level filter, descending order, limit/offset
  • ListGroups: status filter, all three sort fields, limit/offset
  • ListPodCrashes: namespace filter, descending order, limit/offset
  • IncrementGroupCount / UpdateGroupStatus not-found cases
  • DeleteEventsOlderThan: partial delete, none match, empty store

Design decisions:

  • ZSTD compression via options.ZSTD
  • Reverse iterator (reverseIter helper) serves events/crashes newest-first directly from key ordering — no in-memory sort needed for single-prefix scans
  • Groups/cross-namespace listing loads into memory and sorts (arbitrary sort fields can't be served from key layout)
  • Secondary lookup indexes (ei:, ci:) enable O(1) point lookups by ID without knowing the timestamp at call time
  • DeleteEventsOlderThan parses timestamps directly from keys (no JSON decode) via parseEventKey, then uses WriteBatch for bulk deletion; also cleans up lookup
    index entries
  • startGC runs db.RunValueLogGC(0.5) every 5 minutes in a goroutine stopped cleanly by Close()

badger_test.go — Integration tests (74 tests total across both files)

  • Mirrors every memstore scenario: save+get, not-found, wrong-project isolation, list filters, sort orders, pagination, increment, status update, delete old
    events
  • TestBadger_DataSurvivesReopen — closes and reopens the DB to confirm ZSTD-compressed data persists
  • TestBadger_Event_RoundTrip — verifies all field values survive JSON encode/decode
  • TestParseEventKey — unit tests the key-parsing helper used by DeleteEventsOlderThan

@nccapo nccapo added type:documentation Improvements or additions to documentation type:enhancement New feature or request labels Mar 5, 2026
@nccapo
nccapo merged commit 149871e into master Mar 5, 2026
3 of 4 checks passed
@github-actions
github-actions Bot deleted the storage-layer branch March 5, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:documentation Improvements or additions to documentation type:enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant