Skip to content

Feat/persistency import export#166

Open
viktorbeck98 wants to merge 8 commits into
developmentfrom
feat/persistency-import-export
Open

Feat/persistency import export#166
viktorbeck98 wants to merge 8 commits into
developmentfrom
feat/persistency-import-export

Conversation

@viktorbeck98
Copy link
Copy Markdown
Collaborator

feat: export and import EventPersistency state on demand

Problem

There was no way to transfer a detector's trained state between environments (e.g. training → production) or take a manual snapshot outside the auto-save timer cycle, without constructing a full PersistencySaver with its threading machinery.

What changed

Layer 1 — standalone functions (detectmatelibrary.utils.persistency)

save and load are extracted from PersistencySaver's internals and exposed as module-level functions. PersistencySaver now delegates to them, so the serialisation format lives in exactly one place.

Layer 2 — detector-level wrappers (CoreDetector)

export_state and import_state are thin wrappers around the above. import_state is thread-safe: it acquires the saver lock before loading when a PersistencySaver is running.

Bytes overload

Both layers accept an optional path. When omitted, save/export_state return the state as an in-memory zip archive (bytes); load/import_state accept either a path string or those bytes. This lets an API send or receive state without touching the filesystem.

Metadata

event_data_class is now written to the metadata JSON and restored on load, so a detector loaded into a fresh EventPersistency picks the correct backend for new events automatically. Existing snapshots without the field continue to load fine.

Usage

# File-based (unchanged behaviour)
detector.export_state("s3://bucket/snapshots/my-detector")
detector.import_state("s3://bucket/snapshots/my-detector")

# Bytes-based (new — no disk I/O)
data: bytes = detector.export_state()
detector.import_state(data)

# Standalone (advanced / no detector)
from detectmatelibrary.utils import persistency
data = persistency.save(ep)
persistency.load(ep, data)

What did not change

  • PersistencySaver public API (save, load, start, stop) is unchanged.
  • On-disk format is unchanged; one new optional metadata field added, old snapshots remain loadable.
  • EventPersistency gains no new public methods.

Tests

  • TestStandaloneSaveLoad — 7 new tests covering the bytes overload roundtrip
  • TestDetectorExportImportState — 3 new tests covering bytes export/import at the detector level, including a running-saver safety check

viktorbeck98 and others added 8 commits June 3, 2026 10:16
…aviour

json_parser: remove non-existent flatten_nested/ignore_parse_errors fields,
correct content_parser to sibling-entry format, fix template path, add
missing imports and a dict-based usage example with verified output.

new_value: fix broken Python example (undefined cfg, unused BufferMode import).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…metadata

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, fix load() thread safety

- Add module-level _load helper that mirrors _save
- Restore event_data_class from metadata on load
- PersistencySaver.load() now acquires self._lock before mutating persistency
- Use fs.pipe() for atomic file writes in _save to eliminate read/write race
- Add test_load_restores_event_data_class and TestPersistencySaverThreadSafety

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ctor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
save(ep) returns a zip archive as bytes; load(ep, bytes) restores from
it. export_state() and import_state() on CoreDetector gain the same
overload, so callers can transfer state over a network API without
touching the filesystem.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@viktorbeck98 viktorbeck98 requested a review from ipmach June 3, 2026 14:37
@viktorbeck98 viktorbeck98 self-assigned this Jun 3, 2026
@viktorbeck98 viktorbeck98 added the enhancement New feature or request label Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant