Add crash-durability journaling to Periscope (JournalKit)#86
Merged
Conversation
Design loop: crash durability. A standalone macOS SwiftPM prototype (not wired into any target or CI) benchmarks every journal candidate — append-only file (± fsync), raw SQLite WAL (NORMAL/FULL), GRDB, Core Data, and SwiftData (per-record and batched saves) — measuring emit-path latency distributions single-threaded and contended, and actual crash durability via child processes that SIGKILL themselves mid-stream with the parent recovering and counting. Headline data (full tables in the README): every per-record-commit variant survives SIGKILL completely, including unfsynced file appends and synchronous=NORMAL WAL — page-cache writes survive process death. Both batched variants lose exactly their unsaved tail. The file append is the only variant whose worst case stays in microseconds (max 40µs single-threaded, 363µs contended, 560k ops/s); every SQLite-backed option has multi-millisecond emit-path tails from checkpoints or save machinery, and SwiftData per-record is 167µs median with a 790ms contended max.
Crash-durability design loop, step 1 of 4 (groundwork — no behavior change for existing targets). The benchmark prototype settled the implementation: append-only file, page-cache durability by default. JournalKit is payload-agnostic by design — opaque Data entries, no logging knowledge — so Periscope's log journal is just one client. Entries frame as [length][crc32][payload] into numbered segments that rotate at half the byte budget, dropping oldest-first (flight-recorder posture) with drops always observable. append() is synchronous from any thread: returning means the entry is in the kernel page cache and survives process death; .full extends to kernel panics via F_FULLFSYNC. Recovery reads segments in order, validates length + CRC, and never throws over the torn tail a crash leaves. Tests include a truncation fuzz (every byte-level cut point yields exactly the wholly-written entries and flags the tear), CRC corruption, absurd-length rejection, budget rotation, reopen continuation, and concurrent-appender order. Wired as JournalKitTests into the Stuff-iOS-Tests scheme.
Crash-durability design loop, step 2 of 4 (groundwork — nothing writes
these yet). LogJournalEntry is the log-layer envelope PeriscopeCore
will write through JournalKit's payload-agnostic Journal: versioned
JSON ({"v":1,"kind":…,"payload":…}) carrying the session (so a
recovered journal attributes itself), scope definitions (hierarchy),
and records (the store's durable mirror plus the journal sequence that
orders replay). Unknown kinds decode to nil so a newer build's entries
skip instead of failing ingest; attachments inline up to 64KB and
otherwise journal as an omitted-marker with name/type/size — inlining
screenshots would blow the emit-latency budget.
Crash-durability design loop, step 3 of 4. On-disk stores open this
launch's LogJournal beside the database (Periscope-Journals/<session>/)
and write the session entry; adding the store as a sink installs the
journal into Periscope — journaling requires a store, since recovery
is persistence. In-memory stores never journal.
Every buffered record then appends to the journal on the emitting
thread: the sequence is stamped under the state lock (so replay can
restore buffer order, spans included) but the file I/O happens outside
it, so emitters don't serialize on disk. Scope definitions journal as
they're first seen plus a replay at install. Records at fault+
F_FULLFSYNC before returning — kernel-panic durability for the direst
records. Journal failures never throw into the emit path: they count
(appendFailureCount) and log to OSLog, first + every 100th.
Tests pin the core promise — records reach disk with the drain gated
shut ('the process died before delivering'), span pairs journal in
sequence order, failures count instead of throwing, on-disk stores
wire the journal end to end, and in-memory stores don't.
Crash-durability design loop, step 4 of 4. Before a new session starts, the store replays every prior session's journal: entries decode (unknown kinds and corrupt payloads skip — the journal CRC already dropped torn entries), scopes upsert, and records the pipeline never delivered persist in journal-sequence order, deduplicated against what did arrive by event ID. The crashed session's row comes from the store when its startSession save survived, or from the journal's own session entry when the crash beat even that. A .notice marker on the crashed session records the recovery in the story. Ingested journals delete; a journal whose ingest fails stays for the next launch to retry; one with no session entry is unattributable and discards audibly. Ingest runs before startSession on purpose: recovered span begans are in the store when the orphan sweep runs, so a flow the crash abandoned closes as .orphaned — asserted end to end by a test that journals a began, 'relaunches', and finds the pair. Also covered: full-field recovery (tags, call site, scopes, session attribution), dedupe against delivered records, and discarding sessionless journals.
Crash-durability design loop, docs. The Core README's how-it-works gains the journal story (synchronous page-cache appends at emit, fault+ F_FULLFSYNC, launch-time ingest with dedupe and the orphan-sweep handoff, microseconds loss window); the Core AGENTS invariants gain the journal contract (synchronous at emit, silent on failure, ingest before startSession) and the Journal/ layout entry; the root AGENTS lists the JournalKit module and test bundle; TODOs moves the crash durability design item to completed with the data-backed rationale.
Brings the base branch's merge of main into the stacked journal branch. The only conflict was AGENTS.md's library/target lists, taken from the base and re-adding JournalKit and JournalKitTests. Full Stuff-iOS-Tests scheme passes on the merged tree.
kyleve
added a commit
that referenced
this pull request
Jul 15, 2026
Add Periscope: a typed, hierarchical observability framework
## Summary
Adds **Periscope**, a typed, hierarchical observability framework, as three new SPM modules under `Shared/Periscope/`, wired into `Package.swift`, `Project.swift`, and the `Stuff-iOS-Tests` CI scheme. It coexists with `LogKit`/`LogViewerUI` — no `WhereLog` migration in this PR.
### PeriscopeCore — model and machinery
- **Structured, typed events**: `LogEvent` (Codable; stable `eventName` + `eventVersion`), freeform `Message`, extensible `LogLevel` struct (OTel-style name + severity).
- **`Log<Event>` value loggers** over a deterministic **scope tree** (same path = same scope in any process/launch): typed children (`root(PhotoLogs.self)`), entity children (`photos(for: album.id)`), single-expression derive-and-emit (`album(for: id) { .uploaded }`), **links** (`model + ui`), **tags** (`tagged(.paymentID, id)`), attachments (with `Error`/`Codable`/image conveniences).
- **Propagation**: task-local `withContext`/`Log.current`, and `LogContextProviding` for derived per-instance loggers (identity-safe via `InstanceID` + associated-object deallocation trackers, so a recycled pointer can never inherit a dead object's logging identity).
- **Spans**: `measure` with auto-derived exits (success/failure/cancelled) and optional **budgets** (a `SpanOverdue` warning fires *while* the closure hangs, race-free against the span's own end), `begin/end` with explicit `SpanLifetime` (bounded spans expire via a clock-driven watchdog), `SpanExit` modes, `SpanRelaunchPolicy` (relaunch orphan-closes spans the dead process left open), `OSSignposter` mirroring.
- **Pipeline**: ordered `LogSink` fan-out (OSLog sink built in) with level floors (global + per-subtree), coalesced auto-flush at error+, bounded drop policy with synthetic `DroppedEvents` gap reports, redaction hook, and a bounded live-records stream that replays exact buffered order.
- **Persistence**: `PeriscopeStore` (`@ModelActor` sink) — indexed schema, per-launch **sessions** (app/OS/device metadata), versioned JSON payloads that outlive their Swift types, external-storage attachments, rich query API (time/level/type/session/scope-subtree/tag/span-exit/search, paged), retention pruning, rollback-on-failure so a poisoned batch can't wedge later saves.
- **Ambients**: extensible `AmbientEventSource` with built-ins for app lifecycle, memory warnings, network path, thermal state, and low power mode.
### PeriscopeUI
`.logContext(_:)` modifiers (accepting `Log` values or `LogContextProviding` models; stacking links contexts) and the `\.logContext` environment accessor.
### PeriscopeTools
`PeriscopeViewer` (searchable/filterable latest-logs viewer — level, event, scope, session, span exit — with NDJSON export), `LogTraceView` (walks an error back through time, linked scopes, and the ancestor chain), `OpenSpansView` (what's in flight right now, longest-running first), `PeriscopeAlerter` (hookable debug toast, local-notification default with cached authorization), and log view mode (`.logInspectable(_:)` + `PeriscopeInspector`, two-way synced with the system flag).
## Review hardening
Three full code-review passes followed the initial build, each landing fixes as dedicated commits.
**First pass** — every high/medium finding fixed: store rollback + failure injection, instance identity via dealloc trackers, subscribe-before-load in live models, coalesced auto-flush, bounded live buffers, the span lifecycle model, linear drain chunking, relationship prefetching. The closing seeded **fuzz test caught a real bug** — `add(sink:)` appended its scope replay after already-pending records, so late-added sinks could see records before their scope definitions; the replay is now prepended.
**Second pass** — the orphan sweep fetches only `spanID` columns; inspect-mode changes yield inside the state lock so racing setters can't strand `bufferingNewest(1)` subscribers; **span pairs floor together** (the begin-time decision governs the whole lifecycle — no dangling halves across floor changes); the watchdog holds the system weakly. Low items: overdue sentinels serialize with their span's end through a per-measure gate, the six copies of scope-path walking collapsed into `LogScope.ancestry(of:resolve:)`, `showHosted` got `show()`'s animation-speed handling, and the viewer clears export state on store swaps. Also fixed here: the store's nine-condition filter predicate is hand-built in statement form (`PredicateExpressions`, one `let` per condition) after the `#Predicate` macro's single inference tree blew the type-checker budget on CI runners.
**Third pass — span pair integrity.** Writing a second seeded fuzz (spans, floor flips, cross-task supersession, drop pressure) surfaced that "no dangling halves" had three remaining attack vectors; all are now closed, and the invariant holds on every channel:
1. **Atomic begin** — `LogRecorder.beginSpan` registers a span and buffers its `SpanBegan` under one lock acquisition, so no racing supersede or `end(for:)` can deliver a span's end before its began.
2. **Drop-proof pairs** — the overflow drop policy exempts `SpanBegan`/`SpanEnded` (like scope definitions): a dropped began would strand its end; a dropped end reads as "still open" until the next launch's orphan sweep. `SpanOverdue` stays droppable.
3. **Transform-only redaction** — a redaction hook may rewrite pair records but `nil` (suppression) falls back to a stripped copy (tags/attachments dropped, exit reason blanked) instead of stranding the partner; floors are the supported way to silence spans.
Live streams got the same ordering rigor: observer yields moved inside the state lock (`record()` and `beginSpan` now share one delivery helper), so `liveRecords()` consumers see exact buffered order — previously two racing emitters could invert live delivery even though sinks and the store were ordered.
Remaining staged work (`survivesRelaunch` resume mechanics) is tracked in `Shared/Periscope/TODOs.md`, which also logs every completed finding.
**Review response (post-review feedback batch).** Every non-design review comment landed as its own commit: the ambient observer-token bug (the block-based API keeps observations alive in the center — dropped tokens made them unremovable and immortalized the captured system; `AmbientObserverTokens` + `stop()`/`stopAmbientSources()` fix it), typed tag values (`LogTagValue`) with `[LogTag]` lists and multi-tag AND queries, `LogAttachment.ContentType` as an enum, per-level `osLogType` overrides, the missing `SpanExit` factories, `InstanceID` storing the metatype, drop protection as a `LogEvent` opt-in, `#function`/`#fileID` capture on every emit, `LogEvent.externalID` (indexed and queryable), a `StoreWriteFailed` marker after rolled-back writes, retention pruning orphaned sessions/tags/scope branches, an accessibility-settings ambient source, and configurable tool caps (500). The six larger design comments (crash durability, span record modeling, decomposition, readable `ScopeID`s, context parent hierarchy, ambient state snapshots) are tracked in `TODOs.md` for follow-up loops — crash durability is already built and stacked as #86.
## Testing
284 Swift Testing tests across 40 files in three hosted bundles (`PeriscopeCoreTests`, `PeriscopeUITests`, `PeriscopeToolsTests`): deterministic pipeline tests via gated sinks, clock-injected watchdog sweeps, store failure injection, hosted SwiftUI tests for the tools, and **two seeded fuzz suites** — pipeline interleavings (no loss/duplication, per-emitter order, scopes-before-records on every sink) and span lifecycles (strict began-then-ended pairs under concurrent supersession, floor flips, and drop pressure; empty open-span registry after cleanup). `./swiftformat --lint` and `Stuff-iOS-Tests` pass locally and on CI.
Review walkthrough finding 1: a short write(2) — disk-full writes some bytes then fails — left torn bytes mid-segment with the descriptor positioned after them, so every later (successful) append landed behind the corruption and recovery, which stops at the first invalid entry, lost them all. A partial write now accounts its torn bytes, poisons the segment, and the next append rotates to a fresh one — transient disk pressure costs the one interrupted entry, not the rest of the session. A clean failure (nothing landed) doesn't poison. Tested via an @_spi(Testing) short-write injection (DEBUG-only, per the store's injection seam pattern): before/torn/after recovers exactly [before, after] with the tear flagged and 'after' in a fresh segment.
Review walkthrough finding 2, in two connected parts.
Ingest now reads the recovery flags it previously dropped: the marker
event says what the journal could not preserve ('ended in a torn
entry', 'older entries were dropped by the byte budget'), escalates
from .notice to .warning when either gap exists (degraded but
handled), and appears even when zero records inserted if there's a gap
to report. The OSLog line carries both flags too.
Implementing that exposed a real bug: the session entry was written
once, first — exactly where the flight-recorder rotation drops first —
so any journal old enough to have rotated was sessionless and
discarded whole at ingest, making the dropped-entries marker
unreachable. JournalKit gains Configuration.segmentHeader, re-written
as the first entry of every segment (generic: identity/context entries
survive any rotation; recovery returns one copy per surviving
segment), and LogJournal supplies the session entry as that header.
Tests: header survival under rotation (JournalKit), a byte-torn
journal ingesting its intact records with a .warning torn-entry
marker, and a rotated storm journal attributing itself, keeping its
newest records, and admitting the loss.
Review walkthrough finding 3: the tap placement was correct but untested — moving the journal append ahead of redaction would leak PII into a file that outlives the process, invisibly to every existing test. The new test redacts a secret and suppresses another record, then asserts at two levels: the decoded journal payloads carry only the transformed content (the envelope base64-wraps nested payload bytes, so decoded content is where payload leaks would show — the first draft grepped raw segment bytes for the redacted marker and failed for exactly that reason), and the raw segment bytes contain neither secret in plaintext. The suppressed record never journals.
Review walkthrough finding 4: rotate() swallowed removal failures with try? but decremented totalByteCount and bumped droppedSegmentCount anyway; the next rotation re-listed the still-present segment and subtracted its bytes again, drifting the total low until the budget check stopped firing and the journal grew without bound. Removal is now success-conditional: a segment that fails to delete stays in the accounting (later rotations retry it) and the drop loop moves to the next-oldest so the byte budget still wins. Tested via an @_spi(Testing) removal-failure seam (DEBUG-only): sixty appends against a 1KB budget with a failure injected every fifth append keep the on-disk footprint bounded near the budget — under the old accounting it grows with the append count.
Review walkthrough finding 5: ingest deletes journal directories, so an app extension (widget, share sheet) launching mid-app-session must not run it — it would eat the live app's journal out from under its open descriptor. Extensions (detected by the .appex bundle extension) still journal their own sessions; the app's next launch recovers everyone's. The reverse hole — an app launch during a *live* extension session — needs a claim mechanism designed alongside App Group store sharing, which the store doesn't support yet either; tracked in TODOs.md, and the single-live-process assumption is now stated in the AGENTS invariants and README.
Review walkthrough finding 6: each journaled record paid three JSONEncoder passes — event to payload bytes, record struct to bytes, and an envelope wrapping those bytes as Data, which JSON represents as base64 (~33% inflation per entry, eating flight-recorder budget). The envelope is now generic over its payload and nests it as a JSON object: two passes at emit (the inner event payload stays Data — the versioned type-erased bytes the store persists). Decode does a cheap kind peek then a typed decode, paid once per launch at ingest. Done pre-merge on purpose: the wire format is free to change now and costs a version bump later. A wire-shape test pins the nested-object format.
kyleve
commented
Jul 15, 2026
| /// everyone's. (Full multi-process coordination — the reverse case of | ||
| /// an app launch during a live extension session — is tracked in | ||
| /// `Shared/Periscope/TODOs.md`.) | ||
| private static var isAppExtension: Bool { |
Owner
Author
There was a problem hiding this comment.
Let's make this a static let since it will never change during a process lifetime.
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Done in e954ae6 — static let.
kyleve
commented
Jul 15, 2026
| @_spi(Testing) public final class LogJournal: Sendable { | ||
| /// Total on-disk budget per session journal; beyond it the oldest | ||
| /// segment drops whole (flight-recorder posture). | ||
| static let byteBudget = 8 * 1024 * 1024 |
Owner
Author
There was a problem hiding this comment.
I don't feel too strongly about this being too small; let's allow up to 300mb.
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Done in e954ae6 — 300MB, with a doc note on why generous is right: journals live only until the next launch ingests and deletes them, so the budget is a ceiling on pathological sessions, not a steady state.
kyleve
commented
Jul 15, 2026
| /// Close the underlying journal (tests; production journals live for | ||
| /// the process). | ||
| @_spi(Testing) public func close() { | ||
| journal.close() |
Owner
Author
There was a problem hiding this comment.
Do we need this in deinit too, or does the journal do that automatically?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
It's automatic — Journal (JournalKit) has deinit { close() }, and LogJournal holds the only reference, so the descriptor closes when LogJournal deallocates. Production journals also live for the whole process by design (the store retains them), so teardown effectively never runs outside tests. Added a doc line on LogJournal.close saying exactly this (e954ae6).
kyleve
commented
Jul 15, 2026
|
|
||
| /// Attachment blobs above this size are omitted from the journal (the | ||
| /// async path still delivers them when the process survives) — inlining | ||
| /// screenshots would blow the emit-latency budget. |
Owner
Author
There was a problem hiding this comment.
Perhaps we should write them to external storage (aka just a file) instead; and then clean those up on removal from the journal?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Agreed — external storage removes the 64KB cliff entirely (screenshots survive crashes too). Deferring to its own PR after this one per your call; the design is captured in Shared/Periscope/TODOs.md: blobs as files beside the segments, entries referencing them by name, cleanup tied to rotation and journal removal, re-attachment at ingest.
kyleve
commented
Jul 15, 2026
| case omitted(name: String, contentType: LogAttachment.ContentType, byteCount: Int) | ||
| } | ||
|
|
||
| extension LogJournalAttachment: Codable { |
Owner
Author
There was a problem hiding this comment.
Do we need a custom codable, why not rely on the system provided one?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Two load-bearing reasons, now documented on the conformance (e954ae6): LogAttachment/ContentType aren't Codable (the content type persists via its MIME string, matching the store's column), and synthesized enum coding freezes the case names into the wire format as nesting ({"inline":{"_0":…}}) — renaming a case would silently break decoding of old journals. The flat hand-written shape keeps the persisted format independent of Swift-side naming.
- The journal byte budget rises from 8MB to 300MB per review — journals live only until the next launch ingests and deletes them, so the budget is a ceiling on pathological sessions, not a steady state. - isAppExtension becomes a static let (fixed for the process lifetime). - LogJournal.close documents that Journal's own deinit closes the descriptor — production journals need no explicit teardown. - LogJournalAttachment's hand-written Codable gains its rationale: LogAttachment/ContentType don't conform, and synthesized enum coding would freeze case names into the wire format as nesting, where a rename would silently break old journals. - TODOs gains the external-attachment-storage follow-up (write blobs as files beside the segments, clean up on rotation/removal, re-attach at ingest) for its own PR after this one.
kyleve
added a commit
that referenced
this pull request
Jul 15, 2026
## What Trims the largest `AGENTS.md` files down to invariants, hard-fought lessons, and instructions — dropping structure enumerations agents re-derive from the manifests in their first minute of searching — and regroups WhereCore's flat `Sources/` into concern-based subdirectories. One commit per step. **Guiding principle:** an AGENTS.md keeps invariants, lessons, conventions, and recipes; it drops module lists, directory trees, and dependency catalogs that mirror `Package.swift` / `Project.swift`. The root file's libraries list was already stale (missing `WhereIntents` and `SwiftDataInspector`) — proof that content rots. ## Commits 1. **GitHub rules** — new root **GitHub** section: use `gh` for all GitHub interaction; open PRs ready-for-review, never draft; keep open PRs current by pushing each local commit. "Working on PR feedback" now requires replying to a comment when a commit resolves it, and filing deferred feedback durably instead of dropping it. 2. **Root trim** — libraries/targets enumerations become pointers at the manifests; directory layout collapses to the module skeleton. The add-a-target recipe, CI-scheme rule, and the double-linking lesson stay verbatim. 3. **Where trim** — Modules tree becomes a one-paragraph layering stack; developer-overlay tour and feature narration cut; the stylesheet paragraph now points at `WhereUI/AGENTS.md`. 222 → 189 lines. 4. **Module trims** — WhereUI / WhereCore / WhereIntents lose their `Package.swift`-mirroring dependency listings; PeriscopeCore loses its directory catalog. All invariants and the full `WhereStylesheet` guidance stay. 5. **WhereCore regroup** — pure `git mv`, no behavior change: stragglers fold into `Backup/`, `Reminders/`, `Widgets/`, `Location/`, `Evidence/`, `DataResolution/`, `Persistence/`; new `Days/`, `Reporting/`, `Journal/`, `Preferences/` groups. Only `WhereServices*` and `WhereLog` remain top-level. Tests stay flat, matching PeriscopeCore. ## New lessons codified (mined from the last 30 days of PR review) - Prefer synthesized `Codable`; persisted wire formats must survive Swift renames; hand-written conformances document their load-bearing reason (#85, #86) - Retain block-based notification-observer tokens; every `start` has a paired `stop()` (#69) - Group large flat types into sub-structs / child types per behavioral area (#47, #69) - Test doubles conform to the production protocol, never an enum switch in a production type (#54) - Custom full-screen surfaces are accessibility-modal and post focus notifications (#70) - Derive UI dimensions from the live UI (preference keys, `@ScaledMetric`, semantic font styles) instead of repeating magic numbers (#38, #70, #83) - WhereCore: post-write reconciliation is defined once — writes/imports route through `DayJournal.reconcileAfterDayChange()`; cross-collaborator hooks are a single closure (#80, #88) ## Verification - `./swiftformat --lint` clean; `./sync-agents` run after each docs commit - Full `Stuff-iOS-Tests` scheme after the regroup: 1183 tests, 0 failures Made with [Cursor](https://cursor.com)
kyleve
added a commit
that referenced
this pull request
Jul 15, 2026
…store-backed tracked regions (#87) ## What & why Prepares RegionKit for "more locations" ahead of letting users pick their own regions. Two coupled shifts: 1. **Split** the ~2.5 MB monolithic `us-states.geojson` into one GeoJSON file per region, loaded **on demand** — so we only ever parse the regions we track, not the whole US at launch. 2. **Replace** the hardcoded five-case `Region` enum with a **data-driven catalog** (a bundled `regions.json` manifest + a `Region` value type), so any US state (50 + DC + PR) plus Canada/EU are available regions. The user's **tracked** regions live in the synced SwiftData store (WhereCore owns "which"; RegionKit stays the agnostic geometry/lookup engine). Onboarding's region picker and user-chosen per-region styling are **out of scope** here (designed for, not built). ## What changed - **RegionKit — data-driven catalog + on-demand geometry** - `Region` is now a `Hashable`/`Codable` value type over a stable id (`us-CA`, `canada`, …) with a well-known `.other` and conveniences (`.california`, etc.). `RegionCatalog` loads the bundled `regions.json` manifest (all/name/localizedName/geometry file, canonical order). Adding a region is now **pure data** — a manifest + geojson change generated by `Tools/generate-regions.rb`, no code. - `RegionAttributor(for:)` loads only the passed regions' files; `.all` covers the whole catalog, `.shared` the default four. `RegionAttributing` abstracts the engine so the app can supply a live, swappable attributor. `RegionGeometryCatalog.outlines(for:attributor:)` takes an explicit attributor. - **WhereCore — store-backed, synced tracked regions + live attributor** - Tracked regions persist as **one `SDTrackedRegion` row per region** (so concurrent cross-device edits merge instead of last-write-wins), read as a `Set` defaulting to the four until the user chooses. - `RegionAttribution` rebuilds the attributor from the tracked set on the store's `changes()` signal (local edit or remote CloudKit import). `WhereServices.make(...)` (async) derives it from the store; the app launch and the App Intents process (`WhereServices.forIntents()`, now async) both attribute against the same synced set. `make(...)` is the **sole public assembly entry**; the synchronous `init` is `@_spi(Testing)` for tests/previews. - **WhereUI / WhereIntents — catalog-driven** - `RegionStyle` is id-keyed with an isolated, disposable bespoke-override table (the seam user-chosen styling will replace). Ordering uses `Region.inCanonicalOrder(_:)` instead of scanning `Region.allCases`. - Siri's "pick a region" suggestions and the Spotlight index surface the **tracked** set; `RegionEntity` id resolution stays **full-catalog** (so "days in Texas" answers even when untracked). - **Backup** — tracked regions round-trip in the archive (additive field; v1 manifests decode to `[]`); `.replace` restores the set exactly, `.merge` unions it. ## Key decisions / tradeoffs - **Clean id scheme** (`us-CA` vs `california`): existing persisted region raw values change; data is re-exported/re-imported. The `NAME → id` map lives in `Tools/generate-regions.rb`. - **Localization tradeoff:** region names come from the manifest (with an optional `localizationKey` override), so region names lose static string-catalog extraction — consistent with how the App Intents `RegionEntity` already resolves names at runtime. - **Attribution priority** is the manifest's canonical order (regions are mutually exclusive at our resolution). ## Deferred (follow-ups) - The onboarding region picker and user-chosen per-region color/emoji/symbol/tint. - Seeding semantics (the default four apply only at zero rows; the picker will seed/replace) and materializing the implicit default before the first explicit edit. - **Untracking a region:** the store currently *deletes* the tracked-region row, which would re-attribute that region's past GPS days to `.other` on re-aggregation (manual days, stored as region sets, are safe). A `TODO` defers the soft-delete (mark inactive + load every ever-tracked region so history stays stable) to the picker work that defines the untrack UX. Not user-reachable today — nothing calls `setTrackedRegion(false)` outside tests. ## Testing - Full `Stuff-iOS-Tests` scheme green; `./swiftformat --lint` clean. - New/reworked coverage: catalog + value-type (`RegionTests`), subset-only-loads (`RegionAttributorTests`), rebuild-on-`changes()` + store round-trip (`RegionAttributionTests`, `TrackedRegionStoreTests`), end-to-end `make()` wiring (`WhereServicesTests`), tracked-set entity/suggestions (`RegionEntityTests`), and backup round-trip / legacy-manifest decode (`BackupServiceTests`, `BackupCoordinatorTests`). ## Review pass - **Self-review:** log (not swallow) tracked-region read failures; deterministic (canonical-ordered) attributor build; surface unknown stored region ids; backup round-trip (above). - **PR review feedback:** consolidated the RegionKit bundled-data docs into the main `README.md` (removed the separate `Resources/README.md`); documented why `Region`'s `Codable` is hand-written (bare id string vs the synthesized keyed object); made the sync `WhereServices.init` `@_spi(Testing)` so `make()` is the sole public entry; and TODO'd the untrack soft-delete (see Deferred). ## Kept current with `main` Merged `#85` (CalendarDay), `#86` (Periscope JournalKit), `#88` (backup `onImport` reconcile hook), and `#89` (streamlined AGENTS + WhereCore source regroup). Conflicts resolved: the backup `formatVersion` (kept v2 **and** the additive `trackedRegions` field); the `BackupCoordinatorTests` harness swap (kept both the tracked-region tests and `#88`'s `onImport` hook test); and `Where/AGENTS.md` (took `#89`'s streamlined Modules section, but kept the accurate manifest-backed RegionKit localization note over `#89`'s stale "static keys" wording). My changes rode the source regroup into `Backup/`, `Persistence/`, `Days/`, `DataResolution/`, etc.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds crash durability to Periscope: the async pipeline's emit-to-sink loss window — which used to swallow exactly the records describing a crash — closes to the microseconds between a record buffering and its synchronous journal append returning. Built on the Periscope framework from #69 (now merged).
The decision was made on measured data
Shared/Periscope/Prototypes/JournalBenchmark(committed here) benchmarked every candidate — append-only file (±fsync), raw SQLite WAL, GRDB, Core Data, SwiftData, each per-record and batched — for emit-path latency, contended throughput, and actual crash durability via child processes that SIGKILL themselves mid-stream:JournalKit — the generic layer
A new
Shared/JournalKitmodule with zero logging knowledge: opaqueDataentries framed[length][crc32][payload]into numbered segments that rotate at half the byte budget and drop oldest-first (flight-recorder posture, drops observable, accounting only counts removals that actually succeed).appendis synchronous from any thread — returning means the entry is in the kernel page cache and survives process death;.fulldoesF_FULLFSYNCfor kernel-panic coverage. A partial write (disk-full's shape) poisons its segment and the next append rotates, so torn bytes never strand later entries. An optionalsegmentHeaderre-writes as the first entry of every segment, so identity entries survive any amount of rotation. Recovery reads segments in order and never throws over a torn tail, pinned by a fuzz that cuts the file at every byte offset.The log layer in PeriscopeCore
Periscope. In-memory stores never journal — recovery is persistence.F_FULLFSYNCbefore the log call returns.startSession: prior sessions' journals replay at launch — records the drain never delivered persist (deduped by event ID), recovered span begans join the orphan sweep so crashed flows close as.orphaned, and ingested journals delete (failed ingest retries next launch). A marker records the recovery and its gaps: torn final entries and budget-rotated older entries appear in the message and escalate it from.noticeto.warning. Only app processes ingest — extensions journal their own sessions and leave recovery to the app's next launch, so an extension launch can't delete the live app's journal (full multi-process coordination is a tracked TODO).A knock-on: records the pending queue drops under overflow are still journaled, so drops became recoverable too.
Review hardening
A post-build review pass landed six fixes as dedicated commits: torn-write segment poisoning (a moment of disk pressure now costs one entry, not the rest of the session), gap-aware recovery markers — which exposed and fixed a real bug where rotation dropped the session entry first and orphaned the whole journal (hence segment headers), a content-plus-raw-bytes test pinning that the journal only ever writes post-redaction content (the highest-consequence regression a persistent log file could have), success-conditional drop accounting (failed deletes no longer drift the byte budget toward unbounded growth), app-only ingest, and the single-pass nested-JSON envelope — done pre-merge deliberately, while the wire format is still free to change.
Testing
JournalKit's suite: truncation fuzz at every cut point, CRC corruption, rotation, reopen, concurrent appenders, header survival under rotation, and seam-injected torn-write and removal-failure cases (bounded on-disk footprint under transient delete failures). Log-layer suites: records reach disk with the drain gated shut, span pairs journal in sequence order, redaction pinning at the content and raw-byte level, end-to-end store wiring, dedupe against delivered records, the recovered-began → orphan-sweep handoff, torn and rotated journals ingesting with honest
.warningmarkers, and sessionless-journal discard. FullStuff-iOS-Testsscheme and./swiftformat --lintpass locally.