Add Periscope: a typed, hierarchical observability framework#69
Merged
Conversation
Adds the three PeriscopeCore/PeriscopeUI/PeriscopeTools SPM library targets under Shared/Periscope/ with per-module README.md/AGENTS.md, placeholder sources, and hosted test bundles wired into Project.swift (unitTests helper, per-bundle schemes, Stuff-iOS-Tests CI scheme). Pure groundwork — no behavior yet; the framework's API lands in subsequent commits. Closes plan step: Scaffold PeriscopeCore/UI/Tools targets, docs, test bundles, and scheme wiring.
LogEvent is the Codable+Sendable protocol for structured events (stable eventName + eventVersion for persisted payloads, default .info level, rendered message). Message is the built-in freeform event the level conveniences will emit. LogLevel is an extensible struct (name + severity, OTel-style) so apps can define custom levels; OSLogType maps by severity band, with warning intentionally on .default like LogKit. Closes plan step: Core event model: LogEvent protocol, Message event, extensible LogLevel.
Log<Event> is a Sendable value struct over a deterministic scope tree: ScopeID derives from parent + name (SHA-256, namespaced), so the same path is the same scope in any process or launch. Deriving an event type creates a typed child scope; deriving for an identifier keys a child scope to an entity; + / linked(with:) merge scope sets so one event can reference both a model context and a UI context. Freeform level conveniences emit the built-in Message event on any typed logger. Records flow through the LogRecorder protocol (the Periscope system lands next; tests use an in-memory recorder). Closes plan step: Log<Event> value type: scope tree, id scoping, links via +.
Periscope is the process-wide recorder Logs emit into. Emitting never blocks: records and scope definitions append to one ordered, lock-based pending queue and a background drain task delivers batches to each LogSink (scope definitions always precede the records that reference them; late-added sinks get the scope registry replayed, idempotent per scope ID). A bounded recent-records buffer plus a per-record liveRecords() stream feed live UI. flush() awaits full delivery, then each sink's own flush. OSLogSink mirrors records to os.Logger with the root scope as category and the sub-path prefixed on the message; Periscope.shared ships with one under the main bundle ID. Log gains init(system: .shared) to match the planned Log<MyRoot>() spelling. Closes plan step: Periscope system: LogSink pipeline, non-blocking recorder, OSLog sink.
Level floors: Periscope.minimumLevel plus per-subtree overrides (nearest overridden ancestor wins; linked records pass when any of their scopes admits them), checked at emit before other work — Log's freeform conveniences consult shouldRecord first so filtered logging skips message rendering entirely. Flush policy: records at flushThreshold (default .error) trigger an automatic flush so high-severity context reaches sinks promptly. Drop policy: the pending queue is bounded; on overflow the oldest records drop (scope definitions never drop) and a synthetic DroppedEvents record under the Periscope system scope reports the gap. Redaction: an optional configuration hook transforms or suppresses every record before it is buffered or delivered anywhere. Closes plan step: Pipeline policies: level floors, flush/drop policy, redaction hook.
Conforming classes get a .log derived from type + instance identity: a root scope named after the type with numbered child scopes (#1, #2, …) per instance, cached by the system so an instance keeps its identity — no logger threading, no SwiftUI dependency, and a type's subtree finds every instance's events. LogEventType (default Message) types the logger; logSystem (default .shared) picks the system. Also hardens the scope-ordering test to index-based assertions now that systems define their own Periscope scope at init. Closes plan step: LogContextProviding protocol in Core for models and controllers.
log.withContext { … } (async and sync forms; async preserves caller
isolation via #isolation) binds the log's context to a @TaskLocal, so
Log.current — typed to any event, Log<Message>.current for freeform —
carries the full scope set anywhere in the async call tree, including
structured child tasks, without threading loggers through signatures.
Nested contexts link (inner log primary, duplicates collapse); with no
ambient context, Log.current falls back to a root logger on .shared.
Closes plan step: Task-local context propagation: withContext and
Log.current.
PeriscopeStore is the durable LogSink: a @Modelactor over SDLogEvent / SDLogScope / SDLogSession with #Index on the hot query fields. Events persist with their full scope hierarchy (event↔scope many-to-many plus the ordered scope list for display), a store-assigned monotonic sequence for stable same-millisecond ordering, a JSON payload keyed by eventName+eventVersion (decode(_:) recovers the type; tooling degrades to raw JSON), and the session that produced them — one SDLogSession per launch carrying app version, build, OS, and device model. Query API: events(matching:) filters by time range, level floor, event name, session, exact scope, scope subtree, and message search, newest first with limit/offset paging; scopes/sessions/event(id:) round out reads. Retention: pruneEvents(olderThan:)/(keepingNewest:), deleteAllEvents, and a changes() stream that pings after commits. Sink failures log to OSLog and count in an @_spi(Testing) counter — never silently dropped. Closes plan step: PeriscopeStore @Modelactor: schema, session metadata, indexes, retention, query API.
Tags are orthogonal to the scope hierarchy: log.tagged(key, value)
returns a context that stamps every event it emits (typed LogTagKey per
the identifier convention). Tags flow down derivations, merge across
links and nested ambient contexts (primary/inner side wins key
conflicts), mirror into OSLog messages as a sorted {k=v} suffix, and
persist via shared SDLogTag rows with an indexed key+value pair column.
LogQuery gains tag filtering, and the exact/subtree scope filters
collapse into one ScopeFilter enum (which also keeps the fetch
predicate within type-checker limits).
Closes plan step: Tag system: tagged contexts stamping events,
tag-based queries.
log.measure(token) { … } (sync and async, isolation-preserving) emits
paired SpanBegan/SpanEnded events sharing a SpanID, with the end event
emitted even on throw and durations measured by ContinuousClock —
tokens type-check against Event.SpanName (default String), so typed
events measure with leading-dot enums. log.begin(for:)/end(for:) open
and close spans keyed by primary scope + identifier via the recorder,
so a rebuilt logger on the same path closes the pair; mismatched calls
warn instead of silently dropping. Spans mirror to OSSignposter at
emission time (real durations in Instruments, not sink-delivery time).
The store persists an indexed spanID column with events(inSpan:) kept
separate from the general query predicate.
Closes plan step: Spans: measure closures, begin/end groups,
os_signpost mirroring.
LogAttachment (name, MIME content type, Data) attaches arbitrary payloads to any event: log(attachments:) for structured events and an attachments: parameter on the freeform level methods. Conveniences cover the common payloads — .error (description/domain/code as JSON), .json for any Encodable, and .image (PNG, UIKit-gated) — plus raw Data via the base init. Persistence uses @Attribute(.externalStorage) rows cascade-deleted with their event; queried events carry lightweight LogAttachmentInfo metadata and bytes load on demand through PeriscopeStore.attachments(forEvent:), so list fetches never pull blobs. Closes plan step: Attachments: LogAttachment API and external-storage persistence.
AmbientEventSource is the extensible seam for environmental context: Periscope.startAmbientSource(_:) retains a source for the process lifetime and hands it a logger under the shared ambient scope. Sources emit the standard AmbientEvent (typed AmbientKind + value, level raisable). Built-ins — started together via startDefaultAmbientSources() — cover app lifecycle transitions and memory warnings (UIKit-gated), network path changes via NWPathMonitor (connectivity and interface changes), thermal state (serious/critical at .warning), and Low Power Mode. Closes plan step: Ambient event sources: protocol plus built-in system observers.
PeriscopeCore's feature set is complete, so replace the scaffold-status README with the real overview (quick start, public API map, pipeline behavior, contracts) and expand the module AGENTS invariants (deterministic scope IDs, sink failure visibility, versioned JSON payloads). Docs-only change. Part of plan step: Ambient event sources (final Core docs pass).
View.logContext(_:) contributes a logger's scopes and tags to its descendants, linking onto whatever enclosing modifiers already contributed (nearest primary, tags merged — Log.linked(with:) semantics); an overload takes any LogContextProviding model directly, so .logContext(model.photo) + .logContext(screenLog) stack. Views read @Environment(\.logContext) — a Log<Message> that logs freeform immediately or derives typed loggers — with a Periscope.shared root fallback outside any modifier, mirroring Log.current. Core gains Log.retyped(to:), the context-preserving type change the environment accessor is built on. Module README/AGENTS updated to the real API. Closes plan step: PeriscopeUI: logContext modifier and environment accessor.
PeriscopeViewer renders a PeriscopeStore newest-first: searchable, filterable by level (standard ladder plus custom levels present), event type, scope subtree, and session, paged 200 at a time, with live refresh off the store's changes() signal and honest loading/failed/ empty states. Rows show a severity badge, event type, timestamp, and scope path; the detail screen shows tags, the pretty-printed payload JSON, and attachments loaded on demand. Export renders the active filter's full result set as NDJSON (deterministic sorted keys, oldest first, payload embedded as nested JSON) into a share sheet. Closes plan step: PeriscopeTools: latest-logs viewer with search, filters, NDJSON export.
LogTraceView walks backward from an origin event: the trail merges earlier events from the subtrees of all the origin's scopes (linked model + UI contexts both trace), events logged directly at ancestor scopes up each root chain (siblings excluded), and the origin's span pair — deduplicated and ordered newest first with the store's insertion sequence as tiebreak (StoredLogEvent now exposes sequence for exactly that). Every event detail gains a Trace button, and trail rows open their own detail so tracing can continue further back. Closes plan step: PeriscopeTools: log tracer across time and hierarchy.
PeriscopeAlerter watches a Periscope system's live records and routes everything at its threshold or above to a PeriscopeAlertHandler — the hookable seam, so apps with their own toast/notification stack conform instead of overriding UI. The built-in LocalNotificationAlertHandler posts each alerted record as a local notification under provisional authorization (quiet delivery, no permission prompt), logging post failures to OSLog rather than alerting recursively. start()/stop() manage the watch; only records emitted after start alert, and double start is a no-op. Closes plan step: PeriscopeTools: hookable debug toast for warning+ events.
View.logInspectable(_:) — taking a Log or a LogContextProviding model — badges the wrapped view while log view mode is on; tapping the badge presents every stored event in that context's scope subtrees (merged newest first across linked scopes, live-refreshing), each row linking into the standard event detail and from there the tracer. The mode's source of truth is a new Periscope.isInspectModeEnabled flag in Core; PeriscopeInspector is its observable SwiftUI mirror, injected once at the root via View.periscopeInspector(_:) and toggled from an app's developer settings. Finishes the PeriscopeTools docs (README quick start + AGENTS invariants). Closes plan step: PeriscopeTools: log view mode modifier for per-view event inspection.
A failed save previously left the ModelContext dirty: the poisoned batch's inserts stayed staged, every subsequent save re-attempted them, and a persistent failure silently lost all events from then on (a failed prune could even commit its staged deletions with the next unrelated write). Failure paths now run recoverFromFailedWrite(): rollback the transaction, drop the scope/tag row caches (they may hold rolled-back rows), and clear the session-row reference — the store now remembers its session identity as a value, and ensureActiveSession refetches or reinserts the same session on the next write, so recovery never forks the launch attribution. Throwing APIs (startSession, the prune/delete family) roll back and rethrow. Adds a DEBUG-gated @_spi(Testing) injectNextWriteFailure seam at the save funnels and covers each path: poisoned batch then healthy write, stale cache recovery with tags/placeholder scopes, session identity across recovery, scope-definition retry, and prune rollback. Addresses code-review finding 1 (store failure leaves context dirty).
InstanceScopeRegistry keyed instances by bare ObjectIdentifier, so a deallocated object's recycled pointer handed its cached identity — and its scope — to whatever object landed at that address next, even one of a different type. (The new InstanceIDTests demonstrated the reuse live: two back-to-back temporaries compared equal by address.) Two changes: InstanceID is the new cache key — pointer identity plus dynamic type, with a debugDescription that names the type — so a recycled address can never cross types; and each tracked instance now carries a retained deallocation tracker (ObjC associated object, keyed per registry so one object logged into two systems tracks both) whose deinit evicts the cache entry before the allocator can recycle the address, closing the same-type case too. Trackers hold the registry weakly so long-lived objects don't pin short-lived test registries. Instance numbers stay monotonic — #3 always means one specific instance within a run — and the registry no longer grows unboundedly. Addresses code-review finding 2 (instance identity via recycled pointers).
PeriscopeViewerModel.run() and LogInspectorModel.run() loaded first and subscribed to store.changes() after, so a commit landing between the two never triggered a refresh — a live viewer could sit stale until the next unrelated write. Both now acquire the stream before loading; a mid-load commit buffers in the stream and refreshes right after. Adds live-refresh tests for both models (previously untested), racing a write against the initial load deliberately. Addresses code-review finding 3 (subscribe-after-load gap).
Every record at the flush threshold spawned its own Task { flush() },
each looping on the drain until the system went quiet — an error storm
piled up one task per record for no added durability. Records now
request the flush through a coalescing gate: a single auto-flush task
runs at a time, and qualifying records that land mid-flush set a
pending flag that grants exactly one follow-up flush (covering their
durability) before the task retires. Deterministic storm test holds
the drain with a gated sink while 50 errors land and asserts at most
two sink flushes; a second test covers re-arming after settling.
Addresses code-review finding 4 (auto-flush task pileup).
liveRecords() streams used AsyncStream's default unbounded buffering, so a slow or stuck consumer (a paused toast handler, a backgrounded inspector) accumulated every emitted record in memory indefinitely. Streams now buffer with .bufferingNewest under the new Configuration.liveBufferCapacity (default 256): a consumer that falls behind loses the oldest buffered records — live surfaces want the newest activity, and durable history is the store's job. Test covers the drop-oldest behavior and normal flow after catching up. Addresses code-review finding 5 (unbounded live-stream buffering).
Spans could leak forever: a lost end(for:) left its entry (and its signpost interval) in memory permanently, and permanently locked out re-begins for that key. Spans now have full lifecycle semantics: - SpanLifetime (explicit on begin): .bounded(budget:) spans are closed as .expired by a deadline-driven watchdog when they outlive their budget (generation-tagged task, earliest-deadline sleep; the sweep is clock-injectable via @_spi so tests never sleep); .indefinite is the conscious opt-in for open-ended flows; .scoped marks measure spans. - SpanExit on every SpanEnded: success/failure/cancelled (with optional reasons) plus system modes superseded/expired/orphaned. measure now derives exits automatically — thrown errors record as .failure and CancellationError as .cancelled instead of masquerading as success. Abnormal exits log at .warning; messages describe the exit. - Re-begins supersede: beginning an already-open key closes the prior span as .superseded (attributed with its begin-time scopes and tags, which OpenSpan now carries) instead of refusing — no lockout. - SpanRelaunchPolicy, recorded on the SpanBegan payload: at startSession the store closes unmatched endsWithProcess spans from earlier sessions as .orphaned (duration nil — unknowable across process death); survivesRelaunch spans stay open, with resume mechanics staged in Shared/Periscope/TODOs.md. SpanBegan/SpanEnded bump to eventVersion 2 (first real exercise of the versioned-payload contract). Adds Shared/Periscope/TODOs.md, seeded with the staged span work and the remaining code-review findings. Addresses code-review finding 6 (open-span and signpost leaks).
Periscope.chunked(_:) grouped pending items by rewriting the last chunk with array + [element], copying the accumulated run on every iteration — quadratic in batch size, so a full 5,000-record backlog did ~12.5M element copies inside the drain. Runs now accumulate in mutable buffers and close when the item kind flips: O(n). Adds a gated-drain regression test asserting an interleaved backlog (records, scope definitions, records, …) reaches the sink as ordered runs. Addresses code-review finding 7 (quadratic chunking).
eventValue maps every fetched row's tags and attachments relationships into the returned value, so each row of a 200-event viewer page faulted both relationships in their own round trips (N+1). Event-read descriptors (events(matching:), events(inSpan:), fetchEventRow, and the orphan sweep's began fetch, which reuses row tags) now share a readDescriptor helper that sets relationshipKeyPathsForPrefetching for both. Attachment blobs still load lazily — only the metadata rows prefetch. Behavior unchanged; existing read-path tests cover it. Addresses code-review finding 8 (N+1 relationship faulting).
The new seeded fuzz test (fixed SplitMix64 seeds, four concurrent tasks interleaving emit/derive/flush/add-sink) immediately caught a real ordering bug: add(sink:) appended its scope replay to the END of the pending queue, so records already pending reached the new sink before the definitions of the scopes they reference. The replay is now prepended, and the drop report slots after the leading scope run rather than being written first, preserving the same guarantee. The fuzz invariants assert no record loss or duplication, per-emitter emission order, and scopes-before-records on every sink including late-added ones. Also covers the drop policy's scope-definitions-never-drop promise (gated overflow with interleaved definitions and records), and makes the alerter lifecycle tests deterministic: a new @_spi(Testing) Periscope.liveObserverCount asserts the subscription count directly instead of racing duplicate deliveries against a Task.yield(), and the stop test verifies unsubscription before emitting rather than relying on a sentinel from a second alerter. Addresses the code review's missing-tests items.
Periscope.record ran the redaction hook on every record before the floor check, so user redaction code executed — touching PII — for records the floor was about to discard (every filtered debug message under a .warning floor). Floors now gate first; redaction runs only for admitted records. Contract documented on Configuration.redact: floors apply to the record as emitted — redaction is content scrubbing, not routing. Test asserts the hook never fires for floor-filtered records (freeform or structured) and exactly once for admitted ones. Addresses code-review low-severity item: redaction-before-floor ordering.
Resolves the AGENTS.md targets-section conflict by combining main's Foreman removal with this branch's Periscope additions. Package.swift and Project.swift auto-merged: the package is now iOS-only (main dropped the .macOS platform with Foreman), and the Foreman-macOS-Tests scheme is gone.
Foreman's removal on main dropped the .macOS platform from Package.swift, so PeriscopeUI/PeriscopeTools no longer advertise a platform they can't compile for — the latent break is gone from the other side. Core keeps its canImport(UIKit) gating as per-SDK hygiene. Docs-only change (TODOs.md). Addresses code-review low-severity item: macOS compilation of the UI modules.
Review walkthrough finding 1: keep the ergonomic no-arg Log<Event>() init (records into Periscope.shared), and note in the module AGENTS.md that it's a conscious exception to the no-Core-defaults convention — tests must always pass system: explicitly.
Review walkthrough finding 2: the 'cannot fail' encode of the error payload used (try? ...) ?? Data(), which would silently persist a zero-byte application/json attachment that reads as a successful capture if a refactor ever made encoding fail. Per the programmer-error convention (and matching the NDJSON exporter's existing treatment): assertionFailure in debug, a valid empty JSON object as the release fallback.
Review walkthrough finding 3: a stored payload that no longer parses (on-disk corruption — payloads are JSONEncoder output at persist time) was silently dropped from the export line, indistinguishable from an event that never had one. The line now carries a payloadError marker with a byte-count hint, so a bug-report export is honest about exactly the row someone is likely chasing. Covered by an exporter test feeding garbage payload bytes.
Review walkthrough finding 4: exitReason (a SpanEnded's freeform reason lives in the payload, not a column) and prettyPayload were private on the view, so their degradation paths — an undecodable payload, garbage bytes — had no direct coverage. They're now internal StoredLogEvent extensions in the same file, pure functions of the event, with a 1:1 LogEventDetailViewTests covering reason round-trip, reasonless exits, undecodable payloads, key-sorted pretty printing, and garbage degradation.
Review walkthrough finding 5: the LogSpan file comment and README
claimed span names are 'typed tokens, not raw strings', but
Event.SpanName defaults to String — measure("save") is the
out-of-box spelling. Dropping the default isn't viable (an
associatedtype with no default or inference path would force an empty
SpanName enum onto every LogEvent conformance, Message included), so
the docs now state the actual contract: names resolve against
Event.SpanName, String by default; declare a SpanName enum for
compiler-checked leading-dot tokens, the recommended style for
structured events.
kyleve
commented
Jul 14, 2026
|
|
||
| ## How it works | ||
|
|
||
| Log call sites never block: records append to a lock-guarded pending queue |
Owner
Author
There was a problem hiding this comment.
I'm not 100% convinced of this as it can result in dropped events around crash times; exactly when you'd want the logs. Maybe we should write each event to a file synchronously so we never lose any, and then the async queue ingests them? Or just write directly to the store and skip the need for this complexity entirely? Thoughts?
| } | ||
| } | ||
|
|
||
| extension AmbientKind { |
Owner
Author
There was a problem hiding this comment.
Let's add an accessibility observer here too so we know which accessibility settings are enabled.
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Added in 3f9bae3 — AccessibilityAmbientSource logs a summary of enabled settings at start (the state the session's events read against) plus a change event per toggle, across VoiceOver, Switch Control, Reduce Motion, Reduce Transparency, Bold Text, Darker Colors, Invert Colors, and Grayscale. Wired into startDefaultAmbientSources() on UIKit platforms. (The broader ambients-persist-current-state idea from your other comment is tracked as a design-loop item.)
| public struct LowPowerModeAmbientSource: AmbientEventSource { | ||
| public init() {} | ||
|
|
||
| public func start(log: Log<AmbientEvent>) { |
Owner
Author
There was a problem hiding this comment.
Hmm, I think we need a way to stop ambients too, right?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Agreed — added in da2b544: AmbientEventSource gains stop() (remove observers, cancel monitors; nothing may keep logging after it returns), and Periscope.stopAmbientSources() stops and releases every retained source as the counterpart to startAmbientSource(_:). NetworkPathAmbientSource.stop() cancels its monitor.
| public init() {} | ||
|
|
||
| public func start(log: Log<AmbientEvent>) { | ||
| _ = NotificationCenter.default.addObserver( |
Owner
Author
There was a problem hiding this comment.
Doesn't not retaining this token means the observation will go away?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Confirmed — and counterintuitively, the observation did not go away: the block-based addObserver(forName:) API retains the observation in the center regardless of the token. Dropping the token made it unremovable and immortalized everything the block captured, including the logger's whole Periscope system — and a restart would have doubled observers. Tokens now live in AmbientObserverTokens (public, since app-defined notification sources face the same trap), and restarts swap the observation wholesale. Fixed in da2b544, with stop/restart tests using test-unique notification names.
| public init() {} | ||
|
|
||
| public func start(log: Log<AmbientEvent>) { | ||
| _ = NotificationCenter.default.addObserver( |
Owner
Author
There was a problem hiding this comment.
Same comment as above re: not maintaining the token.
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Same fix as the low-power source — token retained via AmbientObserverTokens in da2b544.
| } | ||
|
|
||
| /// Every recorded session, newest first. | ||
| public func sessions() throws -> [LogSession] { |
Owner
Author
There was a problem hiding this comment.
Any concerns about this getting very very large over weeks/months?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Real concern — retention only bounded the events table: one session row per launch and one tag row per distinct pair (unbounded when values are entity IDs) accumulated forever. As of 12e0e05, every prune/deletion also removes orphaned metadata in a follow-up save: sessions with no remaining events (except the active launch's), event-less tag rows, and event-less scope branches unwinding leaf-first so ancestors of populated scopes survive for path resolution.
| /// save after it, and drop state that may reference rolled-back rows: | ||
| /// the row caches, and the session-row reference (`ensureActiveSession` | ||
| /// refetches or reinserts the same session identity on the next write). | ||
| private func recoverFromFailedWrite() { |
Owner
Author
There was a problem hiding this comment.
Do we log an event to mark / denote that there was a write failure?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
It does now — 72bee98 persists a synthetic StoreWriteFailed marker (lost record count + reason, at .warning) in its own tiny save after recovery, mirroring the pipeline's DroppedEvents pattern. Best-effort by design: if the marker's save also fails, the OSLog line is the last signal — no recursion.
Owner
Author
There was a problem hiding this comment.
This file is very large and it does a lot; should we break some of the responsibilities into sub-objects?
| case failed(String) | ||
| } | ||
|
|
||
| static let limit = 200 |
Owner
Author
There was a problem hiding this comment.
200 seems low, maybe 500? And we make this an instance property so it's customizable?
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Done in 800a8d8 — 500, and per-instance: the model takes limit at init, the logInspectable modifiers expose it (default 500), and it's part of the task rebuild key so an in-place change rebinds the model.
| } | ||
|
|
||
| /// Cap on the assembled trail (and on each underlying query). | ||
| static let limit = 100 |
Owner
Author
There was a problem hiding this comment.
Same comment as above; seems low; and should be configurable.
Owner
Author
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Same in 800a8d8 — LogTraceView(store:origin:limit:) with a 500 default, plumbed into the model and the rebuild key.
kyleve
commented
Jul 14, 2026
| /// static let pushToken = AmbientKind("push-token") | ||
| /// } | ||
| /// ``` | ||
| public struct AmbientKind: Hashable, Sendable, Codable, CustomStringConvertible { |
Owner
Author
There was a problem hiding this comment.
Alongside logging actions / changes, I think Ambients' current state should also be logged alongside each event, like we do sessions. That way for a given log event, you could look up the system state.
That means that Ambients evolve to both monitor for changes via start, they should also persist their current state.
PR feedback (ambient token comments): the notification-based ambient sources discarded the token addObserver(forName:) returns. The center keeps the observation alive regardless — so events still flowed, but the observation was unremovable and immortalized everything the block captured, including the logger's whole Periscope system; a restart would also have doubled observers. Tokens now live in AmbientObserverTokens (public — app-defined notification sources face the same trap), which swaps observations wholesale on restart. PR feedback (stop comment): AmbientEventSource gains stop(), and Periscope.stopAmbientSources() stops and releases every retained source — the counterpart to startAmbientSource(_:). NetworkPath's stop cancels its monitor. Covered by stop/no-longer-observes and restart-replaces-observation tests using test-unique notification names.
PR feedback: content types were raw MIME strings. LogAttachment.ContentType models the common capture types as cases (json, png, jpeg, plainText) with .other(String) carrying any uncommon MIME type losslessly; the store persists the MIME string and maps back on read, so no schema change. Factories and LogAttachmentInfo use the typed cases; the detail view displays via mimeType.
PR feedback: osLogType was computed from severity bands with no way for a custom level to choose its Console visibility. It's now a stored property — the severity-band default (unchanged behavior, via defaultOSLogType(forSeverity:)) or an explicit value from the new init(name:severity:osLogType:). The mapping is routing metadata, not identity: equality/hashing stay name + severity, and Codable persists only those two (payloads never carried the mapping), so decoded levels re-derive the band default — asserted by tests. The standard ladder's statics pass their mapping explicitly: defaultOSLogType(forSeverity:) reads those very statics to find its band edges, so routing the ladder's own initializers through it was reentrant static initialization — a runtime trap the first test run caught.
PR feedback: success/failure/cancelled had reason-taking factories but superseded, orphaned, and expired did not (expired only had the budget-deriving one). Every mode now has one, asserted against Mode.allCases so a new mode can't ship without its factory.
PR feedback (three comments): tag values were raw Strings and tag
collections were [LogTagKey: String] dictionaries; queries took one
tag.
- LogTagValue types values (string/int/double/bool, plus any Codable
via .encoding as canonical key-sorted JSON). Literals convert, and
tagged(_:_:) accepts plain String/Int/Double/Bool variables through
LogTagValueConvertible. The persisted pair string gains a kind
segment so .int(3) and .string("3") stay distinct in storage and
filters; SDLogTag grows a valueKind column and values round-trip
typed.
- Log, LogRecord, OpenSpan, StoredLogEvent, and the ambient context
carry [LogTag] (unique by key, insertion-ordered — re-tagging
replaces in place, links append the right side's new keys) instead
of dictionaries.
- LogQuery.tag becomes tags: [LogTag], ANDed. The hand-built predicate
avoids a dynamic expression tree: a row can't carry duplicate pairs,
so matching all N tags is 'N of its tag rows have a pair in the
list' — filter + count, verified against SQLite by a two-tag store
test.
NDJSON exports tag values as native JSON types (numbers stay numbers);
the detail view shows canonical strings.
PR feedback: InstanceID cached the type name string (plus a separate type ObjectIdentifier) at init. It now stores the Any.Type itself — metatypes are Sendable and identity-comparable — and typeName derives from it when asked. Equality and hashing go through the metatype's ObjectIdentifier, unchanged in behavior.
PR feedback: the drop policy identified span pairs by matching concrete types (event is SpanBegan || event is SpanEnded). LogEvent now declares static isProtectedFromDropping (default false); the span pair events opt in, and any app event whose absence would corrupt the story its log tells can too. The transform-only redaction rule rides the same flag, so a protected event can't be suppressed into a dangling reference either. Covered by a custom-event overflow test.
PR feedback (two comments): the inspector's 200 and tracer's 100 were static constants. Both models now take limit at init; LogTraceView and the logInspectable modifiers expose it with a 500 default, and the keyed-task inputs include it so an in-place change rebuilds the model.
PR feedback: a failed write rolled back, counted, and logged to OSLog — but the durable history stayed silent about its own gap. The store now persists a synthetic StoreWriteFailed event (lost record count + reason, .warning) in its own tiny save after recovery, mirroring the pipeline's DroppedEvents pattern. Best-effort by design: if the marker's save also fails, the OSLog line is the last signal — no recursion. Scope/delete failures don't mark; definitions rebuild deterministically on the next batch and deletions lose nothing.
PR feedback: retention only bounded the events table — one session row per launch and one tag row per distinct pair (unbounded when values are entity IDs) accumulated forever, and event-less scope branches lingered. Every event deletion now runs an orphan sweep in a second save (inverse relationships only prove orphanhood after the deletions commit): sessions with no remaining events go (except the active launch's), event-less tag rows go (their cache entries dropped so a reused pair re-creates cleanly), and scopes unwind leaf-first so ancestors of still-populated scopes survive for path resolution.
PR feedback: every emit surface — the structured callAsFunction
overloads (including derive-and-emit) and the freeform level
conveniences — now captures its call site via defaulted
#function/#fileID parameters into LogCallSite, carried on LogRecord,
persisted as columns, exposed on StoredLogEvent, shown in event detail
('Emitted From'), and exported in NDJSON. System-synthesized records
(drop reports, orphan closes, expiry, supersede ends) stay nil — no
call site to claim.
PR feedback: events can now declare an optional externalID — a
photo's local-store URI, a managed object ID's URI form — tying every
event about an object together. Defaults to nil; the format is the
app's. Persisted as an indexed column, exposed on StoredLogEvent,
queryable via LogQuery.externalID ('every event about this object' is
one indexed fetch), shown in event detail, and exported in NDJSON.
Tooling that resolves the IDs back to live objects stays future work,
as the comment envisioned.
PR feedback: logs which accessibility settings are enabled — one summary event at start (the state a session's events read against) and a change event per toggle — across VoiceOver, Switch Control, Reduce Motion, Reduce Transparency, Bold Text, Darker Colors, Invert Colors, and Grayscale. UIAccessibility statics are main-actor, so observers ride the main queue and the summary hops once at start. Wired into startDefaultAmbientSources on UIKit platforms.
Groundwork/docs only: the completed bug + mechanical-reshape items from the PR review land in the completed section, and the six larger design questions (crash durability, span record modeling, Periscope/store decomposition, readable ScopeIDs, context parent hierarchy, ambient state snapshots) are captured as open design items for the upcoming plan/build loops. AGENTS invariants updated for the event-level drop-protection opt-in and the StoreWriteFailed marker.
Conflicts were confined to AGENTS.md's library/target lists — merged to include both sides (Periscope modules and bundles alongside TestHostSupport, Broadway, and the Where app extensions). Periscope's test bundles move from the removed WhereTesting to TestHostSupport (same show()/hostKeyWindow() helpers; WhereTestingError is now TestHostError), which the merged unitTests helper already wires. Full Stuff-iOS-Tests scheme passes on the merged tree.
kyleve
commented
Jul 15, 2026
| /// it makes it unremovable (and immortalizes everything the block | ||
| /// captures, including the logger's whole system). Public because | ||
| /// app-defined notification-based sources face the same trap. | ||
| public struct AmbientObserverTokens: Sendable { |
Owner
Author
There was a problem hiding this comment.
This pattern is weird to me. Just make the ambient observers be classes which can hold their own state.
kyleve
added a commit
that referenced
this pull request
Jul 15, 2026
## Summary Adds **crash durability** to Periscope: the async pipeline's emit-to-sink loss window — which used to swallow exactly the records describing a crash — closes to the microseconds between a record buffering and its synchronous journal append returning. Built on the Periscope framework from #69 (now merged). ### The decision was made on measured data `Shared/Periscope/Prototypes/JournalBenchmark` (committed here) benchmarked every candidate — append-only file (±fsync), raw SQLite WAL, GRDB, Core Data, SwiftData, each per-record and batched — for emit-path latency, contended throughput, and *actual* crash durability via child processes that SIGKILL themselves mid-stream: - Every **per-record-commit** variant survived SIGKILL completely — including unfsynced file appends (page-cache writes survive process death, empirically). Both **batched** ORM variants lost exactly their unsaved tail. - The **file append is the only candidate whose worst case stays in microseconds** (p50 1.3µs, max 40µs, 560k ops/s). Every SQLite-backed option has multi-millisecond emit-path tails from WAL checkpoints or save machinery; SwiftData per-record hit a 790ms contended max. ### JournalKit — the generic layer A new `Shared/JournalKit` module with zero logging knowledge: opaque `Data` entries framed `[length][crc32][payload]` into numbered segments that rotate at half the byte budget and drop oldest-first (flight-recorder posture, drops observable, accounting only counts removals that actually succeed). `append` is synchronous from any thread — returning means the entry is in the kernel page cache and survives process death; `.full` does `F_FULLFSYNC` for kernel-panic coverage. A *partial* write (disk-full's shape) poisons its segment and the next append rotates, so torn bytes never strand later entries. An optional **`segmentHeader` re-writes as the first entry of every segment**, so identity entries survive any amount of rotation. Recovery reads segments in order and never throws over a torn tail, pinned by a fuzz that cuts the file at **every byte offset**. ### The log layer in PeriscopeCore - **Default on, store-owned**: on-disk stores open a per-session journal beside the database; adding the store as a sink installs it into `Periscope`. In-memory stores never journal — recovery *is* persistence. - **Ordering without contention**: the replay sequence is stamped under the pipeline's state lock (buffer order, span pairs preserved) but file I/O happens outside it, so emitters never serialize on disk. - **Fault+ records `F_FULLFSYNC`** before the log call returns. - **Ingest before `startSession`**: prior sessions' journals replay at launch — records the drain never delivered persist (deduped by event ID), recovered span begans join the orphan sweep so crashed flows close as `.orphaned`, and ingested journals delete (failed ingest retries next launch). A marker records the recovery *and its gaps*: torn final entries and budget-rotated older entries appear in the message and escalate it from `.notice` to `.warning`. **Only app processes ingest** — extensions journal their own sessions and leave recovery to the app's next launch, so an extension launch can't delete the live app's journal (full multi-process coordination is a tracked TODO). - **Wire format**: entries are versioned JSON envelopes with the payload nested as a JSON object (single-pass encoding, no base64 inflation); unknown kinds from newer builds skip at ingest instead of failing it. The session entry rides as every segment's header, so a rotated journal always attributes itself. Attachments over 64KB journal as omitted markers. - **Failure posture**: journal problems never throw into the emit path — they count and log to OSLog, throttled. A knock-on: records the pending queue drops under overflow are still journaled, so drops became recoverable too. ## Review hardening A post-build review pass landed six fixes as dedicated commits: torn-write segment poisoning (a moment of disk pressure now costs one entry, not the rest of the session), gap-aware recovery markers — which exposed and fixed a real bug where rotation dropped the session entry first and orphaned the whole journal (hence segment headers), a content-plus-raw-bytes test pinning that the journal only ever writes post-redaction content (the highest-consequence regression a persistent log file could have), success-conditional drop accounting (failed deletes no longer drift the byte budget toward unbounded growth), app-only ingest, and the single-pass nested-JSON envelope — done pre-merge deliberately, while the wire format is still free to change. ## Testing JournalKit's suite: truncation fuzz at every cut point, CRC corruption, rotation, reopen, concurrent appenders, header survival under rotation, and seam-injected torn-write and removal-failure cases (bounded on-disk footprint under transient delete failures). Log-layer suites: records reach disk with the drain gated shut, span pairs journal in sequence order, redaction pinning at the content and raw-byte level, end-to-end store wiring, dedupe against delivered records, the recovered-began → orphan-sweep handoff, torn and rotated journals ingesting with honest `.warning` markers, and sessionless-journal discard. Full `Stuff-iOS-Tests` scheme and `./swiftformat --lint` pass locally.
kyleve
added a commit
that referenced
this pull request
Jul 15, 2026
## What Trims the largest `AGENTS.md` files down to invariants, hard-fought lessons, and instructions — dropping structure enumerations agents re-derive from the manifests in their first minute of searching — and regroups WhereCore's flat `Sources/` into concern-based subdirectories. One commit per step. **Guiding principle:** an AGENTS.md keeps invariants, lessons, conventions, and recipes; it drops module lists, directory trees, and dependency catalogs that mirror `Package.swift` / `Project.swift`. The root file's libraries list was already stale (missing `WhereIntents` and `SwiftDataInspector`) — proof that content rots. ## Commits 1. **GitHub rules** — new root **GitHub** section: use `gh` for all GitHub interaction; open PRs ready-for-review, never draft; keep open PRs current by pushing each local commit. "Working on PR feedback" now requires replying to a comment when a commit resolves it, and filing deferred feedback durably instead of dropping it. 2. **Root trim** — libraries/targets enumerations become pointers at the manifests; directory layout collapses to the module skeleton. The add-a-target recipe, CI-scheme rule, and the double-linking lesson stay verbatim. 3. **Where trim** — Modules tree becomes a one-paragraph layering stack; developer-overlay tour and feature narration cut; the stylesheet paragraph now points at `WhereUI/AGENTS.md`. 222 → 189 lines. 4. **Module trims** — WhereUI / WhereCore / WhereIntents lose their `Package.swift`-mirroring dependency listings; PeriscopeCore loses its directory catalog. All invariants and the full `WhereStylesheet` guidance stay. 5. **WhereCore regroup** — pure `git mv`, no behavior change: stragglers fold into `Backup/`, `Reminders/`, `Widgets/`, `Location/`, `Evidence/`, `DataResolution/`, `Persistence/`; new `Days/`, `Reporting/`, `Journal/`, `Preferences/` groups. Only `WhereServices*` and `WhereLog` remain top-level. Tests stay flat, matching PeriscopeCore. ## New lessons codified (mined from the last 30 days of PR review) - Prefer synthesized `Codable`; persisted wire formats must survive Swift renames; hand-written conformances document their load-bearing reason (#85, #86) - Retain block-based notification-observer tokens; every `start` has a paired `stop()` (#69) - Group large flat types into sub-structs / child types per behavioral area (#47, #69) - Test doubles conform to the production protocol, never an enum switch in a production type (#54) - Custom full-screen surfaces are accessibility-modal and post focus notifications (#70) - Derive UI dimensions from the live UI (preference keys, `@ScaledMetric`, semantic font styles) instead of repeating magic numbers (#38, #70, #83) - WhereCore: post-write reconciliation is defined once — writes/imports route through `DayJournal.reconcileAfterDayChange()`; cross-collaborator hooks are a single closure (#80, #88) ## Verification - `./swiftformat --lint` clean; `./sync-agents` run after each docs commit - Full `Stuff-iOS-Tests` scheme after the regroup: 1183 tests, 0 failures Made with [Cursor](https://cursor.com)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Periscope, a typed, hierarchical observability framework, as three new SPM modules under
Shared/Periscope/, wired intoPackage.swift,Project.swift, and theStuff-iOS-TestsCI scheme. It coexists withLogKit/LogViewerUI— noWhereLogmigration in this PR.PeriscopeCore — model and machinery
LogEvent(Codable; stableeventName+eventVersion), freeformMessage, extensibleLogLevelstruct (OTel-style name + severity).Log<Event>value loggers over a deterministic scope tree (same path = same scope in any process/launch): typed children (root(PhotoLogs.self)), entity children (photos(for: album.id)), single-expression derive-and-emit (album(for: id) { .uploaded }), links (model + ui), tags (tagged(.paymentID, id)), attachments (withError/Codable/image conveniences).withContext/Log.current, andLogContextProvidingfor derived per-instance loggers (identity-safe viaInstanceID+ associated-object deallocation trackers, so a recycled pointer can never inherit a dead object's logging identity).measurewith auto-derived exits (success/failure/cancelled) and optional budgets (aSpanOverduewarning fires while the closure hangs, race-free against the span's own end),begin/endwith explicitSpanLifetime(bounded spans expire via a clock-driven watchdog),SpanExitmodes,SpanRelaunchPolicy(relaunch orphan-closes spans the dead process left open),OSSignpostermirroring.LogSinkfan-out (OSLog sink built in) with level floors (global + per-subtree), coalesced auto-flush at error+, bounded drop policy with syntheticDroppedEventsgap reports, redaction hook, and a bounded live-records stream that replays exact buffered order.PeriscopeStore(@ModelActorsink) — indexed schema, per-launch sessions (app/OS/device metadata), versioned JSON payloads that outlive their Swift types, external-storage attachments, rich query API (time/level/type/session/scope-subtree/tag/span-exit/search, paged), retention pruning, rollback-on-failure so a poisoned batch can't wedge later saves.AmbientEventSourcewith built-ins for app lifecycle, memory warnings, network path, thermal state, and low power mode.PeriscopeUI
.logContext(_:)modifiers (acceptingLogvalues orLogContextProvidingmodels; stacking links contexts) and the\.logContextenvironment accessor.PeriscopeTools
PeriscopeViewer(searchable/filterable latest-logs viewer — level, event, scope, session, span exit — with NDJSON export),LogTraceView(walks an error back through time, linked scopes, and the ancestor chain),OpenSpansView(what's in flight right now, longest-running first),PeriscopeAlerter(hookable debug toast, local-notification default with cached authorization), and log view mode (.logInspectable(_:)+PeriscopeInspector, two-way synced with the system flag).Review hardening
Three full code-review passes followed the initial build, each landing fixes as dedicated commits.
First pass — every high/medium finding fixed: store rollback + failure injection, instance identity via dealloc trackers, subscribe-before-load in live models, coalesced auto-flush, bounded live buffers, the span lifecycle model, linear drain chunking, relationship prefetching. The closing seeded fuzz test caught a real bug —
add(sink:)appended its scope replay after already-pending records, so late-added sinks could see records before their scope definitions; the replay is now prepended.Second pass — the orphan sweep fetches only
spanIDcolumns; inspect-mode changes yield inside the state lock so racing setters can't strandbufferingNewest(1)subscribers; span pairs floor together (the begin-time decision governs the whole lifecycle — no dangling halves across floor changes); the watchdog holds the system weakly. Low items: overdue sentinels serialize with their span's end through a per-measure gate, the six copies of scope-path walking collapsed intoLogScope.ancestry(of:resolve:),showHostedgotshow()'s animation-speed handling, and the viewer clears export state on store swaps. Also fixed here: the store's nine-condition filter predicate is hand-built in statement form (PredicateExpressions, oneletper condition) after the#Predicatemacro's single inference tree blew the type-checker budget on CI runners.Third pass — span pair integrity. Writing a second seeded fuzz (spans, floor flips, cross-task supersession, drop pressure) surfaced that "no dangling halves" had three remaining attack vectors; all are now closed, and the invariant holds on every channel:
LogRecorder.beginSpanregisters a span and buffers itsSpanBeganunder one lock acquisition, so no racing supersede orend(for:)can deliver a span's end before its began.SpanBegan/SpanEnded(like scope definitions): a dropped began would strand its end; a dropped end reads as "still open" until the next launch's orphan sweep.SpanOverduestays droppable.nil(suppression) falls back to a stripped copy (tags/attachments dropped, exit reason blanked) instead of stranding the partner; floors are the supported way to silence spans.Live streams got the same ordering rigor: observer yields moved inside the state lock (
record()andbeginSpannow share one delivery helper), soliveRecords()consumers see exact buffered order — previously two racing emitters could invert live delivery even though sinks and the store were ordered.Remaining staged work (
survivesRelaunchresume mechanics) is tracked inShared/Periscope/TODOs.md, which also logs every completed finding.Review response (post-review feedback batch). Every non-design review comment landed as its own commit: the ambient observer-token bug (the block-based API keeps observations alive in the center — dropped tokens made them unremovable and immortalized the captured system;
AmbientObserverTokens+stop()/stopAmbientSources()fix it), typed tag values (LogTagValue) with[LogTag]lists and multi-tag AND queries,LogAttachment.ContentTypeas an enum, per-levelosLogTypeoverrides, the missingSpanExitfactories,InstanceIDstoring the metatype, drop protection as aLogEventopt-in,#function/#fileIDcapture on every emit,LogEvent.externalID(indexed and queryable), aStoreWriteFailedmarker after rolled-back writes, retention pruning orphaned sessions/tags/scope branches, an accessibility-settings ambient source, and configurable tool caps (500). The six larger design comments (crash durability, span record modeling, decomposition, readableScopeIDs, context parent hierarchy, ambient state snapshots) are tracked inTODOs.mdfor follow-up loops — crash durability is already built and stacked as #86.Testing
284 Swift Testing tests across 40 files in three hosted bundles (
PeriscopeCoreTests,PeriscopeUITests,PeriscopeToolsTests): deterministic pipeline tests via gated sinks, clock-injected watchdog sweeps, store failure injection, hosted SwiftUI tests for the tools, and two seeded fuzz suites — pipeline interleavings (no loss/duplication, per-emitter order, scopes-before-records on every sink) and span lifecycles (strict began-then-ended pairs under concurrent supersession, floor flips, and drop pressure; empty open-span registry after cleanup)../swiftformat --lintandStuff-iOS-Testspass locally and on CI.