Skip to content

Add resumable per-provider metadata archives#104

Open
jmcte wants to merge 5 commits into
mainfrom
codex/issue-93-provider-archives
Open

Add resumable per-provider metadata archives#104
jmcte wants to merge 5 commits into
mainfrom
codex/issue-93-provider-archives

Conversation

@jmcte

@jmcte jmcte commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add versioned per-provider archive sync, document, status, and migration contracts
  • persist cursors, source fingerprints, attempts/successes, freshness, item counts, and redacted failures
  • support bounded idempotent upserts, explicit tombstones, retention, restart/resume, and private atomic storage
  • fail closed for providers and nested body/media fields that have not explicitly opted into metadata archiving

Governing Issue

Closes #93

Validation

  • Relevant local checks passed
  • Required PR checks are expected to satisfy CI Gate
  • Skipped checks are explained below

swift test --disable-sandbox — 130 tests passed

bash scripts/ci/run-fast-checks.sh — passed; source line coverage 89.13%, mutation checks killed expected mutations, debug and release builds passed

Synthetic tests cover restart/resume, idempotency, explicit tombstones, persisted counts, v0 migration, scan and timeout partial state, source partial failure, retention, permissions, CLI sync/status, nested JSON fields, and sensitive-provider/field rejection.

Bootstrap Governance

  • Changes are scoped to the linked issue
  • Contributor or PR guidance changes are reflected in CONTRIBUTING.md, .github/PULL_REQUEST_TEMPLATE.md, and docs/bootstrap/onboarding.md when applicable
  • PR author enabled auto-merge where GitHub allows it, or GitHub plan-limit evidence/unavailable reason is recorded and the fallback merge-readiness policy applies
  • No real secrets, runtime auth, or machine-local env files are committed

This branch is stacked on foundation PRs #101, #102, and #103. The PR remains draft until those merge and the diff collapses to the issue #93 commit; auto-merge will then be armed.

Flow Contract

  • Owner lane: Hermes macOS/native archive foundation
  • Repair owner: jmcte
  • Autonomy class: review-gated implementation from assigned issue
  • Risk class: high; durable private metadata storage and migration contract

Flow Merge Readiness

  • Every blocker has a next actor and next action
  • No active blocking requested changes remain
  • Non-author approval is present when required
  • PR author enabled auto-merge where GitHub allows it, or recorded why it is unavailable/unsafe

Next actor: foundation reviewers, then security/architecture/macOS review on this PR.

Merge Automation

  • PR author enabled auto-merge with gh pr merge --auto --squash, or the reason it is unavailable/unsafe is noted below

Auto-merge is intentionally deferred while the PR is draft and stacked on #101#103.

Notes

  • Archive values remain nested JSON values; no JSON payload is embedded in a JSON string.
  • Automatic backup exclusion is requested but documented as best effort; encrypted explicit backup is required for operators who opt in.
  • Record identifiers, payloads, source paths, and raw provider failures are never logged.

Expose a deterministic metadata-only provider registry through JSON and text output, covering every current Apple-data command family with maturity, source, permission, sensitivity, capability, and polling declarations. Document the v1 compatibility policy.\n\nCloses #89.
Add a private read-only SQLite snapshot engine with WAL and SHM companions, query-only enforcement, busy and process timeouts, structured failures, and deterministic cleanup. Migrate Messages and Safari history and document remaining providers.\n\nCloses #90.
Add explicit scan and wall-clock budgets, structured crawl reports, cancellable Drive enumeration, Finder-tag budget metadata, and redacted watch partial-success envelopes. Extend the live audit with explicit budgets and aggregate outcomes.\n\nCloses #92.
Define versioned archive sync, storage, status, migration, retention, and privacy contracts with idempotent upserts, explicit tombstones, bounded attempts, persisted freshness and counts, private atomic files, and fail-closed sensitivity gates.\n\nCloses #93.
@jmcte

jmcte commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

@athena-omt @pheidon @ares-omt @Hermes-omt — this high-risk archive-contract PR is stacked on #101#103 and intentionally draft until those foundations merge. Early review is requested on cursor advancement, idempotency, migration, retention, nested JSON handling, private atomic storage, backup policy, and fail-closed sensitivity boundaries.

@jmcte
jmcte marked this pull request as ready for review July 12, 2026 19:04
@jmcte
jmcte requested a review from a team as a code owner July 12, 2026 19:04
@jmcte
jmcte enabled auto-merge (squash) July 12, 2026 19:39

@Hermes-omt Hermes-omt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: sensitiveField(in:) only recursively inspects direct object elements of an array. A payload such as {"nested":[[{"body":"..."}]]} is accepted and persisted despite body being explicitly forbidden. I reproduced this through archive sync on the reviewed head: it exited 0 and wrote the synthetic body value into account.json.

Please make sensitive-field traversal recursive for every ArchiveValue shape (including arbitrarily nested arrays), and add a regression test covering an array nested inside an array containing a forbidden key. The existing direct nested-object test does not cover this case.

Hermes validation: exact-head checkout; swift test passed (130 tests). Live CI checks are successful; GitHub currently reports auto-merge disabled and mergeable_state=dirty.

@Hermes-omt Hermes-omt added the state:needs-repair PR needs repair before review can proceed. label Jul 17, 2026
@athena-omt athena-omt added status:needs-review PR is ready for Athena review. review:athena Athena review governance requested. state:waiting-checks Waiting for CI/check status to settle. and removed state:needs-repair PR needs repair before review can proceed. labels Jul 17, 2026

@Hermes-omt Hermes-omt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head 7f74b1639279f0be5ae517a67c6435e83a4eb56c. The prior nested-array sensitive-field bypass is fixed with recursive traversal across all ArchiveValue shapes and a targeted regression test. Exact-head macOS validation passed: swift test --disable-sandbox; PR Fast CI and CI Gate are successful. Branch is current with main, mergeable clean, auto-merge enabled, and no unresolved review threads.

@Hermes-omt Hermes-omt removed status:needs-review PR is ready for Athena review. review:athena Athena review governance requested. state:waiting-checks Waiting for CI/check status to settle. labels Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v2: Build resumable per-provider local archives

4 participants