Bound recursive crawls and preserve partial results

[jmcte]

Summary

• add explicit scan and wall-clock budgets plus icloud-cli.crawl.v1 partial/timeout reports
• bound Drive and Finder-tag recursion behind a cancellable worker deadline
• make watch --once continue after partial/timeout providers with redacted structured failures
• extend the live audit with explicit budgets and aggregate pass/empty/fail/timeout counts

Governing Issue

Closes #92

Validation

• Relevant local checks passed
• Required PR checks are expected to satisfy CI Gate
• Skipped checks are explained below

swift test --disable-sandbox — 112 tests passed

bash scripts/ci/run-fast-checks.sh — passed; source line coverage 88.53%, mutation checks killed expected mutations, debug and release builds passed

python3 scripts/live-command-audit.py --binary .build/debug/icloud-cli --timeout 12 — completed with payload output suppressed. The five formerly hanging recursive commands now exit normally with code 0 and structured state=timeout reports at the internal deadline; watch once continues and completes.

Bootstrap Governance

• Changes are scoped to the linked issue
• Contributor or PR guidance changes are reflected in CONTRIBUTING.md, .github/PULL_REQUEST_TEMPLATE.md, and docs/bootstrap/onboarding.md when applicable
• PR author enabled auto-merge where GitHub allows it, or GitHub plan-limit evidence/unavailable reason is recorded and the fallback merge-readiness policy applies
• No real secrets, runtime auth, or machine-local env files are committed

This branch is stacked on provider-manifest PR #101. The PR remains draft until #101 merges and the diff collapses to the issue #92 commit; auto-merge will then be armed.

Flow Contract

• Owner lane: Hermes macOS/native crawl reliability
• Repair owner: jmcte
• Autonomy class: review-gated implementation from assigned issue
• Risk class: medium; public crawl envelope and bounded worker lifecycle

Flow Merge Readiness

• Every blocker has a next actor and next action
• No active blocking requested changes remain
• Non-author approval is present when required
• PR author enabled auto-merge where GitHub allows it, or recorded why it is unavailable/unsafe

Next actor: #101 reviewers, then architecture/macOS review on this PR.

Merge Automation

• PR author enabled auto-merge with gh pr merge --auto --squash, or the reason it is unavailable/unsafe is noted below

Auto-merge is intentionally deferred while the PR is draft and stacked on #101.

Notes

• Scan limits, result limits, and known matching totals remain separate.
• Timeout and watch failure output contains no crawled payloads or source paths.

[jmcte]

@athena-omt @pheidon @ares-omt @Hermes-omt — this crawl-reliability PR is stacked on #101 and intentionally draft until that dependency merges. Early review is welcome on the public crawl envelope, cancellable worker lifetime, scan/result/total semantics, redacted watch failures, and live-audit classification.

[Hermes-omt]

Hermes Review — Bound recursive crawls and preserve partial results

Reviewed head: 31021a6 (matches expected)
CI: PR Checks ✓ · CI Gate ✓ · 0 attention

What was inspected

• Full diff across 18 files (±602/−343 lines)
• Live macOS build (swift build) — clean
• All 109 tests pass on Darwin/arm64
• New CrawlBudget / CrawlReport<P> types — sensible defaults, lock-safe concurrency via DriveWalkBox
• ICloudDriveInventoryReader walk path: threaded enumeration with wall-clock deadline + scan-limit guard; deterministic partial/timeout state propagation
• CacheWatchStore.refresh() now accepts a budget closure and emits redacted CacheFailure on budget overflow (no credential/data leakage)
• CLI flag additions (--scan-limit, --timeout-ms) consistent across all affected commands in usage text
• CrawlBudgetTests.swift: scan-budget partial state, timeout determinism, report round-trip, replacingData generic — all exercising the boundary conditions
• Test updates (CommandRunnerHarness): crawl result JSON decode via CrawlReport<T>, updated expected output messages
• Command-audit script: payload_shape() now handles crawl envelopes (schemaVersion == "icloud-cli.crawl.v1") with proper nesting detection; TIMEOUT classification added
• Removal of ExternalControlPlaneManifest + fixture: clean, no dangling references in sources/tests/docs

Observations (non-blocking)

• The removal of the external-manifest command is architecturally sound (capability manifest already covers provider introspection); consider a one-line changelog note referencing PR 103 when publishing.
• DriveWalkBox uses NSLock inside an @unchecked Sendable final class — this is correct but worth noting for future reviewers; the lock guards both state.termination checks and files.append atomically.

Status

No blockers. The PR compiles, all tests pass on macOS arm64, crawl budget enforcement is well-scoped with deterministic partial/timeout semantics, and redacted failure handling preserves privacy boundaries.

[Hermes-omt]

Request changes: walkFiles returns after the semaphore timeout, but it only marks box stopped; the worker checks that flag only before/after synchronous enumerator.nextObject() (DriveInventory.swift:225-260). If that call blocks—the iCloud traversal case this timeout is meant to contain—the worker survives past the reported timeout. Repeated watch refreshes can therefore accumulate stuck workers. Please use a design that actually bounds worker lifetime/cancels the blocking operation, and add a regression test that holds enumeration past the timeout and verifies no worker remains. swift test passes (115 tests), but the current timeout test advances an injected clock and cannot detect this leak.