Skip to content

perf(sync-client): continuously drain pushes with pull priority - #36

Merged
hhhjin merged 2 commits into
mainfrom
codex/continuous-push-pull-priority
Sep 5, 2026
Merged

perf(sync-client): continuously drain pushes with pull priority#36
hhhjin merged 2 commits into
mainfrom
codex/continuous-push-pull-priority

Conversation

@hhhjin

@hhhjin hhhjin commented Sep 5, 2026

Copy link
Copy Markdown
Member

Summary

A slow upload could hold the next selection of queued files behind the current 100-entry batch, and incoming pull requests had to wait for a push invocation of up to 1,000 mutations. Continuously replenish available push slots and yield to pending pulls after started work settles.

  • Remove the invocation limit and fixed selection barrier while retaining 12 concurrent preparations/uploads, at most 100 outstanding entries, and sequential commits of at most 100 mutations.
  • Keep each entry reserved through local acceptance. Exclude reserved entries before applying queue limits in Dexie, SQLite, and the in-memory store.
  • Stop supplying work when pull is pending or sync stops; join started uploads before returning control. Resume from the current queue after pull.
  • Route repeated requeues without acceptance through the existing retry backoff instead of spinning indefinitely.
  • Update the Obsidian release notes. Encryption and content verification remain unchanged.

Validation

  • Sync client: 320 tests passed.
  • Dexie store: 12 tests passed; SQLite store: 13 tests passed.
  • Sync client, Obsidian plugin, and CLI typechecks passed.
  • All six benchmarks passed, including post-run decryption and content-hash verification.
  • Tests cover replenishment beyond 100 entries with a blocked upload, pull/stop handoff, pull arriving during a store read, outstanding-work limits, and repeated requeue termination. Existing same-entry edit/rename and failure/retry tests also pass.

Benchmark comparison

Local five-sample means against the preceding ready-batch implementation, using the unchanged mixed fixture and simulated slow-attachment transport:

Metric Before After
Total duration 1,624 ms 1,111 ms
Note completion p95 1,598 ms 1,084 ms
Notes applied before attachment completion 99 173.8
Commit requests 10.4 8.2

Upload attempts and encrypted bytes are unchanged. The 1 GiB push mean remains approximately 1.16 s. These are local simulated-transport measurements; process-wide RSS samples do not establish a memory improvement. Pull priority is cooperative: already-started uploads are joined, not cancelled.

@hhhjin
hhhjin merged commit 39be823 into main Sep 5, 2026
4 checks passed
@hhhjin
hhhjin deleted the codex/continuous-push-pull-priority branch September 5, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant