db_format: shrink PFDB on-disk size without touching query latency - #82
Closed
falconindy wants to merge 4 commits into
Closed
db_format: shrink PFDB on-disk size without touching query latency#82falconindy wants to merge 4 commits into
falconindy wants to merge 4 commits into
Conversation
- no custom gtest_main - unittests option at build time - declare gtest/gmock dependencies as disablers
This keeps pkgfiled's resource usage minimal at idle. Separately, we might consider serializing repo repacking to limit peak usage (or using disk to store intermediate state).
Replace the blocking curl_multi_wait/curl_multi_perform loop in Updater::Update() with curl's multi socket-action interface, bridged onto an sd_event the caller provides. Update() now registers work on that event and returns immediately, invoking a completion callback once every repo has resolved instead of blocking until done. Repack completion (previously joined via WaitForRepacking() at the end of Update()) is now signalled back to the loop through an eventfd source, so a repack finishing doesn't require polling. This is prep for reusing Updater from pkgfiled, which already runs its own sd_event loop and can't cede control to a second, competing one. The pkgfile CLI (the only caller so far) creates its own short-lived sd_event and pumps it itself; output and exit codes are unchanged. DownloadJob now owns its Repo by value instead of holding a reference into the caller's AlpmConfig, and jobs live in a std::list (stable addresses, needed since curl and async repack workers hold raw DownloadJob* for the job's lifetime) rather than a vector local to Update(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three latency-neutral encoding changes to the repo database format,
bumping it to v2:
- String table stores one u32 byte-pool offset per string instead of
an {offset,length} pair; length is implicit from the next entry's
offset, since the byte pool is built by pure concatenation.
- Path trie nodes pack into 6 bytes (two 24-bit fields) instead of 8,
since a PathId/StringId comfortably fits 24 bits for any repo
pkgfile realistically indexes. Unpacked via two bounded memcpy
loads rather than byte-by-byte shifts, since PathNodeAt() is on the
hot path for every full-repo glob/regex scan.
- Postings pool is delta+varint encoded rather than a flat array,
since it's only ever read as one small bounded slice (located by
binary search on the fixed-width basename index, never itself
binary-searched).
All three keep zero-copy mmap and O(1) random access where it's
actually used; only the postings pool -- read as tiny per-basename
slices on the exact-match lookup path, never touched by glob/regex --
takes on a decode step, via a caller-reused scratch buffer.
Measured on Arch's `extra` repo (live snapshot): 248MB -> 202MB
(~18.8% smaller). Query results verified byte-identical against the
v1 format across exact/full-path/verbose/list/glob/regex/case-
insensitive queries, including a 7339-result directory glob. Latency
verified at parity for both an indexed exact search and a full-repo
glob scan (the path the packed path table's hot PathNodeAt() call
could most plausibly have regressed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See commit message for details. Draft PR opened by automated background job.