reader: lazy index reads (Prelude + SparseReader) for WMTS-scale packs - #1
Merged
Conversation
A second reading discipline for indexes too large to fetch whole: a WMTS zoom level packed as one single-level planar file runs to millions of tiles, and the offset index alone reaches tens of MB — hostile to a serving process that holds front matter per open pack. Prelude parses header + descriptors only (~100 bytes, Truncated-staged like FrontMatter) and locates any tile with one 16-byte read into the index region: index_entry_span(ordinal) says which bytes to fetch, tile_range_from_entry decodes them with the per-entry half of the validation FrontMatter::parse does for the whole index (no backwards ranges, nothing inside the front matter, nothing past a known file length; equal offsets = absent tile). SparseReader is the Read+Seek twin — three bounded reads per tile, O(1) memory — and doubles as the executable spec: the roundtrip property test now holds the lazy path equal to the eager index for every ordinal of every generated layout, and tests/lazy.rs pins the golden fixture plus adversarial entries. No wire-format change; FrontMatter remains the right choice whenever the index comfortably fits (panorama viewers are untouched). SPEC.md gains the informative access pattern; docs/integration.md the wiring note.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A second reading discipline for indexes too large to fetch whole. A WMTS-style zoom level packed as one single-level planar tilepack runs to millions of tiles, so the offset index alone reaches tens of MB — more than a serving process wants to hold (or fetch) per open pack. It doesn't need to:
Prelude— parses header + descriptors only (~100 bytes;Truncated-staged exactly likeFrontMatter::parse, needing at mostdescriptors_end). Then:index_entry_span(ordinal)→ the 16-byte file span holding a tile's start/end offsets (24 + 48g + 8i);tile_range_from_entry(&[u8; 16], file_len)→ the blob range, applying the per-entry half of the whole-index validation (backwards ranges, ranges into the front matter, and ranges past a known file length areBadIndex; equal offsets = absent tile,Ok(None)).SparseReader<R: Read + Seek>— the I/O twin: prelude up front, then per tile one 16-byte index read + one blob read. O(1) memory regardless of index size.No wire-format change, no new deps, wasm-clean.
FrontMatterstays the right choice whenever the index fits (panorama viewers untouched).Tests
Preludeentry arithmetic + validation ≡FrontMatter::ordinal_range, andSparseReader≡ the eager view. The lazy path can't drift from the eager one without the oracle catching it.tests/lazy.rs: golden-fixture equivalence (sparse ≡ eager for all ordinals), stagedTruncatedneeds, and adversarial entry rejection (backwards / front-matter overlap / past-EOF, plus the documented "unknown file length ⇒ oversize is the caller's to catch" case).Docs
All four gates pass:
cargo test --workspace, clippy--all-targets -D warnings,fmt --check, wasm32 release build of-p tilepack.