Skip to content

reader: lazy index reads (Prelude + SparseReader) for WMTS-scale packs - #1

Merged
jacovdbergh merged 1 commit into
masterfrom
lazy-reader
Jul 30, 2026
Merged

reader: lazy index reads (Prelude + SparseReader) for WMTS-scale packs#1
jacovdbergh merged 1 commit into
masterfrom
lazy-reader

Conversation

@jacovdbergh

@jacovdbergh jacovdbergh commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What

A second reading discipline for indexes too large to fetch whole. A WMTS-style zoom level packed as one single-level planar tilepack runs to millions of tiles, so the offset index alone reaches tens of MB — more than a serving process wants to hold (or fetch) per open pack. It doesn't need to:

  • Prelude — parses header + descriptors only (~100 bytes; Truncated-staged exactly like FrontMatter::parse, needing at most descriptors_end). Then:
    • index_entry_span(ordinal) → the 16-byte file span holding a tile's start/end offsets (24 + 48g + 8i);
    • tile_range_from_entry(&[u8; 16], file_len) → the blob range, applying the per-entry half of the whole-index validation (backwards ranges, ranges into the front matter, and ranges past a known file length are BadIndex; equal offsets = absent tile, Ok(None)).
  • SparseReader<R: Read + Seek> — the I/O twin: prelude up front, then per tile one 16-byte index read + one blob read. O(1) memory regardless of index size.

No wire-format change, no new deps, wasm-clean. FrontMatter stays the right choice whenever the index fits (panorama viewers untouched).

Tests

  • The roundtrip property test (400 cases, arbitrary layouts/groups/absences) now also asserts, for every ordinal: Prelude entry arithmetic + validation ≡ FrontMatter::ordinal_range, and SparseReader ≡ the eager view. The lazy path can't drift from the eager one without the oracle catching it.
  • tests/lazy.rs: golden-fixture equivalence (sparse ≡ eager for all ordinals), staged Truncated needs, and adversarial entry rejection (backwards / front-matter overlap / past-EOF, plus the documented "unknown file length ⇒ oversize is the caller's to catch" case).

Docs

  • SPEC.md: informative paragraph in HTTP access pattern — the per-tile 16-byte index read and the validation a lazy client MUST do instead of the whole-index pass.
  • docs/integration.md: new Lazy index reads section for the WMTS-scale consumer.

All four gates pass: cargo test --workspace, clippy --all-targets -D warnings, fmt --check, wasm32 release build of -p tilepack.

A second reading discipline for indexes too large to fetch whole: a WMTS
zoom level packed as one single-level planar file runs to millions of
tiles, and the offset index alone reaches tens of MB — hostile to a
serving process that holds front matter per open pack.

Prelude parses header + descriptors only (~100 bytes, Truncated-staged
like FrontMatter) and locates any tile with one 16-byte read into the
index region: index_entry_span(ordinal) says which bytes to fetch,
tile_range_from_entry decodes them with the per-entry half of the
validation FrontMatter::parse does for the whole index (no backwards
ranges, nothing inside the front matter, nothing past a known file
length; equal offsets = absent tile). SparseReader is the Read+Seek
twin — three bounded reads per tile, O(1) memory — and doubles as the
executable spec: the roundtrip property test now holds the lazy path
equal to the eager index for every ordinal of every generated layout,
and tests/lazy.rs pins the golden fixture plus adversarial entries.

No wire-format change; FrontMatter remains the right choice whenever
the index comfortably fits (panorama viewers are untouched). SPEC.md
gains the informative access pattern; docs/integration.md the wiring
note.
@jacovdbergh
jacovdbergh merged commit e2620d7 into master Jul 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant