Search cutoff is wrong for snapshots that only exist on a non-main branch

[YanivZalach]

Concept

search_cutoff needs, for the selected start/end snapshot, the window of metadata.json files that reference it. Today it gets that window by querying metadata_log_entries filtered on latest_snapshot_id.

latest_snapshot_id is main-branch resolved: it is whatever the current-snapshot pointer was at the moment each metadata.json was written. A snapshot that only ever lived on a branch (or a tag) is never that value.

Problem

For a branch-only snapshot, both lookups in backend/search_cutoff/find_search_cutoff.py return NULL and the code silently falls back to the snapshot's own committed_at:

• _get_start_cutoffs line 97
• _get_end_cutoffs line 141

Consequences:

• Start ends up roughly right, off by at most one metadata file.
• End is genuinely wrong. Every metadata.json written after the branch commit that still carries the ref falls outside the window.

User-visible symptom: metadata file nodes are missing from the graph when the selected range ends on a snapshot that lives on a non-main branch.

Root cause is an Iceberg metadata-table gap: no metadata table exposes branch refs per metadata file. refs gives only the current refs, metadata_log_entries gives only the main pointer. The branch-to-snapshot mapping over time exists only inside the metadata.json files themselves.

How to solve

Keep the current fast path. Add a fallback that runs only when the metadata_log_entries lookup comes back empty, which is already the exact signal for "this snapshot was never on main". No branch-detection heuristic is needed.

The fallback reads the metadata files directly:

1. Take candidate file paths from metadata_log_entries, starting one entry before the snapshot's committed_at (absorbs the skew between snapshot commit time and metadata write time) and walking forward.
2. Read them in a single spark.read.schema(...).json(paths) with a minimal explicit schema: refs as map<string, struct<snapshot-id: bigint>> plus last-updated-ms. One pass, no per-file round trip.
3. Keep rows whose refs values contain the selected snapshot id.
4. MIN(last-updated-ms) is the start metadata cutoff, MAX(...) is the end. If the newest metadata file still matches, the end stays open (arrow.Arrow.max).

Doing it as one min/max instead of a sequential walk also handles a ref that disappears and reappears, without extra code.

Constraints

• Bound the candidate set with a new MAX_* env var defined in backend/env.py, so a table with thousands of metadata files cannot turn this into a full scan.