S1 RTC per-acquisition previews break on non-monotonic time axis — render by datetime, not index

[lhoupert]

Problem

Per-tile S1 RTC GeoZarr cubes store one acquisition per time slice (per orbit group, with overview levels r10m…r720m). The per-acquisition STAC items render their preview via titiler using sel=time={INTEGER index} — a positional index into the time axis.

This is fragile and not scalable:

• Cubes go non-monotonic. ingest_s1tiling_acquisition (src/eopf_geozarr/conversion/s1_ingest.py:634-680) always appends a new acquisition at the physical end. A cross-run append of an earlier-dated scene therefore pushes the time axis out of chronological order.
• Once non-monotonic, titiler fails to render the out-of-order slice (HTTP 400), so the per-acquisition item has no preview.
• Keeping positional indices correct would require reordering the cube data and re-registering every item on every append — O(N) data rewrite + O(N) item churn per new acquisition.

Evidence

• Tile 31TEH, descending group: time axis is [idx0 = 2026-06-08T05:43:23, idx1 = 2026-06-07T05:52:23] — decreasing.
• Item s1-rtc-31TEH-20260607t055223 (idx1) renders HTTP 400 "Could not find any valid variables in '/descending:vv'", while idx0 renders fine.
• The idx1 slice is not empty (100% finite, real backscatter) — so it is purely an ordering/selection problem, not missing data.
• Root metadata gap: …/descending/r10m/time/zarr.json has empty attributes — the time array is a bare int64 with no CF datetime metadata, so readers can't treat it as a datetime index.

Root cause

1. time arrays carry no CF datetime encoding (units/calendar), so xarray/titiler can't expose them as a datetime index — the time dimension falls back to an integer index (which is why sel=time=0 works but sel=time=<datetime> 500s).
2. ingest_s1tiling_acquisition appends at the physical end with no chronological guarantee.

Proposed resolution — render by datetime, not index

The deployed titiler v0.5.0 already does label-based selection: da.sel({dim: value}, method=method) (titiler/eopf/reader.py:589,597) with a method::value DSL. The only gap is the cube metadata.

Encode time as a CF datetime coordinate (units = "nanoseconds since 1970-01-01", calendar = "proleptic_gregorian", standard_name = "time"; values unchanged). Then each per-acquisition item can render via sel=time={its own acquisition datetime}. An exact-label .sel(time=<datetime>) works even on a non-monotonic axis (only nearest/slice need monotonicity), so:

• No cube data reorder, ever.
• Append a new acquisition → register only that one item with its datetime; existing items untouched.
• Order-immune and O(1) per append.

Tracking checklist

• Spike validated (datetime sel renders the 31TEH 06-07 slice, correct, on the non-monotonic axis)
• data-model: CF datetime encoding on time (create/append/conditions paths) + tests
• data-pipeline: per-acquisition render links use sel=time={datetime} + incremental registration
• Existing-cube metadata backfill (CF time attrs) + one-time item re-registration
• Release (eopf-geozarr pin bump + image tag) and deploy
• Fleet migrated + verified

Fallback if the spike fails: sort the cube on append (reorder + re-register) — kept as a backup approach only.

[lhoupert]

Spike validated — refinement: CF-encode time at every multiscale level

Confirmed the datetime-rendering approach locally against titiler's exact opener (xarray.open_datatree(decode_times=True) + label .sel):

• ✅ Adding CF units/calendar makes time decode to datetime64; exact .sel(time=<datetime>) returns the correct slice even on a non-monotonic axis (only nearest/slice need monotonicity).
• ✅ Without the attrs it fails (reproduces the current bug).

Important structural refinement: titiler picks a multiscale level by zoom, and previews use a coarse level. Today time exists only at r10m — coarse levels (r20m…r720m) have a time dimension but no coordinate, so datetime .sel can't resolve there.

I tested putting time only at the orbit-group level and relying on DataTree inheritance — but that fails to open when the existing r10m/time stays bare int64 (AlignmentError: int64 vs datetime64 on the shared time dim).

Resolution: CF-encode time consistently as a coordinate at every multiscale level (identical values; tiny [N] int64 arrays). absolute_orbit/relative_orbit/platform can stay at r10m. This opens cleanly and lets datetime .sel resolve at whatever level titiler renders.

[lhoupert]

Progress

• ✅ Spike validated (CF time → datetime64; exact .sel on a non-monotonic axis; reproduced via titiler's exact open_datatree(decode_times=True) + .sel).
• ✅ data-model CF-time fix — merged into feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering #180 (commit on feat/s1-rtc-stac-builder).
• ✅ data-pipeline render-by-datetime + incremental registration — fix(s1-rtc): render per-acquisition previews by datetime + incremental registration data-pipeline#271 (sel=time={datetime}, percent-encoded; default registers only new acquisitions; --reregister-all for migration).
• ⏳ Migrate existing cubes + re-register — the in-place metadata backfill proved fragile (zarr v3 recursively-nested consolidated metadata + restricted S3 creds without LIST). Pivoting to re-ingesting tiles through the fixed pipeline, which natively produces correct CF cubes (and serves as the live end-to-end validation). To be done after the release + deploy.
• ⏳ Release + deploy.

Deploy ordering (important): the data-pipeline sel=time={datetime} change must go live only after cubes are CF-encoded — datetime sel against a bare-int64 time returns 500. So: deploy the CF-time ingest image → re-ingest the fleet (or new cubes only) → then the datetime-sel register.

[lhoupert]

✅ Validated live (2026-06-17)

Built the combined image sha-d746b39 (CF ingest + datetime register). Ingested real GeoTIFFs into a never-cached store and rendered by datetime:

• Fresh single-slice CF cube → 10/10 stable HTTP 200 via sel=time={datetime}.
• titiler (0.9.1, xarray 2026.4.0) CF-decodes time correctly; exact datetime .sel returns the right slice on a non-monotonic axis.

So the code (this issue's fix + data-pipeline#271) is validated end-to-end.

⚠️ The real blocker is titiler caching, not the code

titiler caches the opened datatree keyed by store path with a 24h TTL across 40 replicas. Appending a slice to a cube staleifies that cache, so the newest acquisition's preview 500s/400s on stale-cache replicas until TTL. This hits every cron append and re-ingest (and is pre-existing — index sel has it too). No reliable invalidation is currently available (/admin/cache/invalidate 504s on the Redis SCAN).

A reliable titiler cache-invalidation-on-cube-change is a hard prerequisite before un-suspending the cron. Separately, a titiler robustness ticket: its reader does int(sel_value) and 500s when a stale datatree has int64 time, instead of erroring gracefully.

[lhoupert]

S1 RTC datetime-rendering — validated; provisional tag cut; un-suspend gated on titiler.

• Code complete: CF-encode time at every multiscale level (s1_ingest.py, fix(s1-rtc): CF-encode time at every level for datetime-based rendering #193 → open feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering #180 branch, 16c5f14) + data-pipeline register_per_acquisition.py emitting sel=time={datetime} + incremental registration (#271, feat--s1_grd_phase6@32a173f).
• Live-validated (image sha-d746b39): a fresh CF cube (valid31teh, never-cached store) rendered 10/10 stable HTTP 200 by exact datetime; titiler CF-decodes time and .sel returns the right slice on a non-monotonic axis. 31TEH's earlier flapping was a poisoned titiler cache, not the code.
• Cut provisional tag v0.6.0-s1rtc on feat--s1_grd_phase6@32a173f. Provisional because it pins eopf-geozarr 16c5f14, a squash commit on the unmerged feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering #180 branch (orphaned if feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering #180 squash-merges). Final v0.6.x-s1rtc is gated on feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering #180 landing (re-pin to post-merge commit / release first).
• Un-suspend / deploy is gated on titiler, not our code: datatree cache staleifies on append (24h TTL, ~40 replicas) → newest slice 500s/400s until TTL. Filed: Datatree cache stale-on-append + int() crash on datetime sel break newest-slice renders titiler-eopf#118 (stale-cache + int()-on-datetime-sel crash).