End-to-end sponsor example: catalog → shardmap → dual aggregation → cached reads

[espg]

Closes #168. (espg-directed, in-session.)

The sponsor-facing end-to-end example: two notebooks plus the pipeline behind them, on real public data — ICESat-2 ATL03 v007 over the NEON SERC AOP box (tests/data/benchmark/AOP_NEON.geojson, temporal pin 2018-10-13 .. 2025-06-01, matching tests/data/benchmark/README.md).

What this adds

• notebooks/sponsor_end_to_end.ipynb — the full pipeline as fleet dispatches: CMR → STAC-geoparquet catalog (build-once, add 88 south stress test shard + benchmark #148/88S stress shard: pins, 900s timeout, chunk-boundary extraction, a-priori read plan, streaming tdigest (issue #148) #152 precedent), shardmap with the strict-AOI mask payload (Optional strict-AOI polygon clip for shardmap output (sub-shard clipping) #101) + viewer, then two production Lambda dispatches from the one shardmap (Reuse shardmaps across configs with different aggregators (relax signature guard to spatial-only) #89): located t-digest (High-resolution morton location channel for ragged/CSR fields #87 / PR High-resolution morton location channel for ragged/CSR fields #165) with inline+write_back populating the granule-keyed manifest cache under zagg-index/ATL03/007/ (Virtual index entry point: pluggable chunk-index backends for the read path #160 / PR Virtual chunk-index backends for the read path (issue #160) #163), and gain/bias 128-bin waveforms consuming that cache through the sidecar backend (h5coro-hidefix 0.2.0, zagg.index_backends entry point). Read-back (§5) uses unsigned S3 reads of the published stores: density + strict-AOI maps, digest-quantile maps, an exact digest-vs-waveform cross-check, the morton location channel. Committed unexecuted by design — the live run is espg's architecture test of the 0.14.0 deployment (division of labor below).
• notebooks/sponsor_read_only.ipynb — the Binder-badged, zero-credential companion (anonymous store reads + a sidecar-manifest fetch). Committed unexecuted with a status note: it can only run once the fleet run populates the bucket.
• src/zagg/configs/atl03_tdigest_serc.yaml / atl03_gain_bias_serc.yaml — the two run configs (shared o10 spatial signature → one shardmap; output.aoi_mask: true; data_source.index per Virtual index entry point: pluggable chunk-index backends for the read path #160; canonical sliderule-public-cors store URLs), guarded by tests/test_config.py::TestSercExampleConfigs.
• notebooks/data/ — committed build-once snapshots: cat_serc_atl03_007.parquet (55 KB, 59 granules), sm_serc_healpix_o10.json (232 KB, 9 shards / 326 pairs, aoi_mask payload, NSIDC S3 hrefs — fleet-ready as committed).

Division of labor: what is already proven vs what the live run tests

Every code path in notebook A was pre-validated in-process on this exact shardmap before commit (same worker code via backend="local", h5coro FileDriver over a complete 61-granule local cache):

• both aggregations ran 9/9 shards, 9,564,649 photons each, 0 errors (t-digest 638 s wall, gain/bias-via-sidecar 675 s wall at 4 local workers; per-shard worker times ~50–250 s);
• sidecar consumed the 59 write-back manifests and its output was byte-identical to an inline rerun (waveform_counts, gain_h spot-check) with total_obs equality asserted across the pair;
• all §5 read-back analyses rendered from those stores (digest-vs-waveform cross-check exact: in-window totals matched to the photon).

The reference stores are retained at ~/.cache/zagg/serc_example/ for post-run comparison. So espg's live run exercises the Lambda/deployment layer specifically — dispatch, layer contents, worker IAM, store/manifest writes from the execution role — not untested notebook code.

How espg runs the live test (pre-run checklist)

1. aws sso login --profile nasa (+ export AWS_PROFILE=nasa), and EDL ~/.netrc present.
2. IAM check: the workers write manifests to s3://sliderule-public-cors/zagg-index/ATL03/007/ — the Virtual index entry point: pluggable chunk-index backends for the read path #160/Virtual chunk-index backends for the read path (issue #160) #163 execution-role PutObject grant on zagg-index/* (commit 87b941e granted it in-template) must be in the deployed stack. If run 1 errors with AccessDenied on zagg-index/*, the stack needs the update — pause there.
3. Deploy state: process-shard was last modified 2026-07-06 04:02 UTC with layer process-shard-deps:15 (the 0.14.0 + h5coro-hidefix 0.2.0 rollout) — assumed current.
4. Run notebook A top-to-bottom (jupyter nbconvert --to notebook --execute --inplace notebooks/sponsor_end_to_end.ipynb, or Run-All). §4's final cell prints per-worker min/median/max durations from the async result objects — the architecture-test numbers.
5. Memory caveat: issue Order-10 single-shard processing OOMs at 2 GB (gain_bias) — 2 GB is marginal for order-10 #117 found o10 gain_bias marginal at 2 GB on the densest NEON benchmark shard; the streaming/malloc work (Warm-container OOM: Arrow memory pool residual not reclaimed by malloc_trim (#140) #143/add 88 south stress test shard + benchmark #148) has landed since, but if a shard OOMs, that is Order-10 single-shard processing OOMs at 2 GB (gain_bias) — 2 GB is marginal for order-10 #117 territory, not notebook code.
6. Afterwards: notebook B can be executed (no credentials) and both re-committed rendered — happy to do that in a follow-up session once the bucket is populated.

Statically verified on the dispatch cells (could not be invoked from this session)

agg() kwargs against the live signature; both configs load_config+validate_config clean; the in-notebook sidecar index override passes validate_index_config with the plugin installed and sidecar is entry-point-discoverable; store/index URLs match the canonical prefixes; the committed shardmap carries NSIDC S3 hrefs (fleet-ready), 9 shards, aoi_mask payload, and a spatial signature equal to both configs' grids. Not verifiable statically: the worker-side IAM grant (checklist item 2), the async result-object polling against the deployed worker, and live NSIDC credential exchange.

Phases

• 1. Scaffolding + configs + catalog & shardmap sections (committed snapshots; review findings folded in de2c5f5)
• 2. Dual aggregation pipeline built and pre-validated in-process, end to end (9ab82f2 carried the executed local run as the working state)
• 3. Final deliverable per direction: fleet-dispatch notebook A + Binder companion B, committed unexecuted with outputs cleared (724aa47); adversarial-review findings folded (manifest stem key, version-claim, Binder badge scope)
• 4. (espg) Live fleet run of notebook A → then execute B anonymously and commit both rendered

How it was tested

• CI green (test 3.12/3.13, ruff, builds) including TestSercExampleConfigs; local pre-commit (repo-pinned ruff/mypy/codespell) green on the changed files; pytest tests/test_config.py tests/test_index.py 267 passed on the merged branch (with h5coro-hidefix 0.2.0 installed, exercising the entry-point discovery path).
• Both committed notebooks: nbformat-valid, every code cell ast.parses, zero outputs, stable cell ids. Notebook A's earlier fully-executed state (local backend) is preserved in branch history at 9ab82f2 for reference.

Questions for review

• waveform_counts is emitted uncompressed (codecs: [bytes]): the gain/bias store will be ~1.1 GB in the bucket (568 × 2 MB raw chunks; measured on the reference run). Fine to host, but a compressed template codec would cut it ~10× — follow-up issue?

[espg]

🤖 from Claude (review)

This comment claims drift tests load the committed snapshot, but no such test exists in this PR — nothing under tests/ references cat_serc_atl03_007.parquet, sm_serc_healpix_o10.json, or the two new configs (git grep serc -- tests/ is empty). More broadly, the two configs now shipped in the package (src/zagg/configs/atl03_{tdigest,gain_bias}_serc.yaml, loaded via importlib.resources in cell 6) have no committed validation coverage — test_config.py only checks named configs (atl06 / merra2_storm / atl03_waveform_chunk), not a glob over zagg.configs. The PR's "both configs load_config + validate_index_config clean" was verified manually only. Per CLAUDE.md §4, either add a small test asserting both configs load + validate_index_config clean and that their spatial_signature() match (the #89 reuse contract the notebook asserts), or soften this comment's present-tense "drift tests" claim.

[espg]

🤖 from Claude

Folded in de2c5f5: TestSercExampleConfigs (tests/test_config.py) now covers both shipped configs — load/validate, index-block validation, the issue #89 shared-spatial-signature contract, and the aoi_mask + location-channel declarations. The "drift tests" wording is softened to "later runs" in the next notebook regeneration (phase 2 re-executes the notebook top-to-bottom, so prose and outputs move together).

[espg]

🤖 from Claude (review)

Section-numbering is internally inconsistent. The overview list at the top of this cell numbers the pipeline 1. Catalog, 2. Shardmap, 3. Two aggregations, 4. Read-back — but this sentence (and cell 5's "which is exactly what section 5 does") sends the reader to section 5 for the aggregation runs, and the config walkthrough is headed ## 3. The two run configs. So aggregations are overview-step 3 yet "section 5", and no ## 4/## 5 headers exist in this phase-1 notebook, so "section 5" currently points nowhere. Either renumber the overview to match the eventual section headers or drop the concrete "section 5" pointers until those sections land (phases 2-3).

[espg]

🤖 from Claude

Agreed — the intro list is being renumbered to match the actual section headers (catalog 1, shardmap 2, configs 3, dispatch 4, execute 5, read-back 6) in the phase-2/3 regeneration, which adds the missing sections 4–7.

[espg]

🤖 from Claude

Emergency stop (machine shutdown) — state preserved. Exactly where things stand:

Done / committed (branch claude/168-sponsor-example, head 78df183):

• Phase 1 complete and executed: configs + catalog/shardmap sections with committed outputs (1de3ebd), review findings folded (de2c5f5, both inline threads answered).
• Phase 2 source complete but unexecuted (fbe58ec): sponsor_end_to_end.ipynb now carries §4 (documented Lambda path), §5 (executed-local dual runs: t-digest inline+write_back, then gain/bias through the real sidecar backend consuming the manifest cache), §6 read-back (density/AOI, digest quantiles, digest-vs-waveform cross-check, location channel), §7, the 0.14.0 version premise (pins zagg>=0.14, provisional cell noting execution on 0.13.0 = the same Virtual chunk-index backends for the read path (issue #160) #163+High-resolution morton location channel for ragged/CSR fields #165 feature set), and both review fixes (intro renumbering, drift-test wording). The top-to-bottom nbconvert --execute was killed mid-run by the shutdown — that execution is the only thing lost; every code path in it was previously validated by direct runs (see PR body).
• Phase 3 source committed unexecuted (78df183): sponsor_read_only.ipynb (anonymous obstore reads, binder badge). Cannot run until the bucket is populated.

Local state that survives reboot (home dir, not committed by design):

• ~/.cache/zagg/serc_example/ — full 9-shard t-digest store (120 MB, complete), 59 write-back manifests under zagg-index/ATL03/007/, sm_serc_healpix_o10_local.json; gain_bias.zarr is partial (killed mid-run) — delete and re-run.
• ~/ignore/zagg_neon_atl03_test_shard/granules — the 61-granule cache (covers all 59 catalog granules).
• ~/software/h5coro-hidefix/dist/h5coro_hidefix-0.2.0-cp39-abi3-macosx_11_0_arm64.whl — built from hidefix main @ dabef7a.

Lost on reboot (scratch tmp): the venv (rebuild: uv venv && uv pip install -e '.[test,catalog,analysis,viz]' jupyter nbconvert matplotlib <hidefix wheel> — zagg env is plain origin/main @ dca9a91, no merge needed since #163/#165 are merged) and the single-shard smoke stores (re-derivable).

Bucket: write access verified with the nasa SSO profile (canary put/read/delete under zagg-examples/); no product data published yet. SSO token was fresh at stop time.

Precise next actions for a resumer:

1. Rebuild the venv as above; rm -rf ~/.cache/zagg/serc_example/gain_bias.zarr.
2. cd notebooks && jupyter nbconvert --to notebook --execute --inplace sponsor_end_to_end.ipynb (~25–35 min; §5 re-runs both aggregations with overwrite=True) → commit as "phase 2 of issue End-to-end sponsor example: catalog → shardmap → dual aggregation → cached reads #168" (it may also churn notebooks/data/* — commit together).
3. Publish: aws s3 sync ~/.cache/zagg/serc_example/tdigest.zarr s3://sliderule-public-cors/zagg-examples/serc/tdigest.zarr --profile nasa (likewise gain_bias.zarr, and ~/.cache/zagg/serc_example/zagg-index → s3://sliderule-public-cors/zagg-index). Note the gain/bias store is ~1.1 GB (uncompressed waveform_counts — see PR body question).
4. Execute sponsor_read_only.ipynb the same way (reads the bucket anonymously) → commit as "phase 3 of issue End-to-end sponsor example: catalog → shardmap → dual aggregation → cached reads #168".
5. Phase 4: adversarial self-review, fold findings, verify Binder for the read-only notebook, flip ready-for-review when green.

[espg]

🤖 from Claude

Paused before the production dispatch — two items need espg:

Direction received mid-run: the notebook's executed aggregations should be real Lambda dispatches against the deployed 0.14.0 fleet (writing directly to zagg-examples/serc/ + zagg-index/ATL03/007/), with the local runs kept only as a correctness reference. Preparation for that is underway (notebook A's §5 is being reworked so the executed cells are the dispatches; the local stores are retained for the fleet-vs-local comparison).

Blocked on:

1. AWS SSO — the nasa profile token expired again (aws s3 sync failed with Token has expired and refresh failed). Needs aws sso login --profile nasa.
2. Production-dispatch authorization — this session's permission layer denied agg(backend="lambda") against the deployed fleet from auto mode ("run outside auto mode for user review"); a coordinator relay can't grant that. Needs espg's direct go-ahead in-session (or espg running notebook A's §4 cells himself — they are exactly the documented dispatch).

Also noting for the dispatch attempt when it happens: if worker writes to zagg-index/* come back AccessDenied, the #163 execution-role PutObject grant may not be in the deployed stack yet — per direction, that's a pause-and-report, not a workaround.

Nothing has been published to the bucket. Branch state: phase 2 committed at 9ab82f2 (executed local run); self-review pass in flight.

[espg]

🤖 from Claude (review)

Manifest fetch will 404 when this notebook runs against the published bucket. shardmap.granules[0][0]["id"] is the full granule filename with the .h5 suffix — the committed sm_serc_healpix_o10.json stores "id": "ATL03_20190105163308_01260202_007_01.h5" — but write-back names each manifest by the granule-URL stem: src/zagg/index/inline.py:424 does PurePosixPath(urlsplit(granule_url).path).stem, so the object is ATL03_..._007_01.parquet (no .h5; confirmed by the end-to-end notebook's cell-19 output, which lists ATL03_20190105163308_01260202_007_01.parquet). As written, f"{granule_id}.parquet" requests ATL03_..._007_01.h5.parquet, which does not exist, so this cell errors. Strip the suffix, e.g. granule_id = Path(shardmap.granules[0][0]["id"]).stem (or .removesuffix(".h5")).

[espg]

🤖 from Claude

Folded in 724aa47: the fetch key is now shardmap.granules[0][0]["id"].removesuffix(".h5") with a comment documenting the stem convention — matching inline.py write-back naming.

[espg]

🤖 from Claude (review)

notebooks/sponsor_end_to_end.ipynb, intro cell (the versions note under "Requirements") — couldn't inline-anchor, the notebook diff is too large.

Version claim contradicts the cell it cites. This line says the committed run is "a dev build of the 0.14.0 line, as the version cell shows," but the version cell (cell 2) prints zagg 0.13.1.dev5+g0669a66df — a dev build of the 0.13.1 line (nearest reachable tag 0.13.0, 5 commits back), not the 0.14.0 line. hatch-vcs derives that string from git describe, so no 0.14.0 tag is reachable from this branch's history. Either retag/rebuild so the version cell actually reads 0.14.x, or reword to match what the cell shows (e.g. "a post-0.13.0 dev build carrying the #163 + #165 feature set").

[espg]

🤖 from Claude (review)

notebooks/sponsor_end_to_end.ipynb, intro cell (the Binder badge at the top) — couldn't inline-anchor, the notebook diff is too large.

This notebook carries a Binder badge but is not Binder-runnable end to end. §5 reads an ~88 GB local granule cache under ~/ignore/... (cell 16 GRANULE_CACHE) and §6's read-back opens the Zarr stores that §5 writes locally, so "Run All" on Binder fails from §5 onward. The repo's own .binder/environment.yml / postBuild enumerate the Binder-runnable notebooks and deliberately exclude this one, and CLAUDE.md requires Binder notebooks to read git-tree or anonymously-reachable data only. Consider dropping the badge here (the zero-credential companion sponsor_read_only.ipynb is the Binder entry point) or scoping the claim to the anonymous §1–3.

[espg]

🤖 from Claude

Review findings 2 and 3 folded in 724aa47, in the restructured (fleet-dispatch) notebook:

• Version-line contradiction — the "dev build of the 0.14.0 line, as the version cell shows" sentence is gone. Notebook A is now committed unexecuted (espg runs the fleet dispatch himself), and its status note only claims what is true: it targets zagg 0.14.0 / h5coro-hidefix 0.2.0 (both live on PyPI) and was pre-validated in-process before commit.
• Binder badge on notebook A — removed entirely (it dispatches to Lambda, so like cryocloud_example it is excluded from Binder); the intro now names sponsor_read_only.ipynb as the Binder entry point. The badge remains only on notebook B.

[espg]

🤖 from Claude

Run complete — ball is with espg (label waiting). Final state at 724aa47:

• Notebook A is the production fleet run, committed unexecuted: §4 dispatches both aggregations to the deployed 0.14.0 process-shard fleet (t-digest inline+write_back → zagg-index/ATL03/007/, then gain/bias via sidecar consuming that cache), §5 reads the published stores anonymously. Every section was pre-validated in-process (9/9 shards, 9,564,649 photons per run, 0 errors, sidecar byte-identical to inline on the spot-check); reference stores retained at ~/.cache/zagg/serc_example/.
• Notebook B (Binder-badged, zero-credential) is committed unexecuted behind a status note until the bucket is populated.
• The PR body now carries the pre-run checklist (SSO, the zagg-index PutObject IAM grant to verify on first write-back, deploy state, the issue Order-10 single-shard processing OOMs at 2 GB (gain_bias) — 2 GB is marginal for order-10 #117 memory caveat) and the how-to-run steps.
• All three adversarial-review findings from this pass are folded (724aa47) and replied to inline.

Next session, after the live run: execute sponsor_read_only.ipynb (no credentials) and re-commit both notebooks rendered.

[espg]

first run of tdigest (create sidecar files):

store: s3://sliderule-public-cors/zagg-examples/serc/tdigest.zarr
index: {'backend': 'inline', 'write_back': True, 'store': 's3://sliderule-public-cors/zagg-index/ATL03/007'}
{'total_cells': 9,
 'cells_with_data': 9,
 'cells_error': 0,
 'total_obs': 9564649,
 'wall_time_s': 360.0523192882538,
 'lambda_time_s': 2204.0837520000005,
 'gb_seconds': 4408.167504000001,
 'price_per_gb_sec': 1.33334e-05,
 'estimated_cost_usd': 0.058775860597833615,
 'setup_s': 5.515310287475586,
 'fanout_s': 313.30642008781433,
 'finalize_s': 46.74589157104492,
 'function_timeout_s': 900,
 'worker_max_s': 303.420758,
 'worker_median_s': 206.045903,
 'worker_pstdev_s': 47.69821062512117,
 'worker_pct_timeout': 0.3371341755555555,
 'max_memory_mb': 854.8203125,
 'store_path': 's3://sliderule-public/zagg-examples/serc/tdigest.zarr',
 'backend': 'lambda',
 'function_name': 'process-shard'}

Using it, same call but with sidecar backend:

store: s3://sliderule-public-cors/zagg-examples/serc/tdigest.zarr
index: {'backend': 'sidecar', 'store': 's3://sliderule-public-cors/zagg-index/ATL03/007', 'on_miss': 'fallback'}
  [8[/9](http://localhost:8890/9)] shard 5347405532456026122: no worker result within 990s (worker timed out, was OOM-killed, or crashed before writing its result (check CloudWatch) -- or the deployed worker predates result_url support: redeploy the function, or pass invocation="sync")
  [9[/9](http://localhost:8890/9)] shard 5347397835874631690: no worker result within 990s (worker timed out, was OOM-killed, or crashed before writing its result (check CloudWatch) -- or the deployed worker predates result_url support: redeploy the function, or pass invocation="sync")
{'total_cells': 9,
 'cells_with_data': 7,
 'cells_error': 2,
 'total_obs': 7822387,
 'wall_time_s': 1041.041348695755,
 'lambda_time_s': 1444.253639,
 'gb_seconds': 2888.507278,
 'price_per_gb_sec': 1.33334e-05,
 'estimated_cost_usd': 0.0385136229404852,
 'setup_s': 0.5216705799102783,
 'fanout_s': 994.559802532196,
 'finalize_s': 46.481542348861694,
 'function_timeout_s': 900,
 'worker_max_s': 267.1836,
 'worker_median_s': 181.050427,
 'worker_pstdev_s': 39.223013223787916,
 'worker_pct_timeout': 0.29687066666666667,
 'max_memory_mb': 2017.890625,
 'store_path': 's3://sliderule-public/zagg-examples/serc/tdigest2.zarr',
 'backend': 'lambda',
 'function_name': 'process-shard'}

[espg]

🤖 from Claude

Sidecar-cache A/B/C benchmark (SERC 9-shard fleet, 2026-07-06 ~19:30–20:30 UTC)

Method: backed up + cleared zagg-index/ATL03/007/ (59 manifests), then three timed fleet runs of the t-digest config to fresh sliderule-public stores, each with profile=True (per-worker phase_timings), spaced for container recycling and verified via Init Duration REPORT lines. All arms: 9/9 shards, 9,564,649 obs, 0 errors, worker_pstdev ~40 s.

arm	index config	wall	worker median	read median	agg	write	GB-s
hierarchical	{backend: hierarchical} (the pre-#170 default)	342 s	205 s	139 s	62 s	54 s	4257
inline + write_back (cache create)	{backend: inline, write_back: true}	334 s	226 s	166 s	60 s	52 s	4152
sidecar (cache consume)	{backend: sidecar, on_miss: fallback}	332 s	183 s	126 s	60 s	52 s	3813

• The cache does something real: sidecar cut worker median 19% vs the create run and 10% vs hierarchical, read median 25%/10%, GB-s cost 8–10%. Zero manifest misses (no sidecar: no manifest log lines) — all 59 granule manifests consumed.
• But the ceiling is fetch-bound, not decode-bound, on-Lambda: the densest shard's read barely moved (hier 213 s / inline 218 s / sidecar 211 s). The sidecar path issues one serial ranged GET per chunk (zagg_backend.py:_read_fn_for) where h5coro's cached reads coalesce into fewer 4 MB lines — so the Rust decode win is mostly given back to round trips. That's precisely the Compiled (hidefix) read path is never exercised by default, and its fetch/decode is serial #170 phase-2 / h5coro-hidefix executor work; the PR h5coro read-path benchmark: profile + 4-variant comparison (issue #149) #150 numbers say the read column should drop toward tens of seconds once fetches parallelize.
• Coldness: create arm verified 8/9 cold; sidecar arm all-clean memory (484–822 MB, per-invocation ≡ container HWM); hierarchical arm had 3 warm containers from the prior arm (memory column conflated for those, durations unaffected in the shard-level comparison above).
• Caveat / incident during the run: the 18:59 UTC stack update had set the role's S3 resource to the bare bucket ARN (no /*), which revoked all object access on sliderule-public-cors — the create arm's 59 write_back PUTs all 403'd (swallowed as per-granule WARNINGs; the run still reported success — worth hardening). I restored the manifests byte-identical from the pre-run backup; espg's /* fix (19:51 UTC) is live, so write-back self-population works for future runs. Note the miss is silent: a "cache-create" run that cannot write its manifests looks identical to a successful one in the summary dict.

Full per-worker tables + raw status JSONs retained locally; stores at s3://sliderule-public/zagg-examples/serc/bench_t1_{create,sidecar,hierarchical}.zarr.

Refs: #170 (read-path fix, planned + in progress), #171 / PR #172 (force-cold dispatch), the warm-container forensics earlier in this thread.

[espg]

🤖 from Claude

This run: attempted the anonymous execution of sponsor_read_only.ipynb (phase 4's second half) against the now-populated bucket. It fails at the first code cell — because of bucket state, not notebook code — so nothing was committed. Acknowledging the fleet results and the A/B/C benchmark: the deployment layer and the sidecar cache clearly work, and the waiting state was cleared to act on them. What the unsigned reads (the exact path Binder users take — anonymous S3 through this session's proxy works fine) found:

1. zagg-index/ATL03/007/ is complete — 59/59 stem-keyed granule manifests, anonymously fetchable (restored 19:42 UTC per the benchmark comment).
2. zagg-examples/serc/tdigest.zarr on sliderule-public-cors is a failed-write remnant. Summing 19/count per shard gives 855,626 obs vs the expected 9,564,649 (10.6 MB of chunks vs the ~120 MB reference store). The store's own status objects say why: all 9 shard JSONs under tdigest.zarr.status/83cb4ed9…/ (written 17:52–17:53 UTC, i.e. before the runs reported at 19:08) are statusCode: 500 — Failed to write zarr: … s3:DeleteObject … "no identity-based policy allows the s3:DeleteObject action" for zagg-backend-ExecutionRole-…/process-shard. Each worker computed its shard fully (the 9 per-shard total_obs sum to exactly 9,564,649) and died at the store write: zarr's overwrite path issues DeleteObjects, and the role policy grants Put/Get but not Delete. No chunk in the -cors store is newer than 17:53 UTC — and the 19:08 summary dicts print store_path: s3://sliderule-public/... (no -cors, and that bucket denies anonymous GET/LIST), so the complete 9/9 store apparently lives where the zero-credential notebook can't reach it.
3. zagg-examples/serc/gain_bias.zarr does not exist. The 19:08 runs and all three benchmark arms were the t-digest config; the gain/bias waveform dispatch (notebook A §4's second call, the sidecar consumer) hasn't run yet. Notebook B opens it in cell 2 (GroupNotFoundError) and §5's digest-vs-waveform cross-check depends on it.

One-pass unblock — with these done, the next routine run executes and commits notebook B with no further input: (a) grant the execution role s3:DeleteObject on zagg-examples/* (or pre-clear the prefix) and re-publish the t-digest store to s3://sliderule-public-cors/zagg-examples/serc/tdigest.zarr; (b) dispatch the gain/bias run (notebook A §4, second call) to .../gain_bias.zarr on the same bucket. If you'd rather point notebook B at different prefixes instead, say which.

write_back-403 hardening (from the benchmark comment): deferred as out of this PR's scope. The swallow is core worker code — src/zagg/processing/worker.py wraps index_backend.finish_granule in a log-and-continue except Exception (lines ~304–307) and nothing reaches the worker result or summary dict, so a cache-create run that persisted zero manifests reports clean. This PR touches only notebooks/configs/tests, and the fix implies a worker-result schema + summary aggregation change, so it needs its own issue rather than a phase here (today's DeleteObject finding is the same failure family: permission gap, quiet degradation — an index_write_failures / store_write_errors count in the summary would have caught both). Can file that issue on your go-ahead.

On the two failed sidecar shards at 19:08: the first, 5347405532456026122, is exactly the shard issue #117 flagged as OOM-at-2 GB for o10 (here max_memory_mb: 2017.9 ≈ the cap, "no worker result within 990s"). The benchmark's sidecar arm later going 9/9 green on all-clean containers (484–822 MB) points at the warm-container RSS ratchet rather than the workload itself — #177's recycle-generation counting (with #171 / PR #172) is the mitigation path, #117's 2 GB marginality the underlying ceiling.

Branch unchanged at 724aa47; label stays waiting on (a)+(b).

---

Generated by Claude Code

[espg]

🤖 from Claude

Pushed 5f047f6: output.grid.sharded: true in both sponsor configs, per espg (banking the #170 finding — at identical reads the ShardingCodec collapses the per-inner-chunk aggregation grouping + K CSR writes; on the o9 SERC AOI that's agg 165→19 s and write 148→15 s per worker, −44% cost vs inner; evidence table).

Rerun note for the pre-run checklist: this changes the published store layout (one sharded object per array/shard instead of K chunk objects), so the next fleet run should overwrite=True the existing zagg-examples/serc/*.zarr stores rather than write into the old layout — reads via zarr are layout-transparent, so notebook B needs no changes. The K=64-per-shard comment in these configs is also why the win is larger here than the benchmark's K=16 configs.