Skip to content

docs(spec): add data-publishing feature specification #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Anthony Cintron (prbatero) merged 1 commit into from
Aug 10, 2026
Merged

docs(spec): add data-publishing feature specification #112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions spec/features/data-publishing/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Feature: Data Publishing & Published Datasets

**Status:** draft
**Author:** HASTE engineering team
**Date:** 2026-08-05
**Target Release:** TBD
**Priority:** P2
**Work Item:** —

## Summary

A new **Published Datasets** section (a sibling to the **Model Catalog**) plus a
**Publish dataset** action on model results. Today an analyst can *download* a
model's outputs — the damage GeoPackage, valid-area mask, footprints, processed
COGs and the assessment report. This feature adds the ability to **publish**
those same HASTE-generated artifacts as a first-class, named, described dataset
that others can discover and retrieve. Publishing is routed through a
**provider abstraction** so a dataset can be published to different targets:
initially **Local** (registered inside HASTE-managed storage and listed in the
Published Datasets section) and **Planetary Computer** (Microsoft Planetary
Computer Pro GeoCatalog, using STAC-compatible metadata). The provider interface
is designed so new targets (e.g. an external STAC API, ArcGIS, a data portal)
can be added later without reworking the UI, API, or async workflow.

## Motivation

- Analysts and partners repeatedly ask "where is the *final* dataset for this
event?" — today the answer is a set of ad-hoc publications.
- Downloads are ephemeral (SAS URLs, tied to a single model that may be
re-run or deleted). A published dataset is a **stable, described, discoverable**
record with its own name, description, provenance and status.
- The **Model Catalog** already proves the pattern (curated, listable, catalogued
entities). "Published Datasets" is the output-side analogue and reuses the same
storage, API and UI conventions.
- Publishing to **Planetary Computer Pro** lets HASTE outputs join the broader
geospatial STAC ecosystem (searchable, tileable, standards-based) with no
manual STAC authoring by the analyst.

## Success Criteria

- [ ] From a completed model's results, an analyst can open **Publish dataset**,
see a name pre-filled from `${project} – ${layer}` and a description
pre-filled from the assessment report, pick a target, and publish.
- [ ] Published datasets appear in a new **Published Datasets** section with
empty / loading / success / failure / in-progress states.
- [ ] The **Local** provider registers the dataset in HASTE storage and exposes
stable retrieval links, independent of the source model's lifecycle.
- [ ] The **Planetary Computer** provider creates/updates a STAC Collection and
ingests STAC Item(s) for the dataset's artifacts into a GeoCatalog, and the
published record carries the collection id + explorer links.
- [ ] Adding a new provider requires only a new `PublishingProvider`
implementation + registry entry — no UI/API/queue changes.
- [ ] Publishing that takes time runs as an async job with visible status,
matching the training/inference/zip job pattern.

## HASTE Components Affected

| Component | Impact |
|---|---|
| `hastelib/src/hastegeo/core/models/` | new `publishing.py` (`PublishedDataset`, `PublishRequest`, enums, `ProviderInfo`) |
| `hastelib/src/hastegeo/core/publishing/` | **new subpackage**: provider ABC, registry, `local`, `planetary_computer`, STAC builders |
| `hastelib/src/hastegeo/core/processors/` | new `publishing.py` — orchestrates validate → persist → enqueue → run provider → status |
| `hastelib/src/hastegeo/core/config.py` | `PUBLISHED_DATASETS` metadata type; `publish_queue_name`; PC GeoCatalog config keys |
| `api/hastefuncapi/` | `GetPublishingProviders`, `GetPublishedDatasets`, `GetPublishedDataset`, `PutPublishDatasetQueueMessage`, `DeletePublishedDataset` |
| `api/hastefuncqueues/` | `GetPublishDatasetQueueMessage` trigger on `publish-queue` |
| `ui/src/Components/` | new `PublishedDatasets.jsx`, `PublishedDatasetRow.jsx`, `PublishDatasetModal.jsx`; "Publish dataset…" in `ProjectManagement/ModelResultsButton.jsx`; sidebar/route wiring |
| `ui/src/util/` | new API helpers (via existing `api.js`); shared assessment-summary helper |
| `docker/` | Azurite `publish-queue` seed; optional PC emulator/config env |
| `.github/workflows/` | Component Governance for new Python deps (`azure-identity`, `pystac`, `geopandas`, `pyogrio`, `shapely`) |

## Related Specs

| Spec | Relationship |
|---|---|
| [open-data-catalog](../open-data-catalog/) | related — reuses STAC concepts, TiTiler preview, and the "browse external geospatial data" precedent (this feature is the *publish/output* counterpart to that *discover/input* feature) |
| [gdal-compensating-controls](../gdal-compensating-controls/) | related — STAC item generation reads GeoTIFF/GPKG under the GDAL driver allowlist |

## Document Index

| Document | Purpose | Status |
|---|---|---|
| [user-stories.md](user-stories.md) | Personas, user stories & acceptance criteria (product requirements) | draft |
| [ux-spec.md](ux-spec.md) | UX specification: Published Datasets section + Publish dialog, all UI states | draft |
| [design.md](design.md) | Technical design, provider interface, Local + Planetary Computer providers, API contracts | draft |
| [data-model.md](data-model.md) | Cosmos/Blob metadata schema, published-dataset storage layout, STAC mapping | draft |
| [plan.md](plan.md) | Execution plan, milestones, phases | draft |
| [impact-analysis.md](impact-analysis.md) | Risk, dependencies, blast radius | draft |
| [test-plan.md](test-plan.md) | Test strategy & coverage matrix | draft |
| [rollout.md](rollout.md) | Rollout strategy, flags, rollback | draft |

## Key Design Decisions

- **Model Published Datasets on the Model Catalog pattern** — a single `index`
metadata doc, `Get/Put/Delete` routes, and a catalog-style React page. The two
features are symmetric (curated inputs vs curated outputs) and reuse the same
storage/API/UI conventions.
- **`PublishingProvider` abstraction + registry from day one** — extensibility is
a core requirement; Local + Planetary Computer already prove ≥2 providers, so
the seam must exist. New targets are a new provider subclass + registry entry,
with no UI/API/queue change.
- **All publishing runs through the async `publish-queue`** (even Local) — one
uniform status lifecycle (`PENDING → IN_PROGRESS → PUBLISHED | FAILED`) matching
training/inference/zip; PC ingestion is inherently async, so a single path
avoids a sync/async split in the UI.
- **Local provider copies artifacts** into an immutable `published/{datasetId}/`
prefix so a published dataset survives the source model being re-run or deleted.
- **Users select which existing outputs to publish** (GeoPackage, valid mask,
footprints, image COG …) via a prechecked checklist; the selection travels as
`artifacts: [...]` and providers publish only that subset.
- **Provider configuration is operator-owned** — Azure App Settings + managed
identity, set at deploy; no in-app admin screen in v1. The UI only reflects
`isConfigured` via `GetPublishingProviders`. The `ProviderInfo` contract still
allows a self-service admin UI later with no UI/API rework.
- **Planetary Computer = MPC Pro GeoCatalog STAC API** (`/stac/collections`,
`/stac/collections/{id}/items`, `api-version=2026-04-15`), auth via
`DefaultAzureCredential` (scope `https://geocatalog.spatio.azure.com/.default`).
Item geometry comes from the valid-area mask (`geopandas`/`shapely`); one STAC
Collection per project (≈ per event); private HASTE containers need a `SasToken`
ingestion source (managed-identity sources are portal/ARM only).
- **PC publishing is download-only in v1** — GeoPackage/GeoJSON are stored and
served via the STAC API but not tiled/rendered on the Explorer map (it appears
as item footprints + metadata; consumers download the GeoPackage). Rasterized
COG rendering is a future enhancement.
226 changes: 226 additions & 0 deletions spec/features/data-publishing/data-model.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
# Data Model: Data Publishing & Published Datasets

> HASTE's local/dev stack stores metadata as JSON in Blob Storage
> (`METADATA_STORAGE_TYPE=blob`) and, in cloud, in Cosmos DB, both behind
> `MetadataProcessor`. The "Cosmos" sections below describe the logical
> documents; they apply equally to the blob-backed metadata store. The design
> mirrors the **Model Catalog**, which stores a single `index` document under the
> `MODEL_CATALOG` metadata type (`function_app.py:2942`).

## Cosmos DB Changes

### New metadata type / logical container

| Container (metadata type) | Partition Key | Description |
|---|---|---|
| `PUBLISHED_DATASETS` | none (global `index`, like `MODEL_CATALOG`) | Single document `{"publishedDatasets": [PublishedDataset, ...]}` |

New enum member `MetadataTypes.PUBLISHED_DATASETS` in `config.py`
(`get_metadata_types()`), loaded/saved via
`MetadataProcessor(data_type=...).load("index")` — the exact pattern
`GetModelCatalog` uses.

### Modified Containers

| Container | Change | Migration Needed? |
|---|---|---|
| (none) | Existing `Model`/`ImageLayer`/`Project` documents are **read-only** inputs to publishing | no |

### New Document Schema

**Container:** `PUBLISHED_DATASETS` · **Document id:** `index` (single) holding
an array of `PublishedDataset`:

```jsonc
// PublishedDataset (one array element)
{
"datasetId": "uuid", // primary key within the array
"name": "Hurricane Harvey – Layer 1",// user-edited, prefilled '<project> – <layer>'
"description": "string", // prefilled from assessment report summary
"projectId": "uuid",
"imageLayerId": "string",
"modelId": "string", // source model whose artifacts were published
"target": "local | planetary_computer",
"status": "PENDING | IN_PROGRESS | PUBLISHED | FAILED",
"statusMessage": "string", // appended log, like Model.statusMessage
"publishedByUser": "user@contoso.com",
"createdDate": "ISO 8601",
"publishedDate": "ISO 8601 | null", // set when status → PUBLISHED
"artifacts": [ // user-selected subset that was published
{ "kind": "gpkg", "mediaType": "application/geopackage+sqlite3", "blobPath": "…", "sizeBytes": 12345 },
{ "kind": "valid_mask", "mediaType": "application/geo+json", "blobPath": "…" },
{ "kind": "processed_cog", "mediaType": "image/tiff; application=geotiff", "blobPath": "…" }
],
"links": { // provider output (retrieval)
"gpkg": "https://…sas", // Local
"stac_collection": "https://…/stac/collections/haste-…", // PC
"explorer": "https://…" // PC
},
"providerMetadata": { // provider-specific, opaque to UI
"collectionId": "haste-…", "itemIds": ["…"], "apiVersion": "2026-04-15"
},
"assessmentSummary": { // snapshot for provenance + STAC properties
"predictedDamaged": 1234, "precision": 0.82, "recall": 0.77
}
}
```

**RU / cost:** the `index` document grows by ~1–2 KB per dataset; a single upsert
per publish/status transition — negligible, same profile as the model catalog.

### Modified Document Schema

| Container | Field | Before | After | Notes |
|---|---|---|---|---|
| (none) | — | — | — | No changes to existing documents; publishing only reads them |

---

## Blob Storage Changes

### New path prefix (no new container)

| Container | Access Level | Naming Convention | Content Type |
|---|---|---|---|
| existing artifacts/data container | private | `{hash(projectId)}/published/{datasetId}/{artifact_name}` | GPKG / GeoTIFF / GeoJSON / JSON |

The Local provider copies the source model's artifacts into an **immutable
published prefix** so the dataset survives re-run/deletion of the source model.

### Blob Path Conventions

```
{container}/
{hash(projectId)}/
published/
{datasetId}/
predicted_damage_{modelName}.gpkg
valid_area_mask_{projectId}_{layerId}.geojson
processed_imagery_post_event_cog_{projectId}_{layerId}.tif
building_footprints_{projectId}_{layerId}.gpkg
assessment_report_{datasetId}.json # snapshot for provenance
```

Names reuse the existing artifact templates (`config.py:69-141`); the `published/`
segment and `{datasetId}` are the only new path elements.

### Modified Containers

| Container | Change | Description |
|---|---|---|
| (none) | additive prefix only | Existing artifact paths untouched |

---

## Data Lake Changes

None. Large COGs already live in the artifact/data store; the published prefix is
in the same store. No new filesystem.

---

## Queue Storage Changes

### New Queues

| Queue Name | Message Schema | Producer | Consumer |
|---|---|---|---|
| `publish-queue` | `{ "datasetId": "…", "projectId": "…" }` | `hastefuncapi` (`PutPublishDatasetQueueMessage`) | `hastefuncqueues` (`GetPublishDatasetQueueMessage`) |

Registered in `config.get_queue_config()` as `publish_queue_name`
(default `publish-queue`), alongside the existing `train`/`inference`/`zip`
queues. Azurite seeds it in the dev stack.

---

## Azure Batch Changes

None. Publishing is I/O-bound (blob copy, STAC HTTP calls) and runs in the
Functions queue worker — no GPU/Batch pool. (If future providers need heavy
raster reprocessing, the provider can enqueue Batch work, but v1 does not.)

---

## STAC mapping (Planetary Computer target)

Logical mapping from HASTE artifacts to STAC (see
[design.md](design.md#stac-mapping)):

| HASTE artifact | STAC representation | Key fields |
|---|---|---|
| Valid-area mask GeoJSON | **Item geometry** (+ `aoi` asset) | union polygon → EPSG:4326 `geometry`/`bbox`; `ai4g:aoi_area_km2` computed; asset `application/geo+json`, roles `[metadata]` |
| Damage GPKG (`predicted_damage_*`) | `buildings` asset on the Item | `application/geopackage+sqlite3`, roles `[data]`, `proj:code` of source CRS |
| Building footprints GPKG | `buildings` GPKG already carries footprints (or its own asset) | `application/geopackage+sqlite3` |
| Assessment report | Item `properties` (`ai4g:buildings_total/cloud/clear/damaged`, `…validation_*`) | from `assessmentSummary` |
| Project (≈ event) | STAC Collection | `id=haste-<projectSlug>`, `extent`, `providers`, `keywords`, `summaries`, `item_assets`, `stac_extensions:[item-assets/v1.0.0]` |
| Item | `stac_extensions:[projection/v2.0.0]`, `collection=<id>` | id sanitized (no `-_+().`) |

- **Item geometry is the valid-area mask**, not a raster footprint — the region
actually assessed. `rio-stac`/raster items are only introduced if/when a
rasterized COG is published for map rendering (out of scope v1).
- **Vector assets (GPKG/GeoJSON) are download-only** in PC Pro — stored and
served via the STAC API but **not tiled / not rendered** in the Explorer (see
[design.md render limitation](design.md#planetary-computer-provider--stac-mapping)).
- On ingest, the GeoCatalog **copies assets into its own managed storage and
rewrites hrefs**; reading them needs a collection SAS token
(`GET /sas/token/{collectionId}`).

---

## Data Flow

### Write path

```
UI → PutPublishDatasetQueueMessage (validate + provider.validate)
→ PUBLISHED_DATASETS index doc (PENDING)
→ publish-queue (datasetId)
hastefuncqueues → PublishingProcessor.run → provider.publish
Local: copy → {hash}/published/{datasetId}/… ; links=SAS
PC: pystac/geopandas → POST /stac/collections(/items) → poll operations
→ PUBLISHED_DATASETS index doc (PUBLISHED | FAILED, links, providerMetadata)
```

### Read path

```
UI → GetPublishedDatasets → PUBLISHED_DATASETS index doc (list)
UI → GetPublishedDataset → single record (+ links)
UI → Local artifact SAS URL (direct download) | PC explorer/collection link (external)
```

## Migration Plan

### Forward

1. Add `PUBLISHED_DATASETS` metadata type + `publish_queue_name` (additive).
2. Deploy `hastelib` publishing package + `hastefuncqueues` trigger.
3. Deploy `hastefuncapi` routes.
4. Deploy UI section + dialog.

No backfill: the `index` document is created lazily on first publish (like the
model catalog's `FileNotFoundError → empty catalog`).

### Backward

- Fully reversible. Reverting API/UI hides the feature; the `PUBLISHED_DATASETS`
document and `published/` blobs are inert (unknown metadata type / extra blob
prefix are harmless). Optional cleanup: delete the `index` doc and
`published/*` prefixes. PC collections/items, if created, persist in the
GeoCatalog until deleted via its API (out-of-band).

## Data Volume Estimates

| Entity / Container | Initial Size | Growth Rate | Retention |
|---|---|---|---|
| `PUBLISHED_DATASETS` index doc | ~1 KB | ~1–2 KB per dataset | life of project |
| `published/{datasetId}/` copies | = source artifacts (MB–GB) | per published dataset | until unpublish/project delete |
| `publish-queue` messages | tiny | transient | consumed immediately |

## Caching Strategy

| Data | Cache Layer | TTL | Invalidation |
|---|---|---|---|
| Published dataset list | Browser (per section open) | session | Re-fetch on publish/poll |
| Provider list | Browser | session | Re-fetch on dialog open |
| Local artifact SAS URLs | none (short-lived SAS) | SAS expiry | Re-issued on `GetPublishedDataset` |
Loading
Loading