From 123335d1fc07b45f7f5d90f0c0af01c06932f993 Mon Sep 17 00:00:00 2001 From: Anthony Cintron Roman Date: Sun, 9 Aug 2026 16:50:37 -0400 Subject: [PATCH] docs(spec): add data-publishing feature specification Feature spec for Data Publishing & Published Datasets: a Published Datasets section (analogous to the Model Catalog) plus a "Publish dataset" action on model results, routed through a PublishingProvider abstraction with a Local (HASTE storage) provider and a Planetary Computer (MPC Pro GeoCatalog / STAC) provider. Documents: README (summary + key design decisions), user-stories, ux-spec, design, data-model, plan, impact-analysis, test-plan, rollout. Rollout carries an operator-facing App Settings reference (azd env var -> App Setting mapping, enablement steps, and the operator-owned GeoCatalog grants). --- spec/features/data-publishing/README.md | 123 +++++ spec/features/data-publishing/data-model.md | 226 ++++++++ spec/features/data-publishing/design.md | 489 ++++++++++++++++++ .../data-publishing/impact-analysis.md | 121 +++++ spec/features/data-publishing/plan.md | 131 +++++ spec/features/data-publishing/rollout.md | 179 +++++++ spec/features/data-publishing/test-plan.md | 135 +++++ spec/features/data-publishing/user-stories.md | 341 ++++++++++++ spec/features/data-publishing/ux-spec.md | 203 ++++++++ 9 files changed, 1948 insertions(+) create mode 100644 spec/features/data-publishing/README.md create mode 100644 spec/features/data-publishing/data-model.md create mode 100644 spec/features/data-publishing/design.md create mode 100644 spec/features/data-publishing/impact-analysis.md create mode 100644 spec/features/data-publishing/plan.md create mode 100644 spec/features/data-publishing/rollout.md create mode 100644 spec/features/data-publishing/test-plan.md create mode 100644 spec/features/data-publishing/user-stories.md create mode 100644 spec/features/data-publishing/ux-spec.md diff --git a/spec/features/data-publishing/README.md b/spec/features/data-publishing/README.md new file mode 100644 index 00000000..8a8de473 --- /dev/null +++ b/spec/features/data-publishing/README.md @@ -0,0 +1,123 @@ +# Feature: Data Publishing & Published Datasets + +**Status:** draft +**Author:** HASTE engineering team +**Date:** 2026-08-05 +**Target Release:** TBD +**Priority:** P2 +**Work Item:** — + +## Summary + +A new **Published Datasets** section (a sibling to the **Model Catalog**) plus a +**Publish dataset** action on model results. Today an analyst can *download* a +model's outputs — the damage GeoPackage, valid-area mask, footprints, processed +COGs and the assessment report. This feature adds the ability to **publish** +those same HASTE-generated artifacts as a first-class, named, described dataset +that others can discover and retrieve. Publishing is routed through a +**provider abstraction** so a dataset can be published to different targets: +initially **Local** (registered inside HASTE-managed storage and listed in the +Published Datasets section) and **Planetary Computer** (Microsoft Planetary +Computer Pro GeoCatalog, using STAC-compatible metadata). The provider interface +is designed so new targets (e.g. an external STAC API, ArcGIS, a data portal) +can be added later without reworking the UI, API, or async workflow. + +## Motivation + +- Analysts and partners repeatedly ask "where is the *final* dataset for this + event?" — today the answer is a set of ad-hoc publications. +- Downloads are ephemeral (SAS URLs, tied to a single model that may be + re-run or deleted). A published dataset is a **stable, described, discoverable** + record with its own name, description, provenance and status. +- The **Model Catalog** already proves the pattern (curated, listable, catalogued + entities). "Published Datasets" is the output-side analogue and reuses the same + storage, API and UI conventions. +- Publishing to **Planetary Computer Pro** lets HASTE outputs join the broader + geospatial STAC ecosystem (searchable, tileable, standards-based) with no + manual STAC authoring by the analyst. + +## Success Criteria + +- [ ] From a completed model's results, an analyst can open **Publish dataset**, + see a name pre-filled from `${project} – ${layer}` and a description + pre-filled from the assessment report, pick a target, and publish. +- [ ] Published datasets appear in a new **Published Datasets** section with + empty / loading / success / failure / in-progress states. +- [ ] The **Local** provider registers the dataset in HASTE storage and exposes + stable retrieval links, independent of the source model's lifecycle. +- [ ] The **Planetary Computer** provider creates/updates a STAC Collection and + ingests STAC Item(s) for the dataset's artifacts into a GeoCatalog, and the + published record carries the collection id + explorer links. +- [ ] Adding a new provider requires only a new `PublishingProvider` + implementation + registry entry — no UI/API/queue changes. +- [ ] Publishing that takes time runs as an async job with visible status, + matching the training/inference/zip job pattern. + +## HASTE Components Affected + +| Component | Impact | +|---|---| +| `hastelib/src/hastegeo/core/models/` | new `publishing.py` (`PublishedDataset`, `PublishRequest`, enums, `ProviderInfo`) | +| `hastelib/src/hastegeo/core/publishing/` | **new subpackage**: provider ABC, registry, `local`, `planetary_computer`, STAC builders | +| `hastelib/src/hastegeo/core/processors/` | new `publishing.py` — orchestrates validate → persist → enqueue → run provider → status | +| `hastelib/src/hastegeo/core/config.py` | `PUBLISHED_DATASETS` metadata type; `publish_queue_name`; PC GeoCatalog config keys | +| `api/hastefuncapi/` | `GetPublishingProviders`, `GetPublishedDatasets`, `GetPublishedDataset`, `PutPublishDatasetQueueMessage`, `DeletePublishedDataset` | +| `api/hastefuncqueues/` | `GetPublishDatasetQueueMessage` trigger on `publish-queue` | +| `ui/src/Components/` | new `PublishedDatasets.jsx`, `PublishedDatasetRow.jsx`, `PublishDatasetModal.jsx`; "Publish dataset…" in `ProjectManagement/ModelResultsButton.jsx`; sidebar/route wiring | +| `ui/src/util/` | new API helpers (via existing `api.js`); shared assessment-summary helper | +| `docker/` | Azurite `publish-queue` seed; optional PC emulator/config env | +| `.github/workflows/` | Component Governance for new Python deps (`azure-identity`, `pystac`, `geopandas`, `pyogrio`, `shapely`) | + +## Related Specs + +| Spec | Relationship | +|---|---| +| [open-data-catalog](../open-data-catalog/) | related — reuses STAC concepts, TiTiler preview, and the "browse external geospatial data" precedent (this feature is the *publish/output* counterpart to that *discover/input* feature) | +| [gdal-compensating-controls](../gdal-compensating-controls/) | related — STAC item generation reads GeoTIFF/GPKG under the GDAL driver allowlist | + +## Document Index + +| Document | Purpose | Status | +|---|---|---| +| [user-stories.md](user-stories.md) | Personas, user stories & acceptance criteria (product requirements) | draft | +| [ux-spec.md](ux-spec.md) | UX specification: Published Datasets section + Publish dialog, all UI states | draft | +| [design.md](design.md) | Technical design, provider interface, Local + Planetary Computer providers, API contracts | draft | +| [data-model.md](data-model.md) | Cosmos/Blob metadata schema, published-dataset storage layout, STAC mapping | draft | +| [plan.md](plan.md) | Execution plan, milestones, phases | draft | +| [impact-analysis.md](impact-analysis.md) | Risk, dependencies, blast radius | draft | +| [test-plan.md](test-plan.md) | Test strategy & coverage matrix | draft | +| [rollout.md](rollout.md) | Rollout strategy, flags, rollback | draft | + +## Key Design Decisions + +- **Model Published Datasets on the Model Catalog pattern** — a single `index` + metadata doc, `Get/Put/Delete` routes, and a catalog-style React page. The two + features are symmetric (curated inputs vs curated outputs) and reuse the same + storage/API/UI conventions. +- **`PublishingProvider` abstraction + registry from day one** — extensibility is + a core requirement; Local + Planetary Computer already prove ≥2 providers, so + the seam must exist. New targets are a new provider subclass + registry entry, + with no UI/API/queue change. +- **All publishing runs through the async `publish-queue`** (even Local) — one + uniform status lifecycle (`PENDING → IN_PROGRESS → PUBLISHED | FAILED`) matching + training/inference/zip; PC ingestion is inherently async, so a single path + avoids a sync/async split in the UI. +- **Local provider copies artifacts** into an immutable `published/{datasetId}/` + prefix so a published dataset survives the source model being re-run or deleted. +- **Users select which existing outputs to publish** (GeoPackage, valid mask, + footprints, image COG …) via a prechecked checklist; the selection travels as + `artifacts: [...]` and providers publish only that subset. +- **Provider configuration is operator-owned** — Azure App Settings + managed + identity, set at deploy; no in-app admin screen in v1. The UI only reflects + `isConfigured` via `GetPublishingProviders`. The `ProviderInfo` contract still + allows a self-service admin UI later with no UI/API rework. +- **Planetary Computer = MPC Pro GeoCatalog STAC API** (`/stac/collections`, + `/stac/collections/{id}/items`, `api-version=2026-04-15`), auth via + `DefaultAzureCredential` (scope `https://geocatalog.spatio.azure.com/.default`). + Item geometry comes from the valid-area mask (`geopandas`/`shapely`); one STAC + Collection per project (≈ per event); private HASTE containers need a `SasToken` + ingestion source (managed-identity sources are portal/ARM only). +- **PC publishing is download-only in v1** — GeoPackage/GeoJSON are stored and + served via the STAC API but not tiled/rendered on the Explorer map (it appears + as item footprints + metadata; consumers download the GeoPackage). Rasterized + COG rendering is a future enhancement. diff --git a/spec/features/data-publishing/data-model.md b/spec/features/data-publishing/data-model.md new file mode 100644 index 00000000..55f6dea4 --- /dev/null +++ b/spec/features/data-publishing/data-model.md @@ -0,0 +1,226 @@ +# Data Model: Data Publishing & Published Datasets + +> HASTE's local/dev stack stores metadata as JSON in Blob Storage +> (`METADATA_STORAGE_TYPE=blob`) and, in cloud, in Cosmos DB, both behind +> `MetadataProcessor`. The "Cosmos" sections below describe the logical +> documents; they apply equally to the blob-backed metadata store. The design +> mirrors the **Model Catalog**, which stores a single `index` document under the +> `MODEL_CATALOG` metadata type (`function_app.py:2942`). + +## Cosmos DB Changes + +### New metadata type / logical container + +| Container (metadata type) | Partition Key | Description | +|---|---|---| +| `PUBLISHED_DATASETS` | none (global `index`, like `MODEL_CATALOG`) | Single document `{"publishedDatasets": [PublishedDataset, ...]}` | + +New enum member `MetadataTypes.PUBLISHED_DATASETS` in `config.py` +(`get_metadata_types()`), loaded/saved via +`MetadataProcessor(data_type=...).load("index")` — the exact pattern +`GetModelCatalog` uses. + +### Modified Containers + +| Container | Change | Migration Needed? | +|---|---|---| +| (none) | Existing `Model`/`ImageLayer`/`Project` documents are **read-only** inputs to publishing | no | + +### New Document Schema + +**Container:** `PUBLISHED_DATASETS` · **Document id:** `index` (single) holding +an array of `PublishedDataset`: + +```jsonc +// PublishedDataset (one array element) +{ + "datasetId": "uuid", // primary key within the array + "name": "Hurricane Harvey – Layer 1",// user-edited, prefilled ' – ' + "description": "string", // prefilled from assessment report summary + "projectId": "uuid", + "imageLayerId": "string", + "modelId": "string", // source model whose artifacts were published + "target": "local | planetary_computer", + "status": "PENDING | IN_PROGRESS | PUBLISHED | FAILED", + "statusMessage": "string", // appended log, like Model.statusMessage + "publishedByUser": "user@contoso.com", + "createdDate": "ISO 8601", + "publishedDate": "ISO 8601 | null", // set when status → PUBLISHED + "artifacts": [ // user-selected subset that was published + { "kind": "gpkg", "mediaType": "application/geopackage+sqlite3", "blobPath": "…", "sizeBytes": 12345 }, + { "kind": "valid_mask", "mediaType": "application/geo+json", "blobPath": "…" }, + { "kind": "processed_cog", "mediaType": "image/tiff; application=geotiff", "blobPath": "…" } + ], + "links": { // provider output (retrieval) + "gpkg": "https://…sas", // Local + "stac_collection": "https://…/stac/collections/haste-…", // PC + "explorer": "https://…" // PC + }, + "providerMetadata": { // provider-specific, opaque to UI + "collectionId": "haste-…", "itemIds": ["…"], "apiVersion": "2026-04-15" + }, + "assessmentSummary": { // snapshot for provenance + STAC properties + "predictedDamaged": 1234, "precision": 0.82, "recall": 0.77 + } +} +``` + +**RU / cost:** the `index` document grows by ~1–2 KB per dataset; a single upsert +per publish/status transition — negligible, same profile as the model catalog. + +### Modified Document Schema + +| Container | Field | Before | After | Notes | +|---|---|---|---|---| +| (none) | — | — | — | No changes to existing documents; publishing only reads them | + +--- + +## Blob Storage Changes + +### New path prefix (no new container) + +| Container | Access Level | Naming Convention | Content Type | +|---|---|---|---| +| existing artifacts/data container | private | `{hash(projectId)}/published/{datasetId}/{artifact_name}` | GPKG / GeoTIFF / GeoJSON / JSON | + +The Local provider copies the source model's artifacts into an **immutable +published prefix** so the dataset survives re-run/deletion of the source model. + +### Blob Path Conventions + +``` +{container}/ + {hash(projectId)}/ + published/ + {datasetId}/ + predicted_damage_{modelName}.gpkg + valid_area_mask_{projectId}_{layerId}.geojson + processed_imagery_post_event_cog_{projectId}_{layerId}.tif + building_footprints_{projectId}_{layerId}.gpkg + assessment_report_{datasetId}.json # snapshot for provenance +``` + +Names reuse the existing artifact templates (`config.py:69-141`); the `published/` +segment and `{datasetId}` are the only new path elements. + +### Modified Containers + +| Container | Change | Description | +|---|---|---| +| (none) | additive prefix only | Existing artifact paths untouched | + +--- + +## Data Lake Changes + +None. Large COGs already live in the artifact/data store; the published prefix is +in the same store. No new filesystem. + +--- + +## Queue Storage Changes + +### New Queues + +| Queue Name | Message Schema | Producer | Consumer | +|---|---|---|---| +| `publish-queue` | `{ "datasetId": "…", "projectId": "…" }` | `hastefuncapi` (`PutPublishDatasetQueueMessage`) | `hastefuncqueues` (`GetPublishDatasetQueueMessage`) | + +Registered in `config.get_queue_config()` as `publish_queue_name` +(default `publish-queue`), alongside the existing `train`/`inference`/`zip` +queues. Azurite seeds it in the dev stack. + +--- + +## Azure Batch Changes + +None. Publishing is I/O-bound (blob copy, STAC HTTP calls) and runs in the +Functions queue worker — no GPU/Batch pool. (If future providers need heavy +raster reprocessing, the provider can enqueue Batch work, but v1 does not.) + +--- + +## STAC mapping (Planetary Computer target) + +Logical mapping from HASTE artifacts to STAC (see +[design.md](design.md#stac-mapping)): + +| HASTE artifact | STAC representation | Key fields | +|---|---|---| +| Valid-area mask GeoJSON | **Item geometry** (+ `aoi` asset) | union polygon → EPSG:4326 `geometry`/`bbox`; `ai4g:aoi_area_km2` computed; asset `application/geo+json`, roles `[metadata]` | +| Damage GPKG (`predicted_damage_*`) | `buildings` asset on the Item | `application/geopackage+sqlite3`, roles `[data]`, `proj:code` of source CRS | +| Building footprints GPKG | `buildings` GPKG already carries footprints (or its own asset) | `application/geopackage+sqlite3` | +| Assessment report | Item `properties` (`ai4g:buildings_total/cloud/clear/damaged`, `…validation_*`) | from `assessmentSummary` | +| Project (≈ event) | STAC Collection | `id=haste-`, `extent`, `providers`, `keywords`, `summaries`, `item_assets`, `stac_extensions:[item-assets/v1.0.0]` | +| Item | `stac_extensions:[projection/v2.0.0]`, `collection=` | id sanitized (no `-_+().`) | + +- **Item geometry is the valid-area mask**, not a raster footprint — the region + actually assessed. `rio-stac`/raster items are only introduced if/when a + rasterized COG is published for map rendering (out of scope v1). +- **Vector assets (GPKG/GeoJSON) are download-only** in PC Pro — stored and + served via the STAC API but **not tiled / not rendered** in the Explorer (see + [design.md render limitation](design.md#planetary-computer-provider--stac-mapping)). +- On ingest, the GeoCatalog **copies assets into its own managed storage and + rewrites hrefs**; reading them needs a collection SAS token + (`GET /sas/token/{collectionId}`). + +--- + +## Data Flow + +### Write path + +``` +UI → PutPublishDatasetQueueMessage (validate + provider.validate) + → PUBLISHED_DATASETS index doc (PENDING) + → publish-queue (datasetId) + hastefuncqueues → PublishingProcessor.run → provider.publish + Local: copy → {hash}/published/{datasetId}/… ; links=SAS + PC: pystac/geopandas → POST /stac/collections(/items) → poll operations + → PUBLISHED_DATASETS index doc (PUBLISHED | FAILED, links, providerMetadata) +``` + +### Read path + +``` +UI → GetPublishedDatasets → PUBLISHED_DATASETS index doc (list) +UI → GetPublishedDataset → single record (+ links) +UI → Local artifact SAS URL (direct download) | PC explorer/collection link (external) +``` + +## Migration Plan + +### Forward + +1. Add `PUBLISHED_DATASETS` metadata type + `publish_queue_name` (additive). +2. Deploy `hastelib` publishing package + `hastefuncqueues` trigger. +3. Deploy `hastefuncapi` routes. +4. Deploy UI section + dialog. + +No backfill: the `index` document is created lazily on first publish (like the +model catalog's `FileNotFoundError → empty catalog`). + +### Backward + +- Fully reversible. Reverting API/UI hides the feature; the `PUBLISHED_DATASETS` + document and `published/` blobs are inert (unknown metadata type / extra blob + prefix are harmless). Optional cleanup: delete the `index` doc and + `published/*` prefixes. PC collections/items, if created, persist in the + GeoCatalog until deleted via its API (out-of-band). + +## Data Volume Estimates + +| Entity / Container | Initial Size | Growth Rate | Retention | +|---|---|---|---| +| `PUBLISHED_DATASETS` index doc | ~1 KB | ~1–2 KB per dataset | life of project | +| `published/{datasetId}/` copies | = source artifacts (MB–GB) | per published dataset | until unpublish/project delete | +| `publish-queue` messages | tiny | transient | consumed immediately | + +## Caching Strategy + +| Data | Cache Layer | TTL | Invalidation | +|---|---|---|---| +| Published dataset list | Browser (per section open) | session | Re-fetch on publish/poll | +| Provider list | Browser | session | Re-fetch on dialog open | +| Local artifact SAS URLs | none (short-lived SAS) | SAS expiry | Re-issued on `GetPublishedDataset` | diff --git a/spec/features/data-publishing/design.md b/spec/features/data-publishing/design.md new file mode 100644 index 00000000..f6c1db16 --- /dev/null +++ b/spec/features/data-publishing/design.md @@ -0,0 +1,489 @@ +# Technical Design: Data Publishing & Published Datasets + +## Overview + +A **Publish dataset** action on model results opens a Fluent UI dialog +(name / description / target). Submitting calls `PutPublishDatasetQueueMessage`, +which writes a `PublishedDataset` record (status `PENDING`) to the metadata +store and enqueues a message on `publish-queue`. A queue worker resolves a +`PublishingProvider` from a registry and runs it: **Local** copies artifacts +into an immutable published prefix and records retrieval links; **Planetary +Computer** builds STAC and ingests it into a MPC Pro GeoCatalog. Status flows +`PENDING → IN_PROGRESS → PUBLISHED | FAILED`. A new catalog-style +**Published Datasets** section (modeled on `ModelCatalog.jsx`) lists the +records. The provider abstraction is the extension seam: new targets are new +`PublishingProvider` subclasses + a registry entry, with no UI/API/queue change. + +Reference: HASTE architecture in `docs/architecture.md`; STAC precedent in the +[open-data-catalog](../open-data-catalog/) spec. + +## Architecture + +### Component Diagram + +``` +┌───────────────────────────┐ PutPublishDatasetQueueMessage ┌────────────────────┐ +│ React UI │─────────────────────────────────▶│ hastefuncapi │ +│ ModelResultsButton → │ │ - validate input │ +│ PublishDatasetModal │◀── GetPublishingProviders ────────│ - write record │ +│ PublishedDatasets page │◀── GetPublishedDatasets ──────────│ - enqueue │ +└───────────────────────────┘ └─────────┬──────────┘ + queue msg │ (publish-queue) + ▼ + ┌────────────────────────────────────────────────────┐ + │ hastefuncqueues: GetPublishDatasetQueueMessage │ + │ PublishingProcessor.run(dataset) │ + │ registry.resolve(target) → PublishingProvider │ + └───────┬───────────────────────────────┬─────────────┘ + │ Local │ Planetary Computer + ┌───────────▼──────────┐ ┌────────────▼─────────────────┐ + │ LocalHasteStorage │ │ PlanetaryComputerProvider │ + │ Provider │ │ stac.py (pystac+geopandas) │ + │ copy → published/ │ │ POST /stac/collections │ + │ {datasetId}/ │ │ POST /stac/.../items │ + └───────────┬──────────┘ │ poll ingestion `location` │ + │ └────────────┬─────────────────┘ + ┌───────────▼──────────┐ ┌─────────────▼────────────────┐ + │ Blob / Data Lake │ │ MPC Pro GeoCatalog │ + │ (HASTE artifacts) │◀───────│ copies assets via ingestion │ + └───────────────────────┘ SAS │ source (SAS / managed id) │ + ▲ └───────────────────────────────┘ + │ update status/links + ┌───────────┴──────────┐ + │ PUBLISHED_DATASETS │ metadata (Cosmos/Blob) — single `index` doc + │ MetadataProcessor │ + └───────────────────────┘ +``` + +### New Components + +| Component | Path | Responsibility | Technology | +|---|---|---|---| +| Publishing models | `hastelib/src/hastegeo/core/models/publishing.py` | `PublishedDataset`, `PublishRequest`, `PublishTarget`/`PublishStatus` enums, `ProviderInfo`, `PublishResult` | Python / Pydantic | +| Provider ABC | `hastelib/src/hastegeo/core/publishing/base.py` | `PublishingProvider` abstract base + `ProviderConfigField` | Python | +| Provider registry | `hastelib/src/hastegeo/core/publishing/registry.py` | Register/resolve providers by id; expose `ProviderInfo` list | Python | +| Local provider | `hastelib/src/hastegeo/core/publishing/local_provider.py` | Copy/register artifacts in HASTE storage; return links | Python | +| GeoCatalog client + auth | `hastelib/src/hastegeo/core/publishing/geocatalog_client.py` | Hardened REST wrapper (collections/items/ingestion/SAS) + Entra token cache; no redirects, explicit timeouts, status-only errors | Python / requests / azure-identity | +| PC transport adapter | `hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py` | Turns the async `202` + operation-location flow into resumable steps; SSRF origin-pinning, failed-item accounting | Python | +| PC provider | `hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py` | Orchestrate: ensure collection, build+ingest items, poll, validate, links | Python / azure-identity | +| STAC builder | `hastelib/src/hastegeo/core/publishing/stac.py` | Build STAC Collection + vector Items (geometry from valid-area mask) from a HASTE `ArtifactBundle` + `assessmentSummary` | Python / pystac / geopandas / shapely | +| Publishing processor | `hastelib/src/hastegeo/core/processors/publishing.py` | Orchestrate validate → persist → enqueue → run → status | Python | +| Published Datasets page | `ui/src/Components/PublishedDatasets.jsx` | Catalog-style list (search/sort/paginate/states) | React / Fluent UI | +| Dataset row | `ui/src/Components/PublishedDatasetRow.jsx` | One dataset: metadata, status, retrieve/unpublish actions | React / Fluent UI | +| Publish dialog | `ui/src/Components/PublishDatasetModal.jsx` | Name/description/target form; prefill; submit | React / Fluent UI | + +### Modified Components + +| Component | Path | Change Description | +|---|---|---| +| Model results menu | `ui/src/Components/ProjectManagement/ModelResultsButton.jsx` | Add "Publish dataset…" `MenuItem` (after Assessment Report); open `PublishDatasetModal` | +| API helpers | `ui/src/util/api.js` | New calls via existing `apiGet`/`apiPost`/`apiDelete` (no new transport) | +| Assessment summary | new `ui/src/util/assessmentSummary.js` | Extract `buildSummarySentence()` from `AssessmentReportModal.jsx` for reuse as description prefill | +| Sidebar / routing | `ui/src/Components/AppSidebar.jsx`, `AppBody.jsx` | Register "Published Datasets" nav item + `/published-datasets` route | +| Config | `hastelib/src/hastegeo/core/config.py` | `MetadataTypes.PUBLISHED_DATASETS`; `publish_queue_name`; PC GeoCatalog config block | +| API module | `api/hastefuncapi/function_app.py` | 5 new thin routes (below) | +| Queue module | `api/hastefuncqueues/function_app.py` | `GetPublishDatasetQueueMessage` trigger | + +## Publishing provider interface + +The core abstraction. Providers are stateless, constructed with resolved config. + +```python +# core/publishing/base.py +class ProviderConfigField(BaseModel): + key: str # e.g. "geocatalog_url" + label: str + required: bool + secret: bool = False + +class PublishingProvider(ABC): + # --- metadata --- + @property + @abstractmethod + def provider_id(self) -> str: ... # "local" | "planetary_computer" + @property + @abstractmethod + def display_name(self) -> str: ... # "Local (HASTE storage)" + @property + def description(self) -> str: return "" + @property + def config_requirements(self) -> list[ProviderConfigField]: return [] + def is_configured(self) -> bool: return True # False → shown disabled in UI + + # --- validation (fast, pre-enqueue) --- + @abstractmethod + def validate(self, req: "PublishRequest") -> Optional[str]: + """None if publishable, else a user-facing error string.""" + + # --- publish (slow, in the queue worker) --- + @abstractmethod + def publish(self, dataset: "PublishedDataset", + artifacts: "ArtifactBundle") -> "PublishResult": + """Do the work; return links/status/provider metadata.""" + + # --- teardown (unpublish) --- + def unpublish(self, dataset: "PublishedDataset") -> None: + """Best-effort cleanup; default no-op.""" +``` + +`ProviderInfo` (returned to the UI by `GetPublishingProviders`) is the +serializable projection of a provider's metadata + `is_configured()`. +`ArtifactBundle` resolves a model's publishable artifacts (gpkg, valid mask, +processed COGs, footprints, assessment-report JSON) to `{kind, blob_path, +media_type, sas_url}` entries, built from the `Model`/`ImageLayer` documents. It +exposes which kinds are **available** (so the dialog can render only real +outputs) and is **filtered to the user-selected `artifacts`** before a provider +runs — providers publish exactly the selected subset, never more. + +### Registry + +```python +# core/publishing/registry.py +_REGISTRY: dict[str, PublishingProvider] = {} +def register(p: PublishingProvider): _REGISTRY[p.provider_id] = p +def resolve(provider_id: str) -> PublishingProvider: ... +def list_infos() -> list[ProviderInfo]: ... # for GetPublishingProviders +# Local + PlanetaryComputer registered at import time. +``` + +Adding a provider = implement `PublishingProvider`, call `register(...)`. No +other layer changes. This satisfies the extensibility success criterion. + +### Internal interfaces (hastegeo) + +| Module | Function/Class | Signature | Description | +|---|---|---|---| +| `core/models/publishing.py` | `PublishedDataset` | Pydantic model | Persisted record (see [data-model.md](data-model.md)) | +| `core/publishing/base.py` | `PublishingProvider` | ABC | Provider contract above | +| `core/publishing/registry.py` | `resolve`, `list_infos` | `(id)->provider`, `()->[ProviderInfo]` | Registry access | +| `core/publishing/local_provider.py` | `LocalHasteStorageProvider.publish` | `(dataset, bundle)->PublishResult` | Copy + link | +| `core/publishing/planetary_computer_provider.py` | `PlanetaryComputerProvider.publish` | `(dataset, bundle)->PublishResult` | STAC ingest + poll | +| `core/publishing/stac.py` | `build_collection`, `build_raster_item`, `build_vector_item` | see [STAC mapping](#stac-mapping) | STAC construction | +| `core/processors/publishing.py` | `PublishingProcessor.enqueue` / `.run` | `(request)->PublishedDataset` / `(dataset)->None` | Orchestration | + +## API Design + +### hastefuncapi endpoints + +Per `AGENTS.md`, these are thin wrappers in `function_app.py` delegating to +`PublishingProcessor` / registry in `hastegeo`. + +#### `GET /api/GetPublishingProviders` + +**Auth:** `func.AuthLevel.FUNCTION`. Returns providers for the dialog dropdown. + +**Response (200):** +```json +{ + "providers": [ + { "id": "local", "displayName": "Local (HASTE storage)", + "description": "string", "isConfigured": true, "supportsAsync": true, + "configRequirements": [] }, + { "id": "planetary_computer", "displayName": "Planetary Computer", + "isConfigured": false, + "configRequirements": [ + { "key": "geocatalog_url", "label": "GeoCatalog URL", "required": true, "secret": false } + ] } + ] +} +``` + +#### `GET /api/GetPublishedDatasets` + +**Auth:** `func.AuthLevel.FUNCTION`. Optional `projectId` (GUID) filter. + +**Response (200):** `{ "publishedDatasets": [PublishedDataset, ...] }` sorted by +`publishedDate` desc (mirrors `GetModelCatalog`). + +#### `GET /api/GetPublishedDataset` + +**Auth:** `func.AuthLevel.FUNCTION`. Params: `datasetId` (required). +**Response (200):** `{ "publishedDataset": PublishedDataset }`; **404** if absent. + +#### `POST /api/PutPublishDatasetQueueMessage` + +**Auth:** `func.AuthLevel.FUNCTION`. Starts a publish job (name follows the +existing `PutRunModelQueueMessage` / `PutArtifactsZipQueueMessage` convention). + +**Request:** +```json +{ + "projectId": "guid — required", + "imageLayerId": "string — required", + "modelId": "string — required (source of artifacts)", + "name": "string — required, user-edited, prefilled ' – '", + "description": "string — optional, prefilled from assessment report", + "target": "local | planetary_computer — required", + "artifacts": "string[] — required, ≥1; subset of the model's available kinds (gpkg | valid_mask | footprints | processed_cog | …). Defaults to all available if omitted.", + "providerConfig": "object — optional per-provider overrides (e.g. collectionId)" +} +``` + +**Behavior:** validate params → build `ArtifactBundle` and confirm every +requested kind is actually available (**400/404** otherwise) → resolve provider → +`provider.validate(req)` (400 on error) → create `PublishedDataset` +(`status=PENDING`, new `datasetId`, `artifacts`=selected, `publishedByUser` from +client principal) → upsert into the `index` doc → enqueue `{datasetId, projectId}` +on `publish-queue` → **202** with the record. + +**Error Responses:** + +| Code | Condition | +|---|---| +| 400 | Missing/invalid params, unknown target, empty `artifacts`, or `provider.validate` rejected | +| 401 | Missing/invalid function key or MSAL token | +| 404 | Project / layer / model not found, or a requested artifact kind is not available on the model | +| 409 | A dataset with the same name for this project/layer/target already publishing | +| 500 | Metadata write / enqueue failure | + +#### `DELETE /api/DeletePublishedDataset` + +**Auth:** `func.AuthLevel.FUNCTION`. Params: `datasetId` (required). Permission: +publisher or admin (client principal). Calls `provider.unpublish(dataset)` +(best-effort), removes the record. **Response (200):** `{ "deletedDataset": {...} }`. + +Existing `GET /api/GetAssessmentReport` (function_app.py:4045) is **reused** +unchanged for the description prefill — no new endpoint. + +### Queue message (hastefuncqueues) + +#### Queue: `publish-queue` (`PUBLISH_QUEUE_NAME`, default `publish-queue`) + +**Message Schema:** +```json +{ "datasetId": "string", "projectId": "string" } +``` + +**Trigger behavior (`GetPublishDatasetQueueMessage`):** +1. Load `PublishedDataset` from the `index` doc; set `status=IN_PROGRESS`, + append status message, persist. +2. Build `ArtifactBundle` from the model/layer documents. +3. `provider = registry.resolve(dataset.target)`; `result = provider.publish(...)`. +4. On success: merge `result.links` / `result.providerMetadata`, + `status=PUBLISHED`, `publishedDate=now`; persist. +5. On exception: `status=FAILED`, `statusMessage=`; persist; log. Message + visibility follows the existing immediate-processing convention + (`visibility_timeout=0`). + +## Behavior & Logic + +### Core flow (publish) + +1. Analyst opens a completed model's results → **Publish dataset…**. +2. Dialog loads: name prefilled `${project} – ${layer}`; description prefilled + from `GetAssessmentReport`; **asset checklist populated from the model's + available outputs (all prechecked)**; targets from `GetPublishingProviders`. +3. Analyst optionally trims the asset selection, then Submit → + `PutPublishDatasetQueueMessage` validates (incl. ≥1 available asset) + writes + `PENDING` record (with selected `artifacts`) + enqueues; dialog confirms and + closes. +4. Worker runs the provider; status → `IN_PROGRESS` → `PUBLISHED`/`FAILED`. +5. Published Datasets section lists the record; UI polls while `IN_PROGRESS`. + +### Local provider + +- Iterate the `ArtifactBundle` (already filtered to the **selected** kinds); for + each selected artifact, **copy** the blob to + `published/{hash(projectId)}/{datasetId}/{artifact_name}` (same container as + existing artifacts) using the artifact-storage layer; generate a fresh SAS URL. +- `PublishResult.links = { "": "", ... }`; also store the + assessment-report JSON snapshot for provenance. +- `unpublish`: delete the `published/{datasetId}/` prefix. +- Always **copies** into the immutable published prefix (lifecycle + independence); total copied bytes are bounded by `PUBLISH_MAX_TOTAL_BYTES`. + +### Planetary Computer provider — STAC mapping + +The PC provider builds STAC and ingests it into a Planetary Computer Pro +GeoCatalog over a small, hardened REST client (`geocatalog_client.py`) plus a +resumable transport adapter. The client and STAC builder are HASTE-owned and +transport-agnostic (no `azure-planetarycomputer` SDK dependency), so the +ingestion flow is fully unit-testable against HTTP fixtures. + +Libraries: `azure-identity`, `pystac[validation]`, `geopandas`, `pyogrio`, +`shapely`, `requests`. The damage products are vector, so item geometry is built +from the valid-area mask (`geopandas`/`shapely`), not from a COG — `rio-stac` is +not used in v1 (it would return only if/when rasterized COGs are published; see +the render limitation below). Auth: +`DefaultAzureCredential().get_token("https://geocatalog.spatio.azure.com/.default")`, +token cached with a ~300 s expiry skew; Bearer header; **all** calls carry +`?api-version=2026-04-15`. Redirects are never followed and every request carries +explicit (connect, read) timeouts; errors carry only the HTTP status, never the +response body (which may contain tokens/SAS). + +**GeoCatalog REST surface:** + +| Op | Method + path | Notes | +|---|---|---| +| Ensure collection | `GET /stac/collections/{id}` → 404 ⇒ `POST /stac/collections` else `PUT /stac/collections/{id}` | upsert | +| Ingest item | `POST /stac/collections/{id}/items` | 202 + `location` → poll | +| Replace item | `DELETE …/items/{itemId}` then re-POST | idempotent re-publish | +| Search / verify | `POST /stac/search` `{collections:[id]}` | post-publish validation | +| Configure | `PUT …/configurations/tile-settings`, `POST …/render-options`, `POST …/mosaics` | display config | +| Collection asset | `POST /stac/collections/{id}/assets` (multipart) | thumbnail | +| Sign published asset | `GET /sas/sign?href=` → signed asset URL | **assets live in SAS-protected managed storage** | +| Ingestion source | `GET/POST/DELETE /inma/ingestion-sources` | see below | + +**Collection (per event ≈ per project):** ensure it exists via the upsert above. +Built as a STAC `Collection` with `id`, `title`, `description`, `license`, +`keywords`, `providers`, `extent` (spatial bbox + temporal interval computed from +items), `summaries`, `item_assets` (declares `buildings` GPKG + `aoi` GeoJSON), +`links`, and `stac_extensions: [item-assets/v1.0.0]`. Scope is **one collection +per event**; a HASTE project maps to one event → `id` derived from the project +(slugified to GeoCatalog id rules — no `-_+().`), e.g. `haste-`. + +**Item (per response ≈ per published dataset/layer):** one STAC Item per +assessment response. +- **Geometry = the valid-area mask polygon** (the region actually assessed), + read via `geopandas`, unioned, reprojected to EPSG:4326; `bbox` and + `ai4g:aoi_area_km2` computed from it (better than a raster footprint for damage + products). +- **Assets:** `buildings` — damage GeoPackage (`application/geopackage+sqlite3`, + roles `[data]`, `proj:code` of the source CRS); `aoi` — valid mask GeoJSON + (`application/geo+json`, roles `[metadata]`). Only the analyst-**selected** + artifacts become assets. +- **Properties:** `title`, `description`, `datetime`, `license`, `proj:code`, and + HASTE stats under an `ai4g:`-style prefix (`buildings_total`, `buildings_cloud`, + `buildings_clear`, `buildings_damaged`, `damaged_pct_of_clear`), plus + `…validation_*` (precision/recall/extrapolated) from the assessment report and + `…merge_*` for merged products — sourced from `assessmentSummary`. +- `stac_extensions: [projection/v2.0.0]`; `item["collection"] = collection_id`; + **item id sanitized** to the GeoCatalog charset (no `-_+().`). + +**Ingest:** `POST …/items` (Item or ItemCollection) → **202** + `location`; poll +`location` (falls back to `/inma/operations/{id}`) until a terminal status +(`Succeeded`/`Finished`/`Failed`/`Cancelled`/`Completed`), and also check +`additionalInformation.TotalFailedItems` for partial failures. **The GeoCatalog +copies each asset from its HASTE blob href into its own managed storage and +rewrites the href** — so published assets are served from PC storage, not HASTE. +- **Public source containers** need no ingestion source. +- **Private source containers** must be registered first as a **`SasToken`** + ingestion source (`POST /inma/ingestion-sources` with `{kind:"SasToken", + connectionInfo:{containerUrl, sasToken}}`) — `SasToken` is the **only kind the + API accepts**; managed-identity sources are **portal/ARM-only** (grant the + catalog's user-assigned identity *Storage Blob Data Reader* on the HASTE storage + account). + +**Post-publish validation:** search the collection, confirm each item + required +assets exist, compare item geometry to the source mask, and Range-GET each asset +href (signed via `/sas/sign`) for reachability. + +`PublishResult.links = { "stac_collection": ".../stac/collections/{id}", +"explorer": "" }`; +`providerMetadata = { "collectionId": ..., "itemIds": [...], "apiVersion": ..., +"assetsCopiedToManagedStorage": true }`. + +`unpublish`: `DELETE /stac/collections/{id}/items/{itemId}` per item (best-effort; +collection retained if it still holds other datasets for the project). + +> **Render limitation (important).** Planetary Computer Pro cloud-optimizes and +> renders **raster** data only; **GeoPackage/GeoJSON are stored and served for +> download through the STAC API but are NOT tiled and NOT shown on the Explorer +> map.** So a PC-published damage dataset appears in the Explorer as **item +> footprints + metadata**, and consumers **download** the GeoPackage. Rendering +> damage *on the map* would require rasterizing predictions to COGs and adding +> render options — **out of scope for v1** (a future `processed_cog`/raster asset +> path can use `rio-stac`). The UX must set this expectation (see +> [ux-spec.md](ux-spec.md)); do not promise map rendering of the damage layer. + +#### Provider implementation notes + +The provider runs the ingestion as a **bounded-step state machine** so no single +queue invocation blocks on a long-running ingestion: + +- **Ensure collection:** `GET` the collection; on 404 create it. Collection + creation is **synchronous** (`201`); item ingestion is **asynchronous** (`202` + + `operation-location`). +- **Ingest item:** `POST …/items` → `202`; persist the returned operation URL as + a continuation token (origin-pinned to the GeoCatalog to prevent SSRF). +- **Poll:** each subsequent step polls the operation URL until a terminal status. + Terminal success states include `Succeeded`, `Finished`, `Completed`; also check + `additionalInformation.totalFailedItems` and fail on partial failures. Poll + attempts are bounded by config (`PC_VERIFY_ATTEMPTS`) so a stuck ingestion + eventually FAILs rather than looping forever. +- **Unpublish:** `DELETE …/items/{itemId}` per item; retain the collection while + it still holds other datasets for the project. + +### Edge Cases + +| Case | Expected Behavior | +|---|---| +| Model has no `gpkgUrl` / no publishable artifact | Publish action disabled (UI) + 404 (API) | +| User selects zero assets | Publish disabled (UI) + 400 (API) | +| Requested artifact kind not produced by the model | Grayed/omitted in the dialog; 404 if forced via API | +| Some selected assets missing at publish time (deleted/re-run) | Publish what remains; note skipped kinds in `statusMessage` | +| Assessment report unavailable | Description prefill left blank; publish still allowed | +| Duplicate dataset name (same project/layer/target, still publishing) | 409; UI shows "already publishing" | +| PC provider not configured | Target shown disabled; `validate` returns error if forced | +| GeoCatalog 40x on an item | Job → FAILED with the API error text; no partial links surfaced | +| Ingestion poll never terminates | Bounded poll (timeout) → FAILED "ingestion timed out"; safe to retry | +| Source model deleted after publish (Local, copy mode) | Links still resolve (immutable published copy) | +| Collection id collision across projects | Id is project-slug scoped; 409 → reuse via PUT | +| GeoCatalog item-id illegal chars | Sanitized before POST | + +### Error Handling + +| Error Condition | Response | Recovery | +|---|---|---| +| Blob copy timeout (Local) | status FAILED + message | Re-publish (idempotent on `datasetId`) | +| GeoCatalog auth failure | status FAILED "auth" | Fix credential/ingestion source; re-publish | +| GeoCatalog 409 collection exists | Treat as success (reuse) | PUT to update metadata | +| Ingestion partial failure (`TotalFailedItems>0`) | status FAILED + per-item detail | Replace-and-repost failed items | +| Private HASTE container, no ingestion source | Ingestion can't read assets → FAILED | Register `SasToken` source (or grant MI reader) then re-publish | +| Queue message poison | Standard Azure Queue retry → dead-letter | Admin inspects; status stays IN_PROGRESS until re-run | + +## Configuration + +| Config Key | Type | Default | Where Set | Description | +|---|---|---|---|---| +| `PUBLISHING_ENABLED` | bool | `false` | App Settings | Master feature flag (section + Publish action) | +| `PUBLISH_QUEUE_NAME` | str | `publish-queue` | `config.py` / App Settings / `docker-compose.yml` | Publish job queue (auto-created at runtime) | +| `PC_PROVIDER_ENABLED` | bool | `false` | App Settings | Register/expose the Planetary Computer provider | +| `PC_GEOCATALOG_URL` | url | (unset) | App Settings | MPC Pro GeoCatalog base URL (no trailing `/`) | +| `PC_EXPLORER_URL` | url | (unset) | App Settings | Explorer base URL for published-dataset links | +| `PC_INGESTION_SOURCE` | str | (unset) | App Settings | Only for **private** HASTE containers (`SasToken` source); public need none | +| `PC_COLLECTION_PREFIX` | str | `haste-` | App Settings | Collection id prefix (one collection per project/event) | +| `PUBLISH_MAX_TOTAL_BYTES` | int | 5 GiB | App Settings (override) | Max total published bytes per dataset | +| `PUBLISHED_DOWNLOAD_SAS_MINUTES` | int | `15` | App Settings (override) | Local retrieval SAS TTL | +| `PUBLISHING_LOCK_CONTAINER` | str | `publishing-locks` | App Settings (override) | Blob-lease container (auto-created at runtime) | +| `PC_VERIFY_ATTEMPTS` | int | code default | App Settings (override) | Ingestion poll attempt bound | +| `VITE_*` | — | — | `ui/.env.*` | None new; UI uses existing `api.js` transport | + +The STAC `api-version` (`2026-04-15`) and the Entra token scope +(`https://geocatalog.spatio.azure.com/.default`) are **code constants** in the +GeoCatalog client, not App Settings. For the deploy-time `azd` env var → App +Setting mapping and the enablement steps, see +[rollout.md](rollout.md#operator-configuration-app-settings). + +Credentials use **managed identity** (`DefaultAzureCredential`) in Azure and +`AzureCliCredential`/env locally — no secrets in code (per +`docs/security-configuration.md`). + +**Provider configuration is operator-owned (v1).** The keys above are Azure +Function App Settings set at deploy time by an admin/operator, plus a +pre-registered GeoCatalog ingestion source; there is **no in-app admin screen** +for configuring providers in v1. The app never stores provider credentials +(managed identity only) and never accepts them from the UI. The Publish dialog +only *reflects* the resulting state — `GetPublishingProviders` reports +`isConfigured` (and `configRequirements` purely to explain *why* a provider is +disabled). A self-service admin UI is a possible future addition behind the same +`ProviderInfo` contract (no UI/API rework needed) — see +[user-stories.md](user-stories.md#out-of-scope). + +## Observability + +- **UI:** per-row status chips; dialog error banner from `provider.validate`. +- **Backend:** structured logging in `PublishingProcessor` and each provider + (`Publishing dataset {id} via {provider} …`, `Ingestion status: {status}`); + `statusMessage` on the record surfaces failures (mirrors `Model.statusMessage`). +- **Queue depth:** `publish-queue` monitored like other queues. + +## Open Questions + +- [ ] Collection granularity: one STAC Collection **per project** (default) vs + per event vs per dataset — start per-project, revisit. +- [ ] Should Local publishing register a lightweight **self-hosted STAC** record + too (so both providers are STAC-shaped)? Interface allows it. +- [ ] Hard backend RBAC for unpublish vs UI-gated (v1 checks client principal in + the API) — align with existing admin checks. +- [ ] Retention/GC of `published/{datasetId}/` copies when a project is deleted. diff --git a/spec/features/data-publishing/impact-analysis.md b/spec/features/data-publishing/impact-analysis.md new file mode 100644 index 00000000..73a3af31 --- /dev/null +++ b/spec/features/data-publishing/impact-analysis.md @@ -0,0 +1,121 @@ +# Impact Analysis: Data Publishing & Published Datasets + +## Scope of Change + +### HASTE Components Affected + +| Component | Path | Type of Change | Severity | +|---|---|---|---| +| Core models | `hastelib/src/hastegeo/core/models/publishing.py` | new | low | +| Publishing package | `hastelib/src/hastegeo/core/publishing/` | new | medium | +| Publishing processor | `hastelib/src/hastegeo/core/processors/publishing.py` | new | medium | +| Config | `hastelib/src/hastegeo/core/config.py` | modified (enum, queue, PC block) | low | +| REST API | `api/hastefuncapi/function_app.py` | new (5 routes) | low | +| Queue workers | `api/hastefuncqueues/function_app.py` | new (1 trigger) | medium | +| React UI | `ui/src/Components/PublishedDatasets*.jsx`, `PublishDatasetModal.jsx`, `ModelResultsButton.jsx`, `AppSidebar.jsx`, `AppBody.jsx` | new + modified | medium | +| Docker config | `docker/docker-compose.yml`, Azurite seed | modified | low | +| CI/CD | `.github/workflows/*` (Component Governance) | modified | low | + +> Existing `Model`/`ImageLayer`/`Project` documents and their endpoints are +> **read-only** inputs — no changes, so no regression surface there. + +## Azure Service Impact + +| Service | Change | New Cost Impact | +|---|---|---| +| Cosmos DB | New logical type `PUBLISHED_DATASETS` (single `index` doc, like model catalog) | negligible | +| Blob Storage | New `published/{datasetId}/` prefix in the existing container (copy-on-publish) | storage ∝ published artifact size (duplicates source until unpublish) | +| Queue Storage | New `publish-queue` | negligible | +| Azure Functions | +5 HTTP routes, +1 queue trigger | low consumption | +| Static Web Apps | New `/published-datasets` route + `/api/*` proxy entries | none material | +| **Planetary Computer Pro** | New external dependency (GeoCatalog + ingestion source) | **GeoCatalog resource + ingest/storage costs** (PC target only) | +| Managed Identity | GeoCatalog + storage access for the queue worker | none | + +## Dependency Analysis + +### Upstream Dependencies (things this feature needs) + +| Dependency | Type | Status | Risk if Unavailable | +|---|---|---|---| +| `hastegeo` artifact storage + metadata layer | library | available | Local publish cannot copy/register | +| Completed model with artifacts (`gpkgUrl`, masks, COGs) | data | per-project | Nothing publishable → action disabled | +| `GetAssessmentReport` endpoint | API | available (`function_app.py:4045`) | Description prefill blank (non-blocking) | +| `azure-identity`, `pystac`, `geopandas`, `pyogrio`, `shapely` | pip | to add | PC provider unbuildable; Local unaffected | +| MPC Pro GeoCatalog + ingestion source | infra/external | to provision | PC target disabled; Local unaffected | +| Managed identity / `DefaultAzureCredential` | auth | available | PC ingestion auth fails | + +### Downstream Impact (things affected by this feature) + +| Consumer | How Affected | Breaking? | Migration Needed? | +|---|---|---|---| +| `hastefuncapi` callers | New endpoints only; existing untouched | no | no | +| React UI | New section + one added menu item | no | no | +| Docker Compose stack | +1 queue seed | no | no | +| Existing Cosmos documents | None read-modified | no | no | +| Model deletion flow | Local copy mode makes published data survive model deletion (intended) | no | no | + +## Risk Assessment + +| Risk | Likelihood | Impact | Mitigation | Owner | +|---|---|---|---|---| +| PC GeoCatalog API/version drift (`api-version=2026-04-15`) | medium | medium | Version pinned in config; provider isolates all HTTP; contract tests against mock | `gis` | +| Ingestion source misconfig → GeoCatalog can't read HASTE blobs | medium | medium | `validate()` pre-checks config; clear FAILED `statusMessage`; runbook in rollout | `backend-dev` | +| Copy-on-publish duplicates large artifacts → storage growth | medium | low | `PUBLISH_MAX_TOTAL_BYTES` cap per dataset; GC on unpublish/project delete | `backend-dev` | +| STAC item-id / char rules rejected by GeoCatalog | medium | low | Sanitizer + `pystac.validate` before POST | `gis` | +| Long ingestion polling ties up a queue worker | low | medium | Bounded poll timeout → FAILED; idempotent re-publish | `backend-dev` | +| Credential leakage | low | high | Managed identity only; no secrets in code/logs; `security` review | `security` | +| Unpublish deletes shared PC collection | low | medium | Delete items, retain collection unless empty/owned | `gis` | + +## Performance Impact + +- **API latency:** `PutPublishDatasetQueueMessage` returns after a metadata + upsert + enqueue (fast, like `PutRunModelQueueMessage`); no synchronous + provider work. List/get mirror `GetModelCatalog` cost. +- **Queue throughput:** publish jobs are infrequent and I/O-bound; a slow PC + poll occupies one worker for the ingestion duration — bounded by timeout. +- **Tile serving:** unaffected (`titilerfuncapi` not involved). +- **Storage I/O:** copy-on-publish reads+writes artifact-sized blobs once per + publish. + +## Security Impact + +- [x] New API endpoints exposed? Yes — 5 routes at `func.AuthLevel.FUNCTION`, + same posture as existing catalog/artifact routes; unpublish checks client + principal. +- [x] New data classification handled? Published damage outputs (already handled + class); no new PII. Publishing to PC **exports** data externally — gated by + provider config + the publish action's project access. +- [x] MSAL/Entra ID auth changes? PC uses **managed identity** + (`DefaultAzureCredential`, audience `https://geocatalog.spatio.azure.com`) — + no new user-facing auth. +- [x] New secrets or connection strings? No secrets in code; GeoCatalog URL + + ingestion source name are non-secret app settings; auth via managed identity. +- [ ] CORS changes in SWA? None (same-origin `/api/*`). +- [ ] New federated credentials? None beyond existing deploy OIDC. + +## Compliance & Data Impact + +- [x] Geospatial data sovereignty: publishing to PC egresses imagery-derived + products to a GeoCatalog region — operators must ensure region/partner + terms permit it (surface in provider config/docs). +- [x] Partner data sharing agreements: publishing derived products may be + governed by imagery source terms — publisher responsibility; note in docs. +- [x] Data retention: `published/{datasetId}/` copies persist until unpublish or + project delete — add GC (open question). +- [x] Audit logging: log publisher, target, dataset, status transitions. +- [x] Component Governance: new Python deps (`azure-identity`, `pystac`, + `geopandas`, `pyogrio`, `shapely`) scanned in CI. `geopandas`/`pyogrio` + overlap existing GDAL-stack deps — verify no new native surface. + +## Rollback Assessment + +- **Reversibility:** fully reversible. +- **Cosmos data:** `PUBLISHED_DATASETS` `index` doc is additive; reverting code + leaves it inert (unknown type ignored). Optional delete. +- **Blob data:** `published/*` prefixes are inert extra blobs; optional cleanup. + PC collections/items persist in the GeoCatalog until deleted via its API + (out-of-band, documented). +- **API:** all new endpoints are additive; existing endpoints unchanged → + backward-compatible. +- **Estimated rollback time:** < 15 min (revert deploy); external PC cleanup is + best-effort and asynchronous. diff --git a/spec/features/data-publishing/plan.md b/spec/features/data-publishing/plan.md new file mode 100644 index 00000000..5aa824b4 --- /dev/null +++ b/spec/features/data-publishing/plan.md @@ -0,0 +1,131 @@ +# Execution Plan: Data Publishing & Published Datasets + +## Phases + +### Phase 1: Core Library — models, provider interface, Local, STAC + +**Goal:** Implement the publishing domain in `hastelib/src/hastegeo/` independent +of API/UI. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| Add `models/publishing.py` (`PublishedDataset`, `PublishRequest`, `PublishTarget`/`PublishStatus`, `ProviderInfo`, `PublishResult`) | `backend-dev` | — | US-001..006 | not-started | +| Add `MetadataTypes.PUBLISHED_DATASETS` + `publish_queue_name` + PC config block in `config.py` | `backend-dev` | — | US-003/004/006 | not-started | +| Add `publishing/base.py` (`PublishingProvider` ABC, `ProviderConfigField`) + `registry.py` | `backend-dev` | models | US-006 | not-started | +| Implement `publishing/local_provider.py` (copy → `published/{datasetId}/`, links) | `backend-dev` | base, artifact storage | US-003 | not-started | +| Add `publishing/geocatalog_client.py` (hardened REST client + Entra auth) and `planetary_computer_transport.py` (resumable async-ingestion adapter) | `backend-dev` | — | US-004 | not-started | +| Implement `publishing/stac.py` (collection + vector item builders; geometry from valid-area mask, `ai4g:` stats) | `gis` | models | US-004 | not-started | +| Add `processors/publishing.py` (`enqueue`, `run`, `ArtifactBundle`) | `backend-dev` | providers, registry | US-001/003 | not-started | +| Unit tests in `hastelib/tests/core/{models,publishing,processors}/` | `backend-dev` | all above | US-001/003/006 | not-started | + +**Exit Criteria:** +- [ ] Local provider publishes a fixture bundle end-to-end in a unit test +- [ ] Registry lists provider infos; STAC builders produce valid `pystac`-validated docs +- [ ] Core logic works with no API/UI present + +### Phase 2: API Layer + Queue — enqueue, list, get, delete + +**Goal:** Expose publishing via `hastefuncapi` routes and the `publish-queue` +worker. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| Add `GetPublishingProviders`, `GetPublishedDatasets`, `GetPublishedDataset`, `PutPublishDatasetQueueMessage`, `DeletePublishedDataset` (thin wrappers) | `backend-dev` | Phase 1 | US-001/002/005/006 | not-started | +| Add `GetPublishDatasetQueueMessage` trigger in `hastefuncqueues` | `backend-dev` | Phase 1 | US-001/003 | not-started | +| Seed `publish-queue` in Azurite / `docker-compose.yml` | `backend-dev` | — | — | not-started | +| Update `requirements.txt` (`azure-identity`, `pystac[validation]`, `geopandas`, `pyogrio`, `shapely`, `requests`) | `backend-dev` | — | — | not-started | +| API integration + queue-worker tests | `backend-dev` | above | US-001/002/003 | not-started | + +**Exit Criteria:** +- [ ] Publish → PENDING record + queued message; worker drives Local to PUBLISHED +- [ ] List/get/delete callable via REST; 400/404/409 paths covered +- [ ] Works in Docker Compose local stack + +### Phase 3: Planetary Computer provider + +**Goal:** Ingest STAC into a MPC Pro GeoCatalog. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| Implement `planetary_computer_provider.py` (ensure-collection, build+ingest items, poll operations, `TotalFailedItems`, validate, unpublish) | `gis` | Phase 1 stac/client, Phase 2 | US-004 | not-started | +| Item-id sanitization + media types + damage properties from assessment summary | `gis` | stac | US-004 | not-started | +| Credential/ingestion-source handling (`DefaultAzureCredential`, `inma/ingestion-sources`) | `backend-dev` | provider | US-004 | not-started | +| Security review of new deps + credential flow | `security` | requirements | US-004 | not-started | +| Provider tests (mock GeoCatalog HTTP: 201/202/40x, poll states) | `backend-dev` | provider | US-004 | not-started | + +**Exit Criteria:** +- [ ] Against a mock/dev GeoCatalog: collection ensured, items ingested, status polled to terminal, links stored +- [ ] Failure paths (40x, timeout) → FAILED with message; unpublish deletes items +- [ ] `security`/`security-validation` sign-off on deps + credentials + +### Phase 4: UI — section, dialog, entry point + +**Goal:** Surface publishing in the React app. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| `PublishDatasetModal.jsx` (name/description/target, prefill, submit) | `ui` | Phase 2 | US-001/006 | not-started | +| "Publish dataset…" `MenuItem` in `ModelResultsButton.jsx` | `ui` | modal | US-001 | not-started | +| Extract `util/assessmentSummary.js` (`buildSummarySentence`) from `AssessmentReportModal.jsx` | `ui` | — | US-001 | not-started | +| `PublishedDatasets.jsx` + `PublishedDatasetRow.jsx` (catalog-style, all states, polling) | `ui` | Phase 2 | US-002/005 | not-started | +| Sidebar item + `/published-datasets` route (`AppSidebar.jsx`, `AppBody.jsx`) | `ui` | section | US-002 | not-started | +| API helpers in `util/api.js`; UI component tests | `ui` | above | US-001/002/005 | not-started | + +**Exit Criteria:** +- [ ] Publish flow works from results menu; datasets list with empty/loading/in-progress/success/failure states +- [ ] Works with SWA CLI local dev (`swa start`) against Docker Compose backend + +### Phase 5: Integration & Deployment + +**Goal:** Validate end-to-end and ship. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| E2E via Docker Compose (Local target full loop) | `backend-dev` | Phase 4 | US-001/002/003 | not-started | +| PC target E2E against a real/dev GeoCatalog (gated) | `gis` | Phase 3/4 | US-004 | not-started | +| Update `docs/` (publishing feature) + `CHANGELOG.md` | `backend-dev` | — | — | not-started | +| GitHub Actions: Component Governance for new deps | `backend-dev` | — | — | not-started | + +**Exit Criteria:** +- [ ] `docker compose up` clean with Local publishing working end-to-end +- [ ] CI passes (secret-scan, deploy-apps, Component Governance) +- [ ] Docs + changelog updated + +## Milestones + +| Milestone | Date | Deliverable | +|---|---|---| +| Spec approved | | Signed-off design docs (this folder) | +| Core + Local done | | `hastelib` publishing package, Local provider, tests | +| API + Queue done | | Endpoints + `publish-queue` worker functional | +| PC provider done | | STAC ingestion into GeoCatalog | +| UI done | | Section + dialog + entry point in the app | +| Release | | Deployed to production SWA behind flag | + +## Agent Summary + +| Agent | Tasks Owned | Phases | +|---|---|---| +| `backend-dev` | models, config, base/registry, Local, processor, API, queue, integration | 1, 2, 3, 5 | +| `gis` | STAC builders, Planetary Computer provider, PC E2E | 1, 3, 5 | +| `ui` | dialog, section, entry point, routing, UI tests | 4 | +| `security` | new-dep + credential review | 3 | + +## Resource Requirements + +- **Agents:** `backend-dev`, `gis`, `ui`, `security` (+ validation counterparts). +- **Azure services:** `publish-queue` (Queue Storage); artifact/data Blob prefix + `published/`; a **MPC Pro GeoCatalog** + registered ingestion source for the PC + target (dev + prod); managed identity with GeoCatalog + storage access. +- **New Python deps:** `azure-identity`, `pystac[validation]`, `geopandas`, `pyogrio`, `shapely`, `requests` (Component + Governance). +- **GPU compute:** none. +- **External data:** none (operates on HASTE-generated artifacts). + +## Open Questions + +- [ ] Is a dev/test GeoCatalog available for CI, or is PC E2E manual/gated? +- [ ] Feature flag scope: gate the whole section, or just the PC target, until + GeoCatalog is provisioned? (see [rollout.md](rollout.md)) +- [ ] Collection-per-project vs per-event id strategy (also in design Open + Questions). diff --git a/spec/features/data-publishing/rollout.md b/spec/features/data-publishing/rollout.md new file mode 100644 index 00000000..70653310 --- /dev/null +++ b/spec/features/data-publishing/rollout.md @@ -0,0 +1,179 @@ +# Rollout Plan: Data Publishing & Published Datasets + +## Rollout Strategy + +**Type:** feature-flag (phased) +**Target date:** TBD + +Two flags allow shipping Local first and gating Planetary Computer until a +GeoCatalog is provisioned: +- `PUBLISHING_ENABLED` — the whole section + Publish action. +- `PC_PROVIDER_ENABLED` — the Planetary Computer target only. + +## Deployment Targets + +| Component | Deployment Method | Target | +|---|---|---| +| `hastelib` (publishing package) | pip install / Docker rebuild | All Function Apps | +| `hastefuncapi` (5 routes) | GitHub Actions `deploy-apps.yml` | Azure Functions | +| `hastefuncqueues` (publish trigger) | GitHub Actions `deploy-apps.yml` | Azure Functions | +| React UI (section + dialog) | GitHub Actions `deploy-apps.yml` | Azure Static Web Apps | +| `publish-queue` | Azurite (local) / Storage (cloud) | Queue Storage | + +## Feature Flags + +| Flag Name | Location | Default | Description | Kill Switch? | +|---|---|---|---|---| +| `PUBLISHING_ENABLED` | `hastefuncapi` app setting + UI env (`GetPublishingProviders` gates targets) | off | Enables the Published Datasets section + Publish action | yes | +| `PC_PROVIDER_ENABLED` | `hastefuncapi` app setting | off | Registers/exposes the Planetary Computer provider | yes | + +When `PC_PROVIDER_ENABLED=off` or GeoCatalog config is absent, the provider +reports `isConfigured=false` and the UI shows it disabled — a natural kill switch +that never breaks Local publishing. + +## Operator Configuration (App Settings) + +Provider configuration is **operator-owned**: set at deploy time as Azure +Function App settings (no in-app admin screen). The Bicep threads each setting +from an `azd` environment variable (`azd env set `) into the shared +api + queues app settings; credentials are **managed identity only** (nothing +secret is entered or stored by the app). + +### Local target + +| `azd` env var | App Setting | Default | Purpose | +|---|---|---|---| +| `HASTE_PUBLISHING_ENABLED` | `PUBLISHING_ENABLED` | `false` | Master flag: Published Datasets section + Publish action | +| — | `PUBLISH_QUEUE_NAME` | `publish-queue` | Publish job queue (auto-created at runtime) | + +Enabling Local publishing needs only `HASTE_PUBLISHING_ENABLED=true`. The +`publish-queue` and the `publishing-locks` blob container are auto-created on +first use, so no queue/container resources are provisioned. The remaining Local +knobs use code defaults and are only set to override them (directly on the +Function App): `PUBLISH_MAX_TOTAL_BYTES` (5 GiB), `PUBLISHED_DOWNLOAD_SAS_MINUTES` +(15), `PUBLISHING_LOCK_CONTAINER` (`publishing-locks`). + +### Planetary Computer target + +| `azd` env var | App Setting | Default | Purpose | +|---|---|---|---| +| `HASTE_PC_PROVIDER_ENABLED` | `PC_PROVIDER_ENABLED` | `false` | Register/expose the PC provider | +| `HASTE_PC_GEOCATALOG_URL` | `PC_GEOCATALOG_URL` | (unset) | MPC Pro GeoCatalog base URL (no trailing `/`) | +| `HASTE_PC_EXPLORER_URL` | `PC_EXPLORER_URL` | (unset) | Explorer base URL for published-dataset links | +| `HASTE_PC_INGESTION_SOURCE` | `PC_INGESTION_SOURCE` | (unset) | Ingestion-source name for **private** HASTE containers (`SasToken`); unset for public | +| `HASTE_PC_COLLECTION_PREFIX` | `PC_COLLECTION_PREFIX` | `haste-` | STAC Collection id prefix (one per project/event) | + +The STAC `api-version` and the Entra token scope +(`https://geocatalog.spatio.azure.com/.default`) are **code constants**, not +settings. `PC_VERIFY_ATTEMPTS` (ingestion poll bound) uses a code default and is +set directly on the Function App only to override it. + +### Operator-owned GeoCatalog side (out-of-band) + +The GeoCatalog is provisioned and owned by the operator (external to this +template). Two grants are required and are **not** created by the app deploy: + +1. **Function app identity → GeoCatalog data plane** — the api/queues managed + identity needs a GeoCatalog RBAC role on the GeoCatalog resource so it can + call the STAC/ingestion APIs. Assigned on the operator's GeoCatalog resource; + verify the exact role against the target catalog. +2. **GeoCatalog ingestion → HASTE storage** — to ingest published assets the + GeoCatalog reads them from HASTE blob storage. Either register a `SasToken` + ingestion source (`HASTE_PC_INGESTION_SOURCE`, no role needed), **or** grant + the GeoCatalog's managed identity *Storage Blob Data Reader* on the HASTE + storage account by setting `HASTE_PC_GEOCATALOG_INGEST_PRINCIPAL_ID` to that + identity's object id (the deploy then makes the assignment; empty = skip). + +## Rollout Phases + +### Phase 1: Dev1 Environment — [date] + +- **Target:** SWA `dev1`. +- **Scope:** `PUBLISHING_ENABLED=on`, `PC_PROVIDER_ENABLED=off` (Local only). +- **Duration:** until Local E2E is stable. +- **Deployment:** + 1. Merge PR to `main` (triggers `deploy-apps.yml`). + 2. Seed `publish-queue`; verify in dev1 SWA. +- **Success criteria:** + - [ ] Publish (Local) → PENDING → PUBLISHED; artifacts downloadable + - [ ] Section renders all states; queue worker processes messages + - [ ] Docker Compose stack works +- **Rollback trigger:** publish failures, queue backlog, or section errors. + +### Phase 2: Testing Environment — [date] + +- **Target:** SWA `testing`; provision a **dev GeoCatalog** + ingestion source. +- **Scope:** enable `PC_PROVIDER_ENABLED=on`. Prereq: a GeoCatalog + user-assigned MI with *Storage Blob Data Reader* on HASTE storage (or a `SasToken` ingestion source for private containers). +- **Duration:** until PC E2E passes. +- **Success criteria:** + - [ ] PC target ingests collection/items; explorer links resolve + - [ ] Failure/timeout paths → FAILED with message + - [ ] Performance thresholds met; no Cosmos/index corruption +- **Rollback trigger:** ingestion auth failures, credential issues, data egress concerns. + +### Phase 3: Production — [date] + +- **Target:** Production SWA + Function Apps. +- **Federated credentials:** `fed-cred-main.json` (GitHub Actions OIDC); managed + identity for GeoCatalog + storage. +- **Scope:** `PUBLISHING_ENABLED=on`; `PC_PROVIDER_ENABLED` on only where a + production GeoCatalog + data-egress approval exist. +- **Success criteria:** + - [ ] All health checks green; error rate stable + - [ ] Publisher feedback positive +- **Feature flag cleanup:** remove `PUBLISHING_ENABLED` once GA; keep + `PC_PROVIDER_ENABLED` as an operational toggle. + +## Rollback Plan + +| Step | Action | Owner | ETA | +|---|---|---|---| +| 1 | Set `PUBLISHING_ENABLED=off` (and/or `PC_PROVIDER_ENABLED=off`) | ops | immediate | +| 2 | Revert PR / deploy previous commit | `backend-dev` | < 15 min | +| 3 | Verify `PUBLISHED_DATASETS` index doc intact (inert if reverted) | `backend-dev` | | +| 4 | Verify `publish-queue` drained / paused | `backend-dev` | | +| 5 | Verify UI fallback (section hidden, results menu unchanged) | `ui` | | + +**Cosmos data rollback required?** no — additive `index` doc is inert when the +feature is off. +**Blob artifacts cleanup needed?** optional — `published/*` copies are inert; +delete if reclaiming storage. **PC collections/items** persist in the GeoCatalog +until deleted via its STAC API (out-of-band, best-effort). + +## Monitoring & Alerting + +### Key Metrics to Watch + +| Metric | Source | Baseline | Alert Threshold | +|---|---|---|---| +| Publish success rate | queue worker logs / status transitions | — | < 90% over 1h | +| `publish-queue` depth | Azure Queue Storage metrics | ~0 | > 20 sustained | +| PC ingestion poll duration | provider logs | — | p95 > timeout | +| API error rate (publishing routes) | Azure Functions metrics | — | > 2% | +| `published/` storage growth | Storage metrics | — | unexpected spike | + +### Alerts to Configure + +| Alert | Condition | Severity | Notify | +|---|---|---|---| +| Publish failures spike | success rate < 90% / 1h | P2 | eng on-call | +| Queue backlog | depth > 20 sustained 15m | P2 | eng on-call | +| PC auth/ingestion failures | repeated FAILED with auth/ingest errors | P2 | eng + ops | + +## Communication Plan + +| Audience | Channel | When | Message | +|---|---|---|---| +| Engineering team | GitHub PR / Teams | Pre-deploy | Deployment plan + flags | +| Disaster analysts | Release notes | Post-deploy (Local) | "Publish finished datasets from model results" | +| Partners / operators | Docs | At PC GA | Provider config + data-egress guidance | + +## Post-Rollout Checklist + +- [ ] `PUBLISHING_ENABLED` flag cleaned up (PC flag retained as toggle) +- [ ] Temporary monitoring removed +- [ ] `docs/` updated (publishing feature + provider config) +- [ ] GitHub Pages docs rebuilt (`docs-deploy.yml`) +- [ ] Docker Compose stack verified with `publish-queue` +- [ ] `CHANGELOG.md` updated +- [ ] Retrospective scheduled diff --git a/spec/features/data-publishing/test-plan.md b/spec/features/data-publishing/test-plan.md new file mode 100644 index 00000000..bc6ba27e --- /dev/null +++ b/spec/features/data-publishing/test-plan.md @@ -0,0 +1,135 @@ +# Test Plan: Data Publishing & Published Datasets + +## Test Strategy + +| Level | Scope | Tool/Framework | Coverage Target | +|---|---|---|---| +| Unit | Models, provider ABC/registry, Local provider, STAC builders, processor | `unittest`/pytest (`hastelib/tests/`) | ≥ 85% of `core/publishing/` | +| Integration | 5 API routes + queue worker | pytest + Azure Functions harness / Azurite | All routes + happy/error paths | +| Provider (contract) | Planetary Computer HTTP against a mock GeoCatalog | pytest + `responses`/mock | 201/202/40x + poll states | +| UI | Dialog + section components/states | Vitest + React Testing Library | All states in [ux-spec.md](ux-spec.md#ui-states-all) | +| E2E | Full stack (Local target) via Docker Compose | Docker Compose + manual/Playwright | US-001/002/003 | +| Performance | Publish enqueue + list at volume | custom scripts | thresholds below | + +## Test Scenarios + +### Unit Tests (`hastelib/tests/`) + +| ID | Module | Scenario | Input | Expected Output | Story Ref | +|---|---|---|---|---|---| +| UT-001 | `core/models/publishing` | Serialize/deserialize `PublishedDataset`; enum validation | dict | round-trips; bad `target`/`status` rejected | US-001 | +| UT-002 | `core/publishing/registry` | Register + resolve + list_infos; unknown id | ids | providers listed; `KeyError`/error on unknown | US-006 | +| UT-003 | `core/publishing/base` | Provider `validate` contract (Local) | request | None when valid; message when not | US-001 | +| UT-004 | `core/publishing/local_provider` | Copy bundle → `published/{id}/`; links returned | fixture bundle | blobs copied; PUBLISHED links; unpublish cleans up | US-003 | +| UT-005 | `core/publishing/local_provider` | Total selected bytes exceed `PUBLISH_MAX_TOTAL_BYTES` | oversized bundle | publish rejected before copy; clear error | US-003 | +| UT-006 | `core/publishing/stac` | Item geometry from valid-area mask (union → EPSG:4326, bbox, area_km2); `pystac` validates | mask GeoJSON | valid item, `projection` ext | US-004 | +| UT-007 | `core/publishing/stac` | Vector item (GPKG) geometry/bbox/media types + damage props | GPKG + summary | valid item; correct assets/props | US-004 | +| UT-008 | `core/publishing/stac` | Collection build + item-id sanitization (no `-_+().`) | project | valid collection; safe ids | US-004 | +| UT-009 | `core/processors/publishing` | `enqueue` writes PENDING + queues; `run` drives Local to PUBLISHED | request | record transitions; message enqueued | US-001/003 | +| UT-010 | `core/processors/publishing` | Provider exception → FAILED + statusMessage | failing provider | status FAILED, message set | US-004 | + +### API Integration Tests + +| ID | Endpoint | Method | Scenario | Preconditions | Expected Response | Story Ref | +|---|---|---|---|---|---|---| +| IT-001 | `/api/GetPublishingProviders` | GET | List providers + isConfigured | PC unset | 200; local configured, PC disabled | US-006 | +| IT-002 | `/api/PutPublishDatasetQueueMessage` | POST | Valid publish (Local) | completed model | 202 + PENDING record; message queued | US-001 | +| IT-003 | `/api/PutPublishDatasetQueueMessage` | POST | Missing/invalid fields | — | 400 | US-001 | +| IT-004 | `/api/PutPublishDatasetQueueMessage` | POST | Model without artifacts | model no gpkg | 404 | US-001 | +| IT-005 | `/api/PutPublishDatasetQueueMessage` | POST | Duplicate name still publishing | in-progress dup | 409 | US-001 | +| IT-006 | `/api/GetPublishedDatasets` | GET | List (optional projectId) sorted desc | ≥1 published | 200 + array | US-002 | +| IT-007 | `/api/GetPublishedDataset` | GET | Fetch one; missing id | — | 200 / 404 | US-005 | +| IT-008 | `/api/DeletePublishedDataset` | DELETE | Publisher/admin unpublish; non-owner | records | 200 / 403 | US-005 | + +### Queue Worker Tests + +| ID | Queue | Scenario | Message | Expected Side Effect | Story Ref | +|---|---|---|---|---|---| +| QT-001 | `publish-queue` | Local publish end-to-end | `{datasetId,projectId}` | artifacts copied; record PUBLISHED + links | US-003 | +| QT-002 | `publish-queue` | PC publish (mock GeoCatalog) | `{...}` | collection upserted; item geometry = valid-area mask; ingested; operations polled; `TotalFailedItems=0`; PUBLISHED | US-004 | +| QT-003 | `publish-queue` | Provider raises | `{...}` | record FAILED + statusMessage; logged | US-004 | +| QT-004 | `publish-queue` | Malformed / unknown datasetId | `{...}` | error log; no crash | — | + +### UI Component Tests + +| ID | Component | Scenario | User Action | Expected Behavior | Story Ref | +|---|---|---|---|---|---| +| UI-001 | `ModelResultsButton` | Publish item enabled/disabled | open menu | enabled w/ gpkg; disabled+tooltip otherwise | US-001 | +| UI-002 | `PublishDatasetModal` | Prefill name + description + targets | open dialog | name ` – `; desc from report; dropdown from API | US-001 | +| UI-003 | `PublishDatasetModal` | Validation + submit + 409 | submit | field/banner errors; success closes; 409 banner | US-001 | +| UI-004 | `PublishDatasetModal` | Unconfigured PC target | open dropdown | PC disabled w/ note | US-006 | +| UI-005 | `PublishedDatasets` | Empty / loading / no-results | mount/search | empty state; overlay spinner; `NoResultsMessage` | US-002 | +| UI-006 | `PublishedDatasetRow` | Status chips + polling + actions | render | in-progress polls; success enables retrieve; failed shows message/Retry | US-002/005 | +| UI-007 | `PublishDatasetModal` | Asset checklist: available prechecked, absent grayed, zero-selected blocks Publish | toggle checkboxes | correct list; Publish disabled at zero selected | US-007 | + +### End-to-End Tests (Docker Compose) + +| ID | User Flow | Steps | Expected Outcome | Story Ref | +|---|---|---|---|---| +| E2E-001 | Publish to Local | 1. `docker compose up` 2. Complete a model 3. Publish dataset (Local) 4. Open Published Datasets | Row PUBLISHED; artifacts downloadable | US-001/003 | +| E2E-002 | In-progress → success | Publish; watch section | Row transitions IN_PROGRESS → PUBLISHED without reload | US-002 | +| E2E-003 | Unpublish | Unpublish a Local dataset | Row removed; `published/{id}/` cleaned | US-005 | +| E2E-004 | Publish to PC (gated) | Configure GeoCatalog + (private) SasToken source; publish PC target | Collection/items ingested; assets copied to managed storage; Explorer shows footprints+metadata; GeoPackage downloadable via collection SAS | US-004 | + +### Edge Case & Negative Tests + +| ID | Scenario | Input | Expected Behavior | +|---|---|---|---| +| NEG-001 | Unauthenticated API request | no key | 401 | +| NEG-002 | Non-existent datasetId | random id | 404 | +| NEG-003 | Unknown target value | `target=foo` | 400 | +| NEG-005 | Empty artifact selection | `artifacts=[]` | 400 | +| NEG-006 | Requested kind not on model | `artifacts=["footprints"]` when absent | 404 | +| NEG-004 | Non-owner unpublish | other user | 403 | +| EDGE-001 | Assessment report unavailable | no report | description prefill blank; publish still works | +| EDGE-002 | Source model deleted after Local publish (copy mode) | delete model | links still resolve | +| EDGE-003 | GeoCatalog ingestion never terminates | stuck poll | bounded timeout → FAILED | +| EDGE-004 | STAC item id with illegal chars | dirty name | sanitized; ingest succeeds | + +### Performance Tests + +| ID | Scenario | Load Profile | Target Metric | Threshold | +|---|---|---|---|---| +| PERF-001 | Enqueue latency | 50 concurrent publishes | p99 API latency | < 2s | +| PERF-002 | List at volume | 1,000 datasets in index | `GetPublishedDatasets` p95 | < 1s | +| PERF-003 | Local copy throughput | 500 MB artifact set | copy time | bounded, logged | + +## Test Data Requirements + +| Dataset | Description | Source | Sensitive? | +|---|---|---|---| +| Sample damage GPKG + valid mask + COG | Small model output fixtures | Synthetic | no | +| Sample assessment report JSON | For description prefill + STAC props | Synthetic | no | +| Mock GeoCatalog responses | 201/202 + poll status transitions | Fixtures | no | + +## Coverage Matrix + +| User Story | Unit | API Integration | Queue | UI | E2E | Performance | +|---|---|---|---|---|---|---| +| US-001 | UT-001, UT-009 | IT-002/003/004/005 | QT-001 | UI-001/002/003 | E2E-001 | PERF-001 | +| US-002 | — | IT-006 | — | UI-005/006 | E2E-002 | PERF-002 | +| US-003 | UT-004/005/009 | IT-002 | QT-001 | — | E2E-001/003 | PERF-003 | +| US-004 | UT-006/007/008/010 | — | QT-002/003 | — | E2E-004 | — | +| US-005 | — | IT-007/008 | — | UI-006 | E2E-003 | — | +| US-006 | UT-002 | IT-001 | — | UI-004 | — | — | +| US-007 | — | NEG-005/006 | — | UI-007 | — | — | + +## Environment Requirements + +| Environment | Purpose | Config | +|---|---|---| +| Local (Docker Compose) | Dev + Local-target E2E | `docker-compose.yml` + Azurite + `publish-queue` seed | +| CI (GitHub Actions) | Unit/integration + Component Governance | `secret-scan.yml`, `deploy-apps.yml` | +| Dev1 SWA | Integration incl. PC target (dev GeoCatalog) | SWA `dev1` + PC config | +| Testing SWA | Pre-prod validation | SWA `test` config | + +## Sign-off Criteria + +- [ ] All P0 stories (US-001/002/003) have E2E coverage +- [ ] `core/publishing/` unit coverage ≥ 85% +- [ ] No P0/P1 bugs open +- [ ] Performance thresholds met +- [ ] `docker compose up` runs clean with Local publishing +- [ ] GitHub Actions CI passes (secret-scan, deploy-apps) +- [ ] Component Governance clean (`azure-identity`, `pystac`, `geopandas`, `pyogrio`, `shapely`) +- [ ] PC provider validated against a real/dev GeoCatalog (gated) diff --git a/spec/features/data-publishing/user-stories.md b/spec/features/data-publishing/user-stories.md new file mode 100644 index 00000000..294f74f5 --- /dev/null +++ b/spec/features/data-publishing/user-stories.md @@ -0,0 +1,341 @@ +# User Stories: Data Publishing & Published Datasets + +## Personas + +| Persona | Description | Key Goals | +|---|---|---| +| Disaster Analyst | Domain expert who runs training/inference and interprets damage results | Turn a finished result into a stable, shareable, described dataset without hand-authoring metadata | +| Project Manager | Oversees a response project and its outputs | A single place to see what's been published for the event, and where | +| External Partner / Data Consumer | Downstream user (agency, NGO, researcher) | Discover finished HASTE datasets and retrieve them, ideally via standards (STAC) they already use | +| Admin | Configures system settings, base models, source types, and now publishing targets | Enable/disable providers, set the Planetary Computer GeoCatalog endpoint + credentials | + +> The **ML Engineer** persona from the template is out of scope — publishing acts +> on *finished* artifacts, not the training workflow. + +--- + +## Stories + +### US-001: Publish a dataset from model results + +**As a** Disaster Analyst, +**I want to** click **Publish dataset** on a completed model's results and fill a short form, +**So that** the model's outputs become a named, described, discoverable dataset instead of a one-off download link. + +**Priority:** P0 +**Estimate:** L +**Component(s):** `ui/src/Components/ProjectManagement/ModelResultsButton.jsx`, `ui/src/Components/PublishDatasetModal.jsx`, `api/hastefuncapi` (`PutPublishDatasetQueueMessage`) + +**Acceptance Criteria:** + +```gherkin +Given a model whose inference has completed and produced a GeoPackage +When I open the model's results menu +Then I see a "Publish dataset…" action alongside the download actions +``` + +```gherkin +Given I open the Publish dataset dialog +When the dialog loads +Then the Dataset name is pre-filled with " – " + And the Description is pre-filled from the assessment report summary (when available) + And an "Assets to publish" checklist shows the model's available outputs, all prechecked + And the Target publishing location dropdown lists "Local" and "Planetary Computer" +``` + +```gherkin +Given I have edited the name and chosen a target +When I click Publish +Then a PublishedDataset record is created with status PENDING + And a message is enqueued to the publish-queue + And the dialog confirms "Publishing started" and closes +``` + +**UI Wireframe:** see [ux-spec.md](ux-spec.md#publish-dataset-dialog). + +**Notes:** Publish action is disabled (with tooltip) unless the model has at +least one publishable artifact (e.g. `gpkgUrl`) and status is completed. + +--- + +### US-002: Browse the Published Datasets section + +**As a** Project Manager, +**I want to** open a **Published Datasets** section and see every published dataset, +**So that** I have one authoritative view of the event's finished outputs and where each was published. + +**Priority:** P0 +**Estimate:** M +**Component(s):** `ui/src/Components/PublishedDatasets.jsx`, `PublishedDatasetRow.jsx`, `api/hastefuncapi` (`GetPublishedDatasets`) + +**Acceptance Criteria:** + +```gherkin +Given one or more datasets have been published +When I navigate to Published Datasets +Then I see a searchable, sortable, paginated list showing name, project/layer, + target, status, published-by, and published date +``` + +```gherkin +Given no datasets have been published yet +When I open Published Datasets +Then I see an empty state explaining how to publish from model results +``` + +```gherkin +Given a dataset is still publishing +When I view the list +Then that row shows an in-progress indicator and no broken retrieval links +``` + +**Notes:** Mirror `ModelCatalog.jsx` (search box, `PAGE_SIZE_OPTIONS`, sort +state, `pgrid-*` layout, `NoResultsMessage`). Optional `projectId` query param +lets the section be filtered to one project. + +--- + +### US-003: Publish to Local HASTE storage + +**As a** Disaster Analyst, +**I want to** publish to **Local** (HASTE storage), +**So that** the dataset is registered and retrievable inside HASTE even if the source model is later re-run or deleted. + +**Priority:** P0 +**Estimate:** M +**Component(s):** `hastelib/.../core/publishing/local_provider.py`, `hastefuncqueues` + +**Acceptance Criteria:** + +```gherkin +Given I publish a completed model to the Local target +When the publish-queue worker runs the Local provider +Then the model's artifacts are copied to published/{datasetId}/ in HASTE storage + And the dataset record gains stable retrieval links + status PUBLISHED + And the dataset is visible in the Published Datasets section +``` + +```gherkin +Given the source model is deleted after publishing +When I retrieve the published dataset +Then its links still resolve (published copy is independent) +``` + +**Notes:** Published copies are immutable (copy-on-publish) for lifecycle +independence; total copied bytes are bounded by `PUBLISH_MAX_TOTAL_BYTES`. + +--- + +### US-004: Publish to Planetary Computer (STAC) + +**As a** Disaster Analyst, +**I want to** publish to **Planetary Computer**, +**So that** HASTE outputs are catalogued as STAC and discoverable/tileable in the broader geospatial ecosystem. + +**Priority:** P1 +**Estimate:** XL +**Component(s):** `hastelib/.../core/publishing/planetary_computer_provider.py`, `core/publishing/stac.py`, `hastefuncqueues` + +**Acceptance Criteria:** + +```gherkin +Given the Planetary Computer provider is configured (GeoCatalog URL + credential + ingestion source) +When I publish a dataset to the Planetary Computer target +Then a STAC Collection for the project is created (or reused) via /stac/collections + And STAC Item(s) for the dataset's artifacts are POSTed to /stac/collections/{id}/items + And the worker polls the returned ingestion location until it reaches a terminal state + And the dataset record stores the collection id, item ids, and explorer links with status PUBLISHED +``` + +```gherkin +Given the GeoCatalog rejects an item (validation / 40x) or ingestion fails +When the worker processes the failure +Then the dataset status becomes FAILED with a human-readable statusMessage + And no partial/broken links are surfaced in the UI +``` + +**Notes:** STAC Item ids must avoid `-_+().` (GeoCatalog restriction) — see +[design.md](design.md#planetary-computer-provider--stac-mapping). Assets reference HASTE blob URLs; the +GeoCatalog copies them via the pre-registered ingestion source. + +--- + +### US-005: Retrieve, inspect, and unpublish a dataset + +**As an** External Partner / Data Consumer, +**I want to** open a published dataset's detail and retrieve its artifacts or STAC links, +**So that** I can consume the finished output; and as an owner/admin I want to **unpublish** it. + +**Priority:** P1 +**Estimate:** M +**Component(s):** `ui/src/Components/PublishedDatasetRow.jsx` (detail/menu), `api/hastefuncapi` (`GetPublishedDataset`, `DeletePublishedDataset`) + +**Acceptance Criteria:** + +```gherkin +Given a PUBLISHED dataset +When I open its detail/actions +Then I can download each Local artifact and/or follow the Planetary Computer collection link +``` + +```gherkin +Given I am the publisher or an admin +When I choose Unpublish +Then the record is removed from the section + And Local published copies are cleaned up + And (for Planetary Computer) the collection/items are deleted via the STAC API (best-effort, logged) +``` + +**Notes:** Unpublish permission = publisher or admin (see +[ux-spec.md](ux-spec.md#permissions--access-control) and README access +assumptions). + +--- + +### US-006: Configure and select publishing providers + +**As an** Admin, +**I want to** configure available providers and their requirements, +**So that** analysts only see targets that are actually usable, with clear validation. + +**Priority:** P2 +**Estimate:** S +**Component(s):** `api/hastefuncapi` (`GetPublishingProviders`), `core/publishing/registry.py`, `config.py` + +**Acceptance Criteria:** + +```gherkin +Given the Planetary Computer GeoCatalog settings are not configured +When an analyst opens the target dropdown +Then Planetary Computer is shown as disabled with a "not configured" note + And Local remains available +``` + +```gherkin +Given a provider declares config requirements +When the UI renders the dialog +Then it surfaces provider-specific validation from GetPublishingProviders (no hard-coding per provider) +``` + +**Notes:** Provider metadata (id, display name, config requirements, async flag) +comes from the registry so the UI is provider-agnostic. In v1, the actual +configuration values (GeoCatalog URL, ingestion source, collection prefix) are +**Azure App Settings** set by an operator at deploy time — there is no in-app +provider-config screen. Credentials are managed identity only; this story is +about *reflecting* configuration state in the dialog, not entering it. + +--- + +### US-007: Choose which outputs to publish + +**As a** Disaster Analyst, +**I want to** select which of the model's existing outputs (damage GeoPackage, +valid mask, building footprints, processed image COG, …) are included, +**So that** I publish exactly the deliverables I intend to share and skip the rest. + +**Priority:** P1 +**Estimate:** S +**Component(s):** `ui/src/Components/PublishDatasetModal.jsx`, `api/hastefuncapi` (`PutPublishDatasetQueueMessage`), `core/publishing` (`ArtifactBundle`) + +**Acceptance Criteria:** + +```gherkin +Given the Publish dataset dialog is open for a model +When the asset checklist renders +Then it lists one checkbox per artifact the model actually produced (all prechecked) + And artifacts the model did not produce are grayed/omitted +``` + +```gherkin +Given I uncheck some assets and leave at least one checked +When I publish +Then only the selected assets are copied (Local) or turned into STAC assets/items (Planetary Computer) + And the PublishedDataset.artifacts records exactly the selected kinds +``` + +```gherkin +Given I uncheck every asset +When I try to publish +Then Publish is disabled with "Select at least one asset to publish" + And a forced API call with an empty selection returns 400 +``` + +**Notes:** Selection is sent as `artifacts: [...]` in the publish request; +`ArtifactBundle` is filtered to the selection before any provider runs (see +[design.md](design.md#publishing-provider-interface)). + +--- + +## Agent Assignment Map + +Every user story is assigned to one or more HASTE agents. The **implementing +agent** writes the code; the **validating agent** verifies against acceptance +criteria. See [Agent Architecture](../../architecture/overview.md#agent-architecture). + +### Available Agents + +| Agent | Scope | Touches Code? | +|---|---|---| +| `backend-dev` | Python backend, API, processors, data layers, publishing providers | Yes | +| `gis` | STAC item/collection generation, GDAL/rasterio reads of COG/GPKG | Yes | +| `ui` | React/FluentUI section + dialog, sidebar/routing | Yes | +| `security` | New Python deps (`azure-identity`, `pystac`, `geopandas`, `pyogrio`, `shapely`), credential handling | No (reports only) | +| `backend-validation` | Validates backend/provider code against specs, conventions, tests | No (validates only) | +| `ui-validation` | Validates section + dialog behavior and states | No (validates only) | +| `security-validation` | Validates security findings | No (validates only) | +| `orchestrator` | Tracks spec status and agent work | No (observes only) | + +### Story → Agent Mapping + +| Story | Implementing Agent(s) | Validating Agent(s) | Notes | +|---|---|---|---| +| US-001 | `ui`, `backend-dev` | `ui-validation`, `backend-validation` | Dialog + enqueue endpoint | +| US-002 | `ui` | `ui-validation` | Catalog-style section | +| US-003 | `backend-dev` | `backend-validation` | Local provider + worker | +| US-004 | `gis`, `backend-dev` | `backend-validation`, `security-validation` | STAC + PC provider + creds | +| US-005 | `ui`, `backend-dev` | `ui-validation`, `backend-validation` | Retrieve + unpublish | +| US-006 | `backend-dev`, `ui` | `backend-validation` | Provider metadata/registry | +| US-007 | `ui`, `backend-dev` | `ui-validation`, `backend-validation` | Asset selection in dialog + request | + +> **Rules:** every story has ≥1 implementing + ≥1 validating agent; `hastelib`/`api` +> → `backend-dev`+`backend-validation`; STAC/imagery reads → `gis`; `ui/` → +> `ui`+`ui-validation`; new deps → `security`+`security-validation`. + +### Agent Workflow Per Phase + +| Phase | Lead Agent | Supporting Agents | Validation | +|---|---|---|---| +| Phase 1 — Core Library (models, provider ABC, Local, STAC) | `backend-dev` | `gis` | `backend-validation` | +| Phase 2 — API + Queue | `backend-dev` | — | `backend-validation` | +| Phase 3 — Planetary Computer provider | `gis` | `backend-dev`, `security` | `backend-validation`, `security-validation` | +| Phase 4 — UI | `ui` | — | `ui-validation` | +| Phase 5 — Integration & Deployment | `backend-dev` | `ui` | `backend-validation`, `ui-validation` | + +## Story Map + +| Priority | Story | Phase | Implementing Agent | Component | +|---|---|---|---|---| +| P0 | US-001 | Phase 4 — UI (+ Phase 2 API) | `ui`, `backend-dev` | `ui/`, `hastefuncapi` | +| P0 | US-002 | Phase 4 — UI | `ui` | `ui/src/Components/` | +| P0 | US-003 | Phase 1/2 — Core + Queue | `backend-dev` | `core/publishing/local_provider.py` | +| P1 | US-004 | Phase 3 — PC provider | `gis`, `backend-dev` | `core/publishing/planetary_computer_provider.py` | +| P1 | US-005 | Phase 2/4 — API + UI | `ui`, `backend-dev` | `hastefuncapi`, `ui/` | +| P2 | US-006 | Phase 2/4 — API + UI | `backend-dev`, `ui` | `core/publishing/registry.py` | +| P1 | US-007 | Phase 4 — UI (+ Phase 2 API) | `ui`, `backend-dev` | `ui/`, `hastefuncapi` | + +## Out of Scope + +- [ ] Publishing arbitrary user uploads (only HASTE-generated artifacts of a + project/layer/model are publishable in v1). +- [ ] Versioning / re-publishing history beyond a single current record per + dataset (a re-publish overwrites/updates; no version tree). +- [ ] Access-control sharing lists per dataset (public-within-tenant only in v1; + no per-user ACLs). +- [ ] Additional providers (ArcGIS, generic STAC API, DOI/Zenodo) — the interface + supports them but none are built in v1. +- [ ] Editing STAC metadata by hand in the UI (generated from artifacts + + assessment report only). +- [ ] In-app admin screen for configuring publishing targets — v1 configures + providers via Azure App Settings + managed identity (operator/deploy-time). + A future admin UI can slot behind the existing `ProviderInfo` contract + without UI/API rework. diff --git a/spec/features/data-publishing/ux-spec.md b/spec/features/data-publishing/ux-spec.md new file mode 100644 index 00000000..1648fd64 --- /dev/null +++ b/spec/features/data-publishing/ux-spec.md @@ -0,0 +1,203 @@ +# UX Specification: Data Publishing & Published Datasets + +> New UI reuses the existing `pgrid-*` catalog layout, Fluent UI v9 components, +> `theme.js`/`ThemeContext`, and the `icons.jsx` `FluentIcon` wrapper. The two +> anchor patterns are `ModelCatalog.jsx` (the section) and the existing modals / +> `ModelResultsButton.jsx` menu (the entry point + dialog). + +## Entry points + +### 1. "Publish dataset…" on model results + +Added to the results menu in +[ModelResultsButton.jsx](../../../ui/src/Components/ProjectManagement/ModelResultsButton.jsx), +after **Assessment Report** (the menu today: View · Download Geopackage · +Download Training/Inference Artifacts · Validation Report · Assessment Report): + +``` +┌─ results menu ───────────────┐ +│ View │ +│ Download Geopackage (.gpkg) │ +│ Download Training Artifacts │ +│ Download Inference Artifacts │ +│ Validation Report │ +│ Assessment Report │ +│ ───────────────────────── │ +│ ⇪ Publish dataset… │ ← NEW (FluentIcon "Share"/"CloudArrowUp") +└──────────────────────────────┘ +``` + +- **Enabled** only when the model is completed and has ≥1 publishable artifact + (`model.gpkgUrl` present). Otherwise disabled with tooltip "Run inference to + produce a dataset before publishing." +- Click → opens **Publish Dataset dialog** (below). + +### 2. Sidebar → Published Datasets section + +New nav item in [AppSidebar.jsx](../../../ui/src/Components/AppSidebar.jsx), +grouped near **Model Catalog**; route `/published-datasets` in +[AppBody.jsx](../../../ui/src/Components/AppBody.jsx). Icon: `Database` / +`CloudArrowUp`. Visible to all authenticated users (see +[permissions](#permissions--access-control)). + +## Publish Dataset dialog + +Fluent UI **Dialog** (small form → `Dialog`/`DialogSurface`/`DialogBody`, as in +`SectionModal.jsx`; may use `OverlayDrawer` per the app's modal convention). Fields: + +``` +┌─ Publish dataset ───────────────────────────────── ✕ ┐ +│ │ +│ Dataset name * │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Hurricane Harvey – Downtown Layer │ │ ← prefilled ' – ', editable +│ └────────────────────────────────────────────────┘ │ +│ │ +│ Description │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ 1,234 of 5,000 known buildings predicted │ │ ← prefilled from assessment report +│ │ damaged (precision 0.82, recall 0.77)… │ │ summary; editable (Textarea) +│ └────────────────────────────────────────────────┘ │ +│ │ +│ Assets to publish * │ +│ ☑ Damage GeoPackage (.gpkg) 2.3 MB │ ← checkboxes, one per AVAILABLE +│ ☑ Valid-area mask (.geojson) 120 KB │ output; prechecked; ≥1 required +│ ☑ Building footprints (.gpkg) 1.1 MB │ +│ ☑ Processed image (COG .tif) 48 MB │ +│ ☐ Training artifacts (.zip) — (grayed if │ +│ absent) │ +│ │ +│ Target publishing location * │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Local (HASTE storage) ▾ │ │ ← Dropdown; options from +│ └────────────────────────────────────────────────┘ │ GetPublishingProviders +│ • Local (HASTE storage) │ +│ • Planetary Computer (disabled if not configured)│ +│ │ +│ [provider hint / validation message] │ +│ │ +│ [ Cancel ] [ Publish ] │ +└──────────────────────────────────────────────────────┘ +``` + +- **Dataset name** — `Input`, required, prefilled `${projectName} – ${layerName}`, + editable. Validation via existing `util/validation.js` (`validateEmptyOrInvalid`). +- **Description** — `Textarea`, optional, prefilled from the assessment report + summary. Prefill source: `GetAssessmentReport` + shared + `assessmentSummary.buildSummarySentence()` (extracted from + `AssessmentReportModal.jsx`). If the report is unavailable, field is blank. +- **Assets to publish** — a `Checkbox` list, **one per artifact the source model + actually produced** (resolved from the model/layer documents, with size where + known). All available assets are **prechecked** by default; artifacts the model + did not produce are shown grayed/disabled (or omitted). At least one asset must + be selected. Candidate kinds: Damage GeoPackage (`gpkg`), Valid-area mask + (`valid_mask`), Building footprints (`footprints`), Processed image COG + (`processed_cog`); optionally training/inference artifact zips. The selection is + sent as `artifacts: [...]` in the publish request and determines exactly what is + copied (Local) or turned into STAC assets/items (Planetary Computer). +- **Target** — `Dropdown`, required, options from `GetPublishingProviders`. + Unconfigured providers render **disabled** with a "not configured" note. If a + provider declares `configRequirements`, surface them as helper text / + validation (provider-agnostic — no per-provider UI hard-coding). +- **Publish** — validates, calls `PutPublishDatasetQueueMessage`, shows a success + toast/dialog ("Publishing started — track it in Published Datasets"), closes. +- **Cancel / ✕** — dismiss, no side effects. + +### Dialog states + +| State | UI | +|---|---| +| Loading prefill | Name/target ready immediately; asset checklist populated from the model's available outputs; description shows a subtle spinner until the assessment report resolves (non-blocking) | +| No assets selected | Publish disabled; inline hint "Select at least one asset to publish" | +| Validation error | Field-level `validationMessage` (name) or dialog-level banner (target/provider) | +| Submitting | Publish button shows spinner + disabled; fields locked | +| Submit success | Toast/confirmation + close; new row appears in section as IN_PROGRESS | +| Submit conflict (409) | Banner "A dataset with this name is already publishing" | +| Submit failure (5xx) | Banner with retry; dialog stays open | + +## Published Datasets section + +Catalog-style page modeled on `ModelCatalog.jsx` — `pgrid-page` container, +`pgrid-header` (title + subtitle + info tooltip), `pgrid-toolbar` (`SearchBox` + +optional target/status filters), sortable table, `pgrid-footer` pagination with +`PAGE_SIZE_OPTIONS`. + +``` +┌─ Published Datasets ─────────────────────────────────── ⓘ ┐ +│ Curated, described outputs published from HASTE results. │ +│ │ +│ [ 🔍 Search ] Target: [All ▾] Status: [All ▾] │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ Name ↕ | Project/Layer | Target | Status | By | Date │ │ +│ ├──────────────────────────────────────────────────────┤ │ +│ │ Harvey – Downtown | Harvey/L1 | Local | ✅ Published |…│ │ +│ │ Ida – Coast | Ida/L2 | 🌐 PC | ⏳ Publishing|…│ │ +│ │ Fiona – North | Fiona/L1 | 🌐 PC | ❌ Failed |…│ │ +│ └──────────────────────────────────────────────────────┘ │ +│ Showing 1–8 of 23 Rows: [8 ▾] ‹ Prev Next › │ +└────────────────────────────────────────────────────────────┘ +``` + +Columns: **Name**, **Project / Layer**, **Target** (Local / Planetary Computer +with icon), **Status** (chip), **Published by**, **Published date**, **Actions**. +Sortable columns mirror the catalog (`toggleSort`, `{key, dir}` state). Row +actions (menu on `PublishedDatasetRow.jsx`): + +- **Published (Local):** Download each artifact (reuses `fileDownload`). +- **Published (PC):** Open in Explorer / Copy STAC collection link. +- **In progress:** actions disabled; live status. +- **Failed:** show `statusMessage`; Retry (re-publish); Remove. +- **Owner/admin:** Unpublish (`DeletePublishedDataset`, with confirm). + +## UI states (all) + +| State | Trigger | UX | +|---|---|---| +| **Empty** | No datasets published | Icon + "No published datasets yet" + one-line guide "Publish from a model's results menu" (mirrors ModelCatalog empty state) | +| **Loading** | Section fetch in flight | Full-page overlay spinner via `appContext.setIsLoading()` (catalog pattern) | +| **No results (filtered)** | Search/filter matches nothing | `NoResultsMessage` with a clear-search action | +| **In progress** | Dataset status IN_PROGRESS/PENDING | Row chip "⏳ Publishing…"; UI polls `GetPublishedDatasets` (or per-row `GetPublishedDataset`) on an interval until terminal; no broken links | +| **Success** | Status PUBLISHED | Row chip "✅ Published"; retrieval actions enabled | +| **Failure** | Status FAILED | Row chip "❌ Failed"; expandable `statusMessage`; Retry / Remove | + +Status chips use Fluent tokens (`colorPaletteGreenForeground1`, +`colorPaletteYellowForeground1`, `colorPaletteRedForeground1`) via +`theme.js`/`ThemeContext`, consistent with the app's theming. + +## How published datasets are displayed and accessed + +- **Discovery:** the Published Datasets section (searchable/sortable/paginated). +- **Local retrieval:** direct SAS download per artifact via `fileDownload` + (same mechanism as the current results downloads). +- **Planetary Computer retrieval:** external links to the GeoCatalog collection + and Explorer; STAC collection URL is copyable for programmatic use. **Set + expectations in the UI:** in the PC Explorer a published dataset shows as item + **footprints + metadata**, and the damage **GeoPackage is download-only** — PC + Pro does not tile/render vector data, so the damage layer is not drawn on the + Explorer map (rasterized-COG rendering is a future enhancement). A short + helper note on PC-target rows/detail should say "Footprints + metadata in + Explorer; download the GeoPackage for the damage layer." +- **Provenance:** each row exposes source project/layer/model + the assessment + summary snapshot. + +## Permissions & access control + +| Action | Who | Enforcement | +|---|---|---| +| See the Published Datasets section | Any authenticated user | Route available to all logged-in users (like Model Catalog viewing) | +| Publish a dataset | Any user who can access the source project/model | Same access as viewing/downloading the model's results today | +| Unpublish / Retry / Remove | The publisher (`publishedByUser`) or an Admin | API checks the client principal (`_decode_client_principal`); UI hides the action otherwise | +| Configure Planetary Computer target | Admin / operator (app settings) | Config via App Settings; UI reflects `isConfigured` from `GetPublishingProviders` | + +> v1 assumes "public within the tenant" visibility (no per-dataset ACLs), matching +> the current Model Catalog model. Per-dataset sharing lists are out of scope +> ([user-stories.md](user-stories.md#out-of-scope)). + +## Accessibility & responsiveness + +- Fluent UI components provide keyboard/focus/ARIA defaults; dialog traps focus + and closes on Esc. +- Section table follows the responsive `pgrid`/Bootstrap breakpoints + (`appParams.bootstrapBreakpoint`); on narrow widths the table collapses to + stacked cards (as ModelCatalog does). +- Status conveyed by icon **and** text (not color alone).