Prbatero/feat/data publishing pc - #123
Open
Anthony Cintron (prbatero) wants to merge 17 commits into
Open
Conversation
…lease
Bucket A (additive) from the -sol audit split, Phase 1 (Local only; no PC):
- core/publishing/: base ABC + registry (PC lazy-loaded, disabled by default),
repository (per-dataset docs + optimistic concurrency), source (eligibility +
artifact resolution), lease (blob-lease coordinator), local_provider (immutable
copy to published/{datasetId}/).
- models/publishing.py; processors/publishing.py + assessment.py.
- Additive infra: config publishing block + PUBLISHED_DATASET type + publish
queue; data-layer load_bounded/load_page/update_locked; artifact-storage
copy/scoped-download/etag/size/exists/delete_prefix; blob.py max_bytes.
- hastefuncqueues publish trigger.
- Phase-1 unit tests (registry/repository/source/local_provider/lease): 45 pass.
Excludes Bucket B: reverted the high-risk load_all/load_all_from_partition
rewrite to main's versions; endpoint hardening + DTOs left for a separate
-hardening PR. See spec/features/data-publishing/sol-classification.md.
Bucket A only. Adds to api/hastefuncapi/function_app.py: publishing imports, active-caller/authorization helpers, the DeleteProject published-dataset guard, and 7 publishing routes (GetPublishingProviders, GetPublishDatasetOptions, GetPublishedDatasets, GetPublishedDataset, PutPublishDatasetQueueMessage, PutRetryPublishedDatasetQueueMessage, DeletePublishedDataset). All existing endpoints reverted to main's versions — the Bucket-B endpoint hardening (IDOR checks, strict *Request DTOs, server-managed-field enforcement, assessment size caps) is deferred to a separate -hardening PR. Tests: 19 publishing route tests + 6 queue-handler tests pass. Existing core suite: 191 pass (3 failures are pre-existing on main — clipBbox mocks).
Bucket A UI, gated behind the publishingEnabled flag: - New: PublishedDatasets section, PublishedDatasetRow, PublishDatasetModal (name/description prefill, asset checklist, provider-driven target dropdown), util/publishing.js, util/assessmentSummary.js (extracted from AssessmentReportModal). - Additive wiring: App/AppContext (publishingEnabled + provider fetch), AppBody route + AppSidebar nav, "Publish dataset..." in ModelResultsButton + EmbeddingModelRow, Database icon. - api.js: apiPut/apiDelete now accept any 2xx (async 202 queue responses) — minimal additive change; the shared-helper ApiError rewrite is NOT taken (deferred to -hardening). Existing 200 callers unaffected. Validated: full `vite build` compiles cleanly (all imports resolve). Vitest unit tests + config are preserved on -sol/archive and land in a follow-up UI-test-infra slice (needs vitest devDeps + package-lock regen).
UI (Published Datasets section): - Clickable Project/Layer cell -> opens the source project with that layer expanded (/project/:projectId/:imageLayerId). - Labeled "Actions" button (replacing the bare "...") and an "Actions" column header; keep the menu (download / open in Explorer / retry / unpublish). - New "View details" action -> modal with the generated description, project/ layer, model, target, status, publisher, headline assessment stats, and the published assets (kind + size). - "Published by" now mirrors the project Creator display: the publisher's login/email via limitTextLength, not the opaque object id. - Desktop column widths moved into a @media (min-width: 993px) rule scoped to .pgrid-page--published-datasets so the mobile card layout (inherited from .pgrid-page--model-catalog) is preserved; font URLs forced to https (CSP). Backend: - Capture the publisher's display name (client-principal userDetails) at publish time as PublishedDataset.publishedByName, so the UI can show a human name while the ownership check still uses the object id.
Add the data-publishing feature flag and queue name to the api/queues
Function App settings so the Local target runs in Azure:
- functions.bicep: PUBLISH_QUEUE_NAME ('publish-queue') and
PUBLISHING_ENABLED (from a new publishingEnabled param) in appConfigSettings.
- main.bicep / main.bicepparam: publishingEnabled param, default off,
sourced from HASTE_PUBLISHING_ENABLED.
The publish-queue and the publishing-locks blob container are auto-created at
runtime, and APIM operations are auto-synced from the deployed function app by
the postdeploy hook, so no queue/container/APIM resources are needed here.
Other Local knobs (PUBLISH_MAX_TOTAL_BYTES, PUBLISHED_DOWNLOAD_SAS_MINUTES,
PUBLISHING_LOCK_CONTAINER) use code defaults.
Note: infra/main.json (compiled ARM artifact) is intentionally not regenerated
here to avoid a bicep CLI-version reformat; regenerate it with the repo's
pinned bicep version.
Add the publishing feature flag + queue/lock settings to the api and queues services so `docker compose up` exercises Local publishing against Azurite (the publish-queue and publishing-locks container auto-create at runtime). PC_PROVIDER_ENABLED stays false — no local GeoCatalog.
Drop the accidental "install" and "npm" entries from the UI dependencies (and their lockfile trees). Neither is imported anywhere; the npm CLI as a runtime dependency is what caused local image builds to fetch an npm tarball. Mirrors the earlier cleanup in #64 before they regressed back in.
… deploy paths
The Config drift check failed with:
Dead application settings (set but never read):
PUBLISHING_ENABLED (set by infra/modules/functions.bicep, read by no code)
That was a false positive. PUBLISHING_ENABLED is read in hastegeo.core.config
via _get_bool_env("PUBLISHING_ENABLED", False), but check_env_drift.py's AST
scanner only recognized os.getenv, os.environ.get, and os.environ[...]. The new
typed wrappers _get_bool_env and _get_bounded_int_env were invisible to it, so
no reader was found and the setting looked dead.
The blind spot covered five variables, not one: PC_PROVIDER_ENABLED,
PUBLISH_MAX_TOTAL_BYTES, PUBLISHED_DOWNLOAD_SAS_MINUTES and PC_VERIFY_ATTEMPTS
are read through the same helpers and were equally unseen. They simply did not
error because no deploy path emits them yet.
Teach the scanner to treat _get_*_env(NAME, ...) calls as reads. The wrappers
supply their own default in the signature, so such a read counts as optional
even when the call site passes no default.
Separately, fix the real drift the check could not see: PUBLISHING_ENABLED was
emitted by functions.bicep but not by deploy_apps.sh, so environments deployed
through the GitHub Actions path would never receive it and would silently fall
back to false while Bicep-provisioned environments got the flag. Emit it from
deploy_apps.sh too, defaulting off to match the Bicep param, and wire it
through deploy-apps.yml as a GitHub Environment variable (non-sensitive feature
flag, same treatment as VITE_SHOW_FOOTER).
Copilot started reviewing on behalf of
Anthony Cintron (prbatero)
August 12, 2026 12:31
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end dataset publishing through local storage and Planetary Computer, including queue processing, persistence, infrastructure configuration, UI management, and tests.
Changes:
- Introduces publishing models, providers, storage operations, queue handlers, and reconciliation.
- Adds publishing dialogs, catalog browsing, downloads, retries, and unpublishing.
- Extends infrastructure configuration and backend test coverage.
Reviewed changes
Copilot reviewed 65 out of 68 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
ui/src/util/publishing.js |
Publishing status and artifact helpers. |
ui/src/util/icons.jsx |
Adds database icon support. |
ui/src/util/assessmentSummary.js |
Extracts assessment summary formatting. |
ui/src/util/api.js |
Supports additional successful HTTP statuses. |
ui/src/Components/PublishedDatasets.jsx |
Adds published-dataset catalog. |
ui/src/Components/PublishedDatasetRow.jsx |
Adds dataset details and actions. |
ui/src/Components/PublishDatasetModal.jsx |
Adds publishing form. |
ui/src/Components/ProjectManagement/ModelResultsButton.jsx |
Adds publish action for model results. |
ui/src/Components/ProjectManagement/EmbeddingModelRow.jsx |
Adds publish action for embeddings. |
ui/src/Components/BuildingValidation/AssessmentReportModal.jsx |
Reuses summary utility. |
ui/src/Components/AppSidebar.jsx |
Adds publishing navigation. |
ui/src/Components/AppBody.jsx |
Registers publishing route. |
ui/src/assets/css/style.css |
Styles publishing table and font URLs. |
ui/src/AppContext.jsx |
Adds publishing state. |
ui/src/App.jsx |
Loads publishing capabilities. |
ui/package.json |
Removes unused npm dependencies. |
infra/modules/roles.bicep |
Adds optional ingestion storage role. |
infra/modules/functions.bicep |
Adds publishing app settings. |
infra/main.bicepparam |
Maps publishing environment parameters. |
infra/main.bicep |
Wires publishing infrastructure parameters. |
hastelib/tests/core/publishing/test_source.py |
Tests source validation. |
hastelib/tests/core/publishing/test_repository.py |
Tests persistence and concurrency. |
hastelib/tests/core/publishing/test_registry.py |
Tests provider registration. |
hastelib/tests/core/publishing/test_planetary_computer_transport.py |
Tests GeoCatalog transport. |
hastelib/tests/core/publishing/test_local_provider.py |
Tests local publishing. |
hastelib/tests/core/publishing/test_lease.py |
Tests blob lease coordination. |
hastelib/src/hastegeo/core/utils/blob.py |
Adds bounded streaming downloads. |
hastelib/src/hastegeo/core/publishing/source.py |
Resolves verified source artifacts. |
hastelib/src/hastegeo/core/publishing/schemas/projection-v2.0.0.json |
Vendors projection schema. |
hastelib/src/hastegeo/core/publishing/schemas/item-assets-v1.0.0.json |
Vendors item-assets schema. |
hastelib/src/hastegeo/core/publishing/repository.py |
Adds publishing persistence. |
hastelib/src/hastegeo/core/publishing/registry.py |
Adds provider registry. |
hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py |
Adds GeoCatalog adapter. |
hastelib/src/hastegeo/core/publishing/local_provider.py |
Adds local provider. |
hastelib/src/hastegeo/core/publishing/lease.py |
Adds distributed operation locks. |
hastelib/src/hastegeo/core/publishing/geocatalog_client.py |
Adds authenticated REST client. |
hastelib/src/hastegeo/core/publishing/base.py |
Defines provider contract. |
hastelib/src/hastegeo/core/publishing/__init__.py |
Exports publishing primitives. |
hastelib/src/hastegeo/core/processors/metadata.py |
Adds bounded and paged reads. |
hastelib/src/hastegeo/core/processors/assessment.py |
Adds assessment snapshot processor. |
hastelib/src/hastegeo/core/models/publishing.py |
Defines publishing schemas. |
hastelib/src/hastegeo/core/data_layer/unified.py |
Exposes paged storage operations. |
hastelib/src/hastegeo/core/data_layer/local_file_system_data_layer.py |
Adds bounded local reads. |
hastelib/src/hastegeo/core/data_layer/azure_postgresql_data_layer.py |
Adds bounded PostgreSQL reads. |
hastelib/src/hastegeo/core/data_layer/azure_data_lake_data_layer.py |
Adds bounded Data Lake reads. |
hastelib/src/hastegeo/core/data_layer/azure_cosmos_db_data_layer.py |
Adds bounded Cosmos reads. |
hastelib/src/hastegeo/core/data_layer/azure_blob_storage_data_layer.py |
Adds indexed blob pagination. |
hastelib/src/hastegeo/core/data_layer/abstract_data_layer.py |
Extends data-layer interface. |
hastelib/src/hastegeo/core/artifact_storage/unified_artifact_storage.py |
Exposes publishing storage operations. |
hastelib/src/hastegeo/core/artifact_storage/local_file_system_artifact_storage.py |
Implements local artifact operations. |
hastelib/src/hastegeo/core/artifact_storage/azure_blob_artifact_storage.py |
Implements cloud artifact operations. |
hastelib/src/hastegeo/core/artifact_storage/abstract_artifact_storage.py |
Extends artifact storage contract. |
hastelib/pyproject.toml |
Adds optional publishing dependencies. |
docker/docker-compose.yml |
Enables local publishing configuration. |
api/hastefuncqueues/tests/test_publishing_handlers.py |
Tests publishing queue handlers. |
api/hastefuncqueues/tests/__init__.py |
Initializes queue test package. |
api/hastefuncqueues/requirements.txt |
Updates queue dependencies. |
api/hastefuncqueues/function_app.py |
Registers publishing queue functions. |
api/hastefuncapi/tests/__init__.py |
Initializes API test package. |
Files not reviewed (1)
- ui/package-lock.json: Generated file
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Anthony Cintron (prbatero)
force-pushed
the
prbatero/feat/data-publishing-pc
branch
from
August 12, 2026 12:43
e5dc652 to
9a6d1a0
Compare
RC artifacts readyAll branch deployment references use the same RC tag:
|
…s, polling, privacy Resolve Copilot review findings on the publishing feature: - Embedding eligibility: source resolver gated on inferenceStatus, but embedding models signal completion via `status`. Gate on `status` for modelType "embedding", else inferenceStatus — otherwise every embedding publish was rejected before the embedding artifact logic ran. - Publish action on EmbeddingModelRow used inferenceStatus (never set for embeddings); use the already-computed isProcessed. - Owner detection: populate appParams.identityId from the SWA principal object id so it matches PublishedDataset.publishedByUser; previously identityId was never assigned, so non-admin publishers were never recognized as owners and Retry/Unpublish were hidden. - Publish dialog reported "Publishing started" on a 409 conflict (apiPut returns the status rather than throwing); surface the conflict as an error instead. - Published Datasets polling captured a stale query (page/filters/search/sort); drive the interval through a ref to the latest fetch and pause it while the search is not ready. - Persist the publisher email/login rather than the display name (PublishedDataset.publishedByName) — display names resolve from Entra at read. - Remove aria-modal=false from the publish dialog (it is a focus-trapping modal). - Provider registry: a Planetary Computer target is "configured" once the GeoCatalog URL is set; do not also require an ingestion source (needed only for private containers), so the public-container mode is configurable.
Anthony Cintron (prbatero)
force-pushed
the
prbatero/feat/data-publishing-pc
branch
from
August 12, 2026 13:09
9a6d1a0 to
7fca5ab
Compare
RC artifacts readyAll branch deployment references use the same RC tag:
|
Anthony Cintron (prbatero)
temporarily deployed
to
demo1
August 12, 2026 13:53 — with
GitHub Actions
Inactive
Publishing adds the queues app's first timer trigger (ReconcilePublishingOperations), which acquires a host-scoped Singleton lock. A colliding auto-generated host id (long app name / shared storage / deployment slots) then surfaces as "Unable to acquire Singleton lock" and NoScriptHost, leaving the app unhealthy. Document setting a unique AzureFunctionsWebHost__hostId per app/slot in the rollout operator notes. Applies to Local and PC (the timer ships with the Local feature).
Anthony Cintron (prbatero)
force-pushed
the
prbatero/feat/data-publishing-pc
branch
from
August 12, 2026 15:09
154fcd0 to
30f818a
Compare
RC artifacts readyAll branch deployment references use the same RC tag:
|
…proxy
Published copies were written to `{hash(projectId)}/published/{datasetId}/<file>`
— a five-segment blob path (data/hash/published/datasetId/file). The UI already
rewrites download URLs through the managed-identity storage proxy
(get-artifacts) via toBrowserStorageUrl, but that proxy only accepts four
segments, so the rewrite returned null and the UI fell back to the raw blob SAS
URL. On the VNet-only storage account (defaultAction: Deny) a direct blob
request from the browser is rejected with AuthorizationFailure.
Flatten the published prefix to `published/{datasetId}/<file>` (three
in-container segments). datasetId is a UUID, so the project-hash level was
redundant for uniqueness. Downloads now match the get-artifacts proxy and are
served on-network via managed identity, like every other HASTE artifact
download.
Note: datasets published before this change keep their old five-segment path and
must be re-published to be downloadable.
Anthony Cintron (prbatero)
force-pushed
the
prbatero/feat/data-publishing-pc
branch
from
August 12, 2026 15:38
30f818a to
6bdaa62
Compare
The section fetched its first page with fetchDatasets(false), so no loading indicator showed and the page rendered blank (the component returns null while items === null) until data arrived — noticeable on slower cloud loads. Trigger the shared full-page loading overlay on the first load (items === null), matching the Model Catalog pattern; later filter/search/sort/page changes still refetch silently so the overlay doesn't flash on every interaction.
Anthony Cintron (prbatero)
force-pushed
the
prbatero/feat/data-publishing-pc
branch
from
August 12, 2026 16:37
6bdaa62 to
8a13ae3
Compare
Anthony Cintron (prbatero)
temporarily deployed
to
demo1
August 12, 2026 17:20 — with
GitHub Actions
Inactive
added 4 commits
August 12, 2026 15:11
…aining submit) The initial publishing commit moved the module-level `from hastegeo.core.processors.train import TrainPreprocessor` import into a local import inside PutCancelModelQueueMessage, but PutRunModelQueueMessage (and the retrain path) still reference TrainPreprocessor at module scope. That raised an unhandled NameError on every training submit, surfaced to the client as a 500 from PutRunModelQueueMessage. Restore the top-level import and drop the now redundant local one.
Phase 2, stacked on -local. Adds the Planetary Computer publishing target using a vendored REST GeoCatalog client instead of the azure-planetarycomputer SDK — offline-verifiable, no proprietary-SDK coupling (per the spec decision). - geocatalog_client.py: hardened REST client + Entra auth (scope .../.default), redirect suppression, per-request timeouts, sanitized errors (never embeds server response bodies), api-version 2026-04-15. - planetary_computer_transport.py: resumable one-request-per-step adapter (start/continue create-collection/item, delete-item; get collection/item/ ingestion-source; asset signing) satisfying the provider's interface. Carries over the SSRF operation-URL pinning, endpoint validation, and failed-item accounting; FIXES the "Finished" terminal-status bug (was failing good ingests). Handles synchronous (201) collection creation vs async (202) items. - planetary_computer_provider.py + stac.py + schemas/: the resumable bounded- step provider and STAC builder (item geometry from the valid-area mask, ai4g: stats, one collection per project) reused verbatim from the audited implementation; provider gains a branch for sync collection creation. - config: PC_VERIFY_ATTEMPTS default 5->20 (bound 60). Continue-steps re-enqueue at ~30s, so this is a ~10 min default (30 min max) ingestion budget. - pyproject: planetary-computer extra -> vendored-REST deps (azure-identity, requests, pystac[validation], geopandas, pyogrio, pyproj, shapely). Tests: 97 publishing unit tests (incl. 16 new transport HTTP-fixture tests: SSRF pinning, Finished, failed-items, sanitization, sync/async, 404/409) + 25 API tests pass. Registry lazy-loads the provider and PC is disabled by default, so -local is unaffected when PC is off. Follow-up: end-to-end verification against a live GeoCatalog (confirm the /sas/sign asset-signing endpoint and sync-vs-async collection creation).
…ion role Stacked on the Local publishing infra wiring; adds the PC-target configuration so the Planetary Computer provider runs in Azure: - functions.bicep: PC_PROVIDER_ENABLED, PC_GEOCATALOG_URL, PC_EXPLORER_URL, PC_INGESTION_SOURCE, PC_COLLECTION_PREFIX app settings (new pc* params). The api-version and Entra token scope are code constants, not settings; auth is managed-identity only. - main.bicep / main.bicepparam: pc* params, default off/unset, sourced from HASTE_PC_* env vars. The GeoCatalog itself is operator-provisioned/external. - roles.bicep: conditional Storage Blob Data Reader on the HASTE storage account for an operator-supplied GeoCatalog ingestion managed identity (asset ingestion). Off by default (empty principal id); not needed for public containers or the SasToken ingestion path. The complementary grant (function-app identity on the external GeoCatalog data plane) is operator-owned and configured out-of-band; the exact GeoCatalog RBAC role needs verification against a live catalog before wiring. Note: infra/main.json (compiled ARM) not regenerated here to avoid a bicep CLI-version reformat; regenerate with the repo's pinned bicep version.
Resolve Copilot review findings on the Planetary Computer target: - Ingestion source is optional (public-container mode): the provider's info isConfigured and _require_configuration no longer require a non-empty PC_INGESTION_SOURCE, and start_publish only validates the source when one is set. A GeoCatalog URL alone now configures the target — matching the spec, the Bicep param, and verified live behavior (public blobs ingest without a source). - GeoCatalog client no longer appends api-version to absolute operation URLs that already carry it, avoiding a duplicate api-version query parameter on ingestion-operation polling. - deploy_apps.sh (GitHub Actions deploy path) now emits the PC_* Function App settings (PC_PROVIDER_ENABLED, PC_GEOCATALOG_URL, PC_EXPLORER_URL, PC_INGESTION_SOURCE, PC_COLLECTION_PREFIX) to match infra/modules/functions.bicep. - Remove the unused azure-planetarycomputer dependency from hastefuncqueues (the implementation uses the vendored REST client, not the SDK). Also aligns the provider's PC_VERIFY_ATTEMPTS upper bound (was 20) with the config's allowed maximum (60), so operator overrides above 20 no longer fail validation at publish time.
The PC provider's stac.py imports geopandas/pyproj (and reads GeoPackages via pyogrio, validates STAC via pystac). These are lazy-loaded only when resolving the Planetary Computer target, so they don't affect host startup — but a PC publish (queue worker) or a PC-target validate (api create path) would fail at runtime with ModuleNotFoundError. Add geopandas/pyproj/pyogrio (and pystac to the api) to both function apps' requirements so PC publishing works end to end. Local publishing does not import these and is unaffected.
Anthony Cintron (prbatero)
force-pushed
the
prbatero/feat/data-publishing-pc
branch
from
August 12, 2026 19:11
8a13ae3 to
697d356
Compare
RC artifacts readyAll branch deployment references use the same RC tag:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request adds a new publishing queue processing system to the
api/hastefuncqueuesAzure Functions app, introduces new dependencies for publishing support, and expands the artifact storage interface for improved cloud operations. It also includes new tests for the publishing handlers and updates configuration for local development and deployment.Publishing queue and handlers:
function_app.pyto process dataset publishing operations:GetPublishDatasetQueueMessage,GetPublishDatasetPoisonQueueMessage, andReconcilePublishingOperations, which handle publishing, poison queue messages, and reconciliation of stale operations, respectively.PublishingProcessorandPublishQueueMessageimports to support the new publishing workflow. [1] [2]Dependency and configuration updates:
azure-planetarycomputerandpystac[validation]torequirements.txtfor publishing support, and updatedazure-coreto 1.39.0. [1] [2]planetary-computeroptional dependency group inpyproject.tomlfor the publishing target, including geospatial libraries.docker-compose.ymlto include publishing-related environment variables for local development and testing. [1] [2]Artifact storage enhancements:
AbstractArtifactStorageinterface with new abstract methods for resolving paths, copying, deleting, checking existence, and getting metadata for artifacts, supporting more robust cloud operations.AzureBlobArtifactStorageinitialization for thread-safe container creation, added identity-based client, and madestore_artifactmethod more robust toNonevalues. [1] [2] [3] [4] [5]Testing:
test_publishing_handlers.pywith comprehensive tests for the publishing queue handlers, covering normal operation, error cases, and poison queue handling.Fixes #
Type of change
Checklist
cd hastelib && hatch run test:pytest) and the UI lints clean (cd ui && npm run lint)Testing
Additional context