Prbatero/feat/data publishing local - #122
Open
Anthony Cintron (prbatero) wants to merge 13 commits into
Open
Conversation
added 7 commits
August 11, 2026 13:37
…lease
Bucket A (additive) from the -sol audit split, Phase 1 (Local only; no PC):
- core/publishing/: base ABC + registry (PC lazy-loaded, disabled by default),
repository (per-dataset docs + optimistic concurrency), source (eligibility +
artifact resolution), lease (blob-lease coordinator), local_provider (immutable
copy to published/{datasetId}/).
- models/publishing.py; processors/publishing.py + assessment.py.
- Additive infra: config publishing block + PUBLISHED_DATASET type + publish
queue; data-layer load_bounded/load_page/update_locked; artifact-storage
copy/scoped-download/etag/size/exists/delete_prefix; blob.py max_bytes.
- hastefuncqueues publish trigger.
- Phase-1 unit tests (registry/repository/source/local_provider/lease): 45 pass.
Excludes Bucket B: reverted the high-risk load_all/load_all_from_partition
rewrite to main's versions; endpoint hardening + DTOs left for a separate
-hardening PR. See spec/features/data-publishing/sol-classification.md.
Bucket A only. Adds to api/hastefuncapi/function_app.py: publishing imports, active-caller/authorization helpers, the DeleteProject published-dataset guard, and 7 publishing routes (GetPublishingProviders, GetPublishDatasetOptions, GetPublishedDatasets, GetPublishedDataset, PutPublishDatasetQueueMessage, PutRetryPublishedDatasetQueueMessage, DeletePublishedDataset). All existing endpoints reverted to main's versions — the Bucket-B endpoint hardening (IDOR checks, strict *Request DTOs, server-managed-field enforcement, assessment size caps) is deferred to a separate -hardening PR. Tests: 19 publishing route tests + 6 queue-handler tests pass. Existing core suite: 191 pass (3 failures are pre-existing on main — clipBbox mocks).
Bucket A UI, gated behind the publishingEnabled flag: - New: PublishedDatasets section, PublishedDatasetRow, PublishDatasetModal (name/description prefill, asset checklist, provider-driven target dropdown), util/publishing.js, util/assessmentSummary.js (extracted from AssessmentReportModal). - Additive wiring: App/AppContext (publishingEnabled + provider fetch), AppBody route + AppSidebar nav, "Publish dataset..." in ModelResultsButton + EmbeddingModelRow, Database icon. - api.js: apiPut/apiDelete now accept any 2xx (async 202 queue responses) — minimal additive change; the shared-helper ApiError rewrite is NOT taken (deferred to -hardening). Existing 200 callers unaffected. Validated: full `vite build` compiles cleanly (all imports resolve). Vitest unit tests + config are preserved on -sol/archive and land in a follow-up UI-test-infra slice (needs vitest devDeps + package-lock regen).
UI (Published Datasets section): - Clickable Project/Layer cell -> opens the source project with that layer expanded (/project/:projectId/:imageLayerId). - Labeled "Actions" button (replacing the bare "...") and an "Actions" column header; keep the menu (download / open in Explorer / retry / unpublish). - New "View details" action -> modal with the generated description, project/ layer, model, target, status, publisher, headline assessment stats, and the published assets (kind + size). - "Published by" now mirrors the project Creator display: the publisher's login/email via limitTextLength, not the opaque object id. - Desktop column widths moved into a @media (min-width: 993px) rule scoped to .pgrid-page--published-datasets so the mobile card layout (inherited from .pgrid-page--model-catalog) is preserved; font URLs forced to https (CSP). Backend: - Capture the publisher's display name (client-principal userDetails) at publish time as PublishedDataset.publishedByName, so the UI can show a human name while the ownership check still uses the object id.
Add the data-publishing feature flag and queue name to the api/queues
Function App settings so the Local target runs in Azure:
- functions.bicep: PUBLISH_QUEUE_NAME ('publish-queue') and
PUBLISHING_ENABLED (from a new publishingEnabled param) in appConfigSettings.
- main.bicep / main.bicepparam: publishingEnabled param, default off,
sourced from HASTE_PUBLISHING_ENABLED.
The publish-queue and the publishing-locks blob container are auto-created at
runtime, and APIM operations are auto-synced from the deployed function app by
the postdeploy hook, so no queue/container/APIM resources are needed here.
Other Local knobs (PUBLISH_MAX_TOTAL_BYTES, PUBLISHED_DOWNLOAD_SAS_MINUTES,
PUBLISHING_LOCK_CONTAINER) use code defaults.
Note: infra/main.json (compiled ARM artifact) is intentionally not regenerated
here to avoid a bicep CLI-version reformat; regenerate it with the repo's
pinned bicep version.
Add the publishing feature flag + queue/lock settings to the api and queues services so `docker compose up` exercises Local publishing against Azurite (the publish-queue and publishing-locks container auto-create at runtime). PC_PROVIDER_ENABLED stays false — no local GeoCatalog.
Drop the accidental "install" and "npm" entries from the UI dependencies (and their lockfile trees). Neither is imported anywhere; the npm CLI as a runtime dependency is what caused local image builds to fetch an npm tarball. Mirrors the earlier cleanup in #64 before they regressed back in.
Copilot started reviewing on behalf of
Anthony Cintron (prbatero)
August 11, 2026 19:11
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an asynchronous dataset-publishing workflow spanning the API, queue worker, storage layer, infrastructure, and UI.
Changes:
- Adds publishing models, persistence, locking, artifact handling, API routes, and queue triggers.
- Adds publish/catalog UI workflows with status polling and local downloads.
- Adds deployment configuration and backend tests.
Reviewed changes
Copilot reviewed 54 out of 57 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
ui/src/util/publishing.js |
Adds publishing status and artifact helpers. |
ui/src/util/icons.jsx |
Adds database icon support. |
ui/src/util/assessmentSummary.js |
Extracts assessment-summary formatting. |
ui/src/util/api.js |
Supports additional successful HTTP statuses. |
ui/src/Components/PublishedDatasets.jsx |
Adds the published-dataset catalog. |
ui/src/Components/PublishedDatasetRow.jsx |
Adds dataset details and actions. |
ui/src/Components/PublishDatasetModal.jsx |
Adds the publishing dialog. |
ui/src/Components/ProjectManagement/ModelResultsButton.jsx |
Adds publishing for model results. |
ui/src/Components/ProjectManagement/EmbeddingModelRow.jsx |
Adds publishing for embedding results. |
ui/src/Components/BuildingValidation/AssessmentReportModal.jsx |
Reuses the summary helper. |
ui/src/Components/AppSidebar.jsx |
Adds catalog navigation. |
ui/src/Components/AppBody.jsx |
Registers the catalog route. |
ui/src/assets/css/style.css |
Styles the dataset table and font URLs. |
ui/src/AppContext.jsx |
Adds publishing state. |
ui/src/App.jsx |
Loads publishing capabilities. |
ui/package.json |
Removes unnecessary install dependencies. |
infra/modules/functions.bicep |
Configures publishing for Function Apps. |
infra/main.bicepparam |
Reads the publishing feature flag. |
infra/main.bicep |
Passes publishing configuration. |
hastelib/tests/core/publishing/test_source.py |
Tests source resolution. |
hastelib/tests/core/publishing/test_repository.py |
Tests persistence and concurrency. |
hastelib/tests/core/publishing/test_registry.py |
Tests provider registration. |
hastelib/tests/core/publishing/test_local_provider.py |
Tests local publishing. |
hastelib/tests/core/publishing/test_lease.py |
Tests blob leases. |
hastelib/src/hastegeo/core/utils/blob.py |
Adds bounded streaming downloads. |
hastelib/src/hastegeo/core/publishing/source.py |
Resolves publishable artifacts. |
hastelib/src/hastegeo/core/publishing/repository.py |
Persists published datasets. |
hastelib/src/hastegeo/core/publishing/registry.py |
Registers publishing targets. |
hastelib/src/hastegeo/core/publishing/local_provider.py |
Implements local publishing. |
hastelib/src/hastegeo/core/publishing/lease.py |
Coordinates publishing operations. |
hastelib/src/hastegeo/core/publishing/base.py |
Defines the provider contract. |
hastelib/src/hastegeo/core/publishing/__init__.py |
Exports publishing primitives. |
hastelib/src/hastegeo/core/processors/metadata.py |
Adds bounded and paginated reads. |
hastelib/src/hastegeo/core/processors/assessment.py |
Generates bounded assessment reports. |
hastelib/src/hastegeo/core/models/publishing.py |
Defines publishing schemas and states. |
hastelib/src/hastegeo/core/data_layer/unified.py |
Exposes paginated storage operations. |
hastelib/src/hastegeo/core/data_layer/local_file_system_data_layer.py |
Adds bounded local reads. |
hastelib/src/hastegeo/core/data_layer/azure_postgresql_data_layer.py |
Adds bounded PostgreSQL reads. |
hastelib/src/hastegeo/core/data_layer/azure_data_lake_data_layer.py |
Adds bounded Data Lake reads. |
hastelib/src/hastegeo/core/data_layer/azure_cosmos_db_data_layer.py |
Adds bounded Cosmos reads. |
hastelib/src/hastegeo/core/data_layer/azure_blob_storage_data_layer.py |
Adds indexed blob pagination. |
hastelib/src/hastegeo/core/data_layer/abstract_data_layer.py |
Extends the data-layer contract. |
hastelib/src/hastegeo/core/artifact_storage/unified_artifact_storage.py |
Exposes publishing storage operations. |
hastelib/src/hastegeo/core/artifact_storage/local_file_system_artifact_storage.py |
Implements local artifact operations. |
hastelib/src/hastegeo/core/artifact_storage/azure_blob_artifact_storage.py |
Implements blob copy and scoped downloads. |
hastelib/src/hastegeo/core/artifact_storage/abstract_artifact_storage.py |
Extends the artifact-storage contract. |
docker/docker-compose.yml |
Enables publishing locally. |
api/hastefuncqueues/tests/test_publishing_handlers.py |
Tests publishing queue handlers. |
api/hastefuncqueues/tests/__init__.py |
Initializes the queue test package. |
api/hastefuncqueues/requirements.txt |
Updates publishing dependencies. |
api/hastefuncqueues/function_app.py |
Adds publishing queue triggers. |
api/hastefuncapi/tests/test_publishing_routes.py |
Tests publishing API routes. |
api/hastefuncapi/tests/__init__.py |
Initializes the API test package. |
api/hastefuncapi/function_app.py |
Adds publishing endpoints and authorization. |
Files not reviewed (1)
- ui/package-lock.json: Generated file
Suppressed comments (1)
ui/src/Components/PublishedDatasetRow.jsx:92
apiPutrepresents every 409 as the numeric value409, so thisawaitis treated as success. A retry rejected because of a stale/active operation silently refreshes the unchanged row instead of showing the existing “Retry failed” dialog.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
RC artifacts readyAll branch deployment references use the same RC tag:
|
… deploy paths
The Config drift check failed with:
Dead application settings (set but never read):
PUBLISHING_ENABLED (set by infra/modules/functions.bicep, read by no code)
That was a false positive. PUBLISHING_ENABLED is read in hastegeo.core.config
via _get_bool_env("PUBLISHING_ENABLED", False), but check_env_drift.py's AST
scanner only recognized os.getenv, os.environ.get, and os.environ[...]. The new
typed wrappers _get_bool_env and _get_bounded_int_env were invisible to it, so
no reader was found and the setting looked dead.
The blind spot covered five variables, not one: PC_PROVIDER_ENABLED,
PUBLISH_MAX_TOTAL_BYTES, PUBLISHED_DOWNLOAD_SAS_MINUTES and PC_VERIFY_ATTEMPTS
are read through the same helpers and were equally unseen. They simply did not
error because no deploy path emits them yet.
Teach the scanner to treat _get_*_env(NAME, ...) calls as reads. The wrappers
supply their own default in the signature, so such a read counts as optional
even when the call site passes no default.
Separately, fix the real drift the check could not see: PUBLISHING_ENABLED was
emitted by functions.bicep but not by deploy_apps.sh, so environments deployed
through the GitHub Actions path would never receive it and would silently fall
back to false while Bicep-provisioned environments got the flag. Emit it from
deploy_apps.sh too, defaulting off to match the Bicep param, and wire it
through deploy-apps.yml as a GitHub Environment variable (non-sensitive feature
flag, same treatment as VITE_SHOW_FOOTER).
Contributor
|
Pushed The failure was a false positive. The blind spot covered five variables, not one — Plus one real issue the check couldn't see. To turn publishing on for an environment: I deliberately did not add it to All 13 checks green. Nothing else in the PR touched. |
RC artifacts readyAll branch deployment references use the same RC tag:
|
…s, polling, privacy Resolve Copilot review findings on the publishing feature: - Embedding eligibility: source resolver gated on inferenceStatus, but embedding models signal completion via `status`. Gate on `status` for modelType "embedding", else inferenceStatus — otherwise every embedding publish was rejected before the embedding artifact logic ran. - Publish action on EmbeddingModelRow used inferenceStatus (never set for embeddings); use the already-computed isProcessed. - Owner detection: populate appParams.identityId from the SWA principal object id so it matches PublishedDataset.publishedByUser; previously identityId was never assigned, so non-admin publishers were never recognized as owners and Retry/Unpublish were hidden. - Publish dialog reported "Publishing started" on a 409 conflict (apiPut returns the status rather than throwing); surface the conflict as an error instead. - Published Datasets polling captured a stale query (page/filters/search/sort); drive the interval through a ref to the latest fetch and pause it while the search is not ready. - Persist the publisher email/login rather than the display name (PublishedDataset.publishedByName) — display names resolve from Entra at read. - Remove aria-modal=false from the publish dialog (it is a focus-trapping modal). - Provider registry: a Planetary Computer target is "configured" once the GeoCatalog URL is set; do not also require an ingestion source (needed only for private containers), so the public-container mode is configurable.
RC artifacts readyAll branch deployment references use the same RC tag:
|
added 2 commits
August 12, 2026 11:09
Publishing adds the queues app's first timer trigger (ReconcilePublishingOperations), which acquires a host-scoped Singleton lock. A colliding auto-generated host id (long app name / shared storage / deployment slots) then surfaces as "Unable to acquire Singleton lock" and NoScriptHost, leaving the app unhealthy. Document setting a unique AzureFunctionsWebHost__hostId per app/slot in the rollout operator notes. Applies to Local and PC (the timer ships with the Local feature).
…proxy
Published copies were written to `{hash(projectId)}/published/{datasetId}/<file>`
— a five-segment blob path (data/hash/published/datasetId/file). The UI already
rewrites download URLs through the managed-identity storage proxy
(get-artifacts) via toBrowserStorageUrl, but that proxy only accepts four
segments, so the rewrite returned null and the UI fell back to the raw blob SAS
URL. On the VNet-only storage account (defaultAction: Deny) a direct blob
request from the browser is rejected with AuthorizationFailure.
Flatten the published prefix to `published/{datasetId}/<file>` (three
in-container segments). datasetId is a UUID, so the project-hash level was
redundant for uniqueness. Downloads now match the get-artifacts proxy and are
served on-network via managed identity, like every other HASTE artifact
download.
Note: datasets published before this change keep their old five-segment path and
must be re-published to be downloadable.
RC artifacts readyAll branch deployment references use the same RC tag:
|
RC artifacts readyAll branch deployment references use the same RC tag:
|
The section fetched its first page with fetchDatasets(false), so no loading indicator showed and the page rendered blank (the component returns null while items === null) until data arrived — noticeable on slower cloud loads. Trigger the shared full-page loading overlay on the first load (items === null), matching the Model Catalog pattern; later filter/search/sort/page changes still refetch silently so the overlay doesn't flash on every interaction.
RC artifacts readyAll branch deployment references use the same RC tag:
|
…aining submit) The initial publishing commit moved the module-level `from hastegeo.core.processors.train import TrainPreprocessor` import into a local import inside PutCancelModelQueueMessage, but PutRunModelQueueMessage (and the retrain path) still reference TrainPreprocessor at module scope. That raised an unhandled NameError on every training submit, surfaced to the client as a 500 from PutRunModelQueueMessage. Restore the top-level import and drop the now redundant local one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces a new publishing queue processing system to the
hastefuncqueuesAPI, adds support for publishing operations in the artifact storage layer, and enhances local development/test support for publishing workflows. The main changes include new queue handlers for publishing, updates to the artifact storage abstraction, and supporting infrastructure and dependency updates.Publishing queue processing and handlers:
function_app.pyfor handling publishing queue messages, poison queue messages, and periodic reconciliation of publishing operations, utilizing the newPublishingProcessorandPublishQueueMessagetypes.test_publishing_handlers.pywith isolated async tests for the publishing queue handlers, including message decoding, error handling, poison queue logic, and reconciliation.Artifact storage abstraction and Azure Blob implementation:
AbstractArtifactStorageinterface with new methods for resolving artifact paths, copying artifacts, deleting by prefix, checking existence, getting size/etag, and generating scoped download URLs.AzureBlobArtifactStorageto implement the new interface methods, improved initialization and container creation logic, and fixed input validation instore_artifact. [1] [2] [3] [4] [5]Dependency and configuration updates:
azure-planetarycomputerandpystac[validation], and bumpedazure-coreversion. [1] [2]docker-compose.ymlto include new environment variables for publishing queue configuration and enabled publishing features for local development. [1] [2]Fixes #
Type of change
Checklist
cd hastelib && hatch run test:pytest) and the UI lints clean (cd ui && npm run lint)Testing
Additional context