feat(services): add a shared read-only org metadata catalog - W-23613533

[peternhale]

What does this PR do?

This PR introduces OrgMetadataCatalog in the Services extension as the shared, read-oriented authority for metadata facts about the active org. It gives extensions one service for discovering metadata, correlating it with workspace source, observing changes, and opening org-only source while allowing each extension to retain ownership of its domain-specific behavior and UI.

OrgMetadataCatalog

The catalog provides a common model for:

• Metadata types, components, folders, SObjects, fields, and source-tracking observations.
• Org and workspace presence, correlated against live project source.
• Stable metadata references and document URIs partitioned by org ID.
• Cached reads, explicit refresh, semantic invalidation, and deduplicated change notifications.
• Lazy org-source materialization and read-only editor documents.
• Persisted remote observations with workspace presence recalculated from the current project.

It uses the source best suited to each fact:

Fact	Authority
Metadata membership, folders, manageable state, and modification timestamps	Metadata API inventory
SObject and field details	REST SObject describe
Runnable Apex tests	Tooling Test Discovery, consumed by Apex Testing
Readable Apex source	Tooling API when appropriate
Remote revision state	Source tracking
Workspace presence and local navigation	Current project source

Metadata API inventory remains authoritative for membership and timestamps. REST descriptions enrich SObjects and Custom Fields with details such as type, length, precision, scale, and relationship name without deciding whether metadata exists.

Org-only source is materialized lazily into a revision-aware shadow store. The catalog exposes that content through a read-only TextDocumentContentProvider using sf-org-metadata: URIs, allowing editor navigation, language support, formatting, highlighting, and CodeLens without exposing the shadow location to consumers. Non-Apex metadata retrieval is grouped by metadata type.

Remote observations are persisted as formatted JSON under .sf/orgs/<orgId>/metadata-catalog/catalog.json. State is hydrated once per org, while workspace presence is always rescanned rather than trusted from persisted data. Inventory acquisition, persistence, and notifications are coalesced to prevent duplicate requests, file churn, and refresh storms.

The public OrgMetadataCatalog service remains the consumer boundary. Its implementation is decomposed into private modules for state, workspace correlation, inventory acquisition, SObjects, tree projection, remote retrieval, source materialization, documents, and source tracking. OrgMetadataCatalogChangePubSub provides consumers with deduplicated catalog-change notifications.

Extension consumption

Extension or service	How it uses the catalog	Responsibility it retains
Org Browser	Queries catalog projections for types, folders, components, Custom Fields, presence, cached filtering, and refresh; listens for catalog changes	Hierarchy presentation, filters, view state, prompts, and retrieval UX
Apex Testing	Resolves discovered tests to catalog presence and source identity; opens org-only source through catalog documents	Tooling Test Discovery, runnable-test authority, Test Explorer hierarchy, and test execution
Metadata	Uses catalog timestamps, source-tracking observations, and fresh grouped source materialization for conflict detection and diff	Deploy/retrieve/diff command orchestration and presentation
SOQL	Uses catalog-backed SObject listing and description for completion and editor features	Query editing, completion presentation, and execution
Services deploy/retrieve/tracking lifecycle	Publishes observations and semantic invalidations into the catalog	Salesforce transport and lifecycle coordination

Org Browser

• Discovers metadata types, folders, components, and Custom Object → Custom Field children through catalog projections.
• Uses catalog org/workspace presence for Local and Org filtering and local-presence decorations.
• Supports manageable-state filtering plus exact, wildcard, regex, and cross-type filters.
• Uses cached inventories for broad filtering unless the user approves acquisition.
• Refresh invalidates the requested catalog slice, and catalog notifications update the view after deploys, retrieves, workspace changes, and tracking observations.
• Metadata type retrieval operates on the currently visible filtered component set.

Apex Testing

• Keeps Tooling Test Discovery as the authority for runnable tests while using the catalog for presence and source identity.
• Removes the Apex Testing discovery VFS and private org-source cache.
• Opens workspace source when present and lazy sf-org-metadata: source when the test class is org-only.
• Reconciles Test Explorer items after local creation, deletion, or download without duplicate classes or unnecessary tree collapse.
• Keeps the Download CodeLens workflow and transitions navigation to the retrieved workspace URI.

Metadata, tracking, and SObjects

• Diff requests use fresh catalog source materialization rather than depending on periodic source-status refresh.
• Folder diff retrieval remains grouped by metadata type.
• Non-tracking conflict detection uses catalog inventory timestamps.
• Deploy/retrieve results and workspace changes produce semantic invalidations before persistence and publication.
• Source-tracking status and remote revision comparison use catalog observations while retaining one coherent tracking update event.
• Refresh SObjects preserves custom and standard artifact folders when refreshed separately.

SOQL

• SObject discovery and description are routed through the catalog.
• Completion and editor behavior continue to consume the resulting SObject model without owning a separate org-data cache.

API and ownership boundaries

• The catalog owns active-org metadata facts, observations, correlation, persistence, invalidation, and org-only document projection.
• Consumers own their UI, commands, domain projections, and explicit retrieval into the workspace.
• There is no org-metadata FileSystemProvider, public content-write API, writer registration API, or requirement that consumers share one tree or filesystem layout.

The architectural decision, implementation plan, and manual test matrix are recorded in:

• packages/salesforcedx-vscode-services/docs/adr/0001-org-catalog-over-shared-vfs.md
• docs/adr/0021-org-metadata-catalog.md
• docs/superpowers/plans/2026-07-30-org-metadata-catalog-generalization.md
• docs/superpowers/plans/2026-08-03-orgcatalog-service-decomposition.md
• docs/superpowers/manual-tests/2026-07-31-org-metadata-catalog.md

What issues does this PR fix or reference?

@W-23613533@

Functionality Before

• Extensions independently requested or retained overlapping org metadata state.
• Apex Testing maintained a discovery VFS and org-source cache.
• Org-only editor integration was coupled to consumer-owned virtual storage.
• Metadata diff, tracking, SObject refresh, deploy/retrieve, and workspace changes did not feed one coherent metadata lifecycle.

Functionality After

• Extensions consume one Services-owned catalog while retaining domain-specific behavior and presentation.
• Metadata acquisition sources have explicit authority and freshness boundaries.
• Org-only source opens lazily through sf-org-metadata:; explicit download places source in the workspace and updates consumers coherently.
• Refresh, diff, deploy/retrieve, file changes, and target-org changes invalidate the relevant catalog slices and publish deduplicated updates.
• Catalog observations persist across restart and remain partitioned by org, while workspace presence reflects the current project.

Validation

Automated:

• The latest full repository pre-commit workflow passed: 51 scripts run and 40 unchanged scripts skipped.
• The latest full pre-push workflow passed: 57 scripts run and 111 unchanged scripts skipped.
• Services and consumers compile with the generated public catalog types.
• Lint, circular-dependency checks, package-lock validation, and Effect diagnostics pass.
• Catalog coverage includes persistence/hydration, org partitioning, acquisition coalescing, overlapping invalidation, Custom Field dual-source projection, foldered metadata, grouped source materialization, tracking deduplication, and document URI behavior.
• Consumer coverage includes Org Browser hierarchy/filter/refresh behavior, Apex Testing presence and navigation reconciliation, Metadata conflict and diff behavior, SObject artifact preservation, and SOQL completion.

Manual:

• Verified Org Browser discovery, filtering, refresh, retrieval, Custom Fields, and local/org presence transitions.
• Verified Apex Test Explorer local deletion to ephemeral source to download, with no duplicate items and no tree collapse.
• Verified single-file and folder metadata diff, including fresh remote changes and grouped retrieval notifications.
• Verified deploy to a fresh scratch org followed by catalog-backed discovery.
• Verified custom and standard Refresh SObjects results coexist.
• Verified catalog persistence, restart hydration, workspace rescanning, source-tracking updates, and target-org partitioning.

Reviewer focus

• Are the authority and freshness boundaries correct for Metadata API, REST SObject describe, Tooling API, source tracking, and workspace source?
• Does the public catalog API expose the shared metadata facts consumers need without absorbing their domain-specific responsibilities?
• Are invalidation, persistence, and notification ordering sufficient to prevent stale reads, duplicate work, and refresh storms?
• Do the consumer migrations remove duplicated org-state ownership while preserving each extension's established behavior?
• Does any current requirement need filesystem semantics beyond the registered read-only document scheme?

[mshanemc]

why not use Registry to get these? What's special about class/trigger

[mshanemc]

please use Schema/struct stuff. You get the isWhatever for free.