feat(schema): split release Catalog into per-product shards to stop main-branch conflicts#656
Closed
typefield wants to merge 2 commits into
Closed
Conversation
The release Catalog was committed as a single 17.6 MB / 435k-line schema_catalog.json. Any feature PR that touched a command had to regenerate and commit the whole file, so two concurrent product PRs always collided on an unmergeable blob. Split the committed layout into a per-product tree: internal/cli/schema_catalog/catalog.json global envelope + Catalog map (~1 MB) internal/cli/schema_catalog/tools/<product>.json that product's leaf ToolSpecs A PR that touches a single product now only rewrites that product's shard plus the small global catalog.json, instead of the whole 12.7 MB Tools map. The global catalog.json still holds the irreducible cross-product aggregates (agent_metadata, interface_metadata, totals) and stays mergeable. The split is a storage concern only: - The loader embeds the directory and reassembles the exact same SchemaCatalogSnapshot, then runs the existing loadSchemaCatalogSnapshot validation unchanged. - source_hash is computed over Version+SurfaceHash+Catalog+Tools, so it is independent of how the JSON is partitioned; any missing, stale, or tampered shard still fails the content-hash check exactly as before. Policy jq queries keep their single-document interface via a new scripts/policy/with-catalog.sh helper that merges the shards back into one document. The drift check compares directory trees (diff -qr) and the go:generate / Makefile targets point at the new directory. Verified: go build ./..., full cli/generator/outputguard suites, generated-drift, schema-catalog policy, and the schema binary smoke check (built dws schema list/--all/<canonical> vs embedded catalog).
2 tasks
staticcheck flags one pre-existing ST1005 finding on main:
internal/generator/agentmetadata/metadata.go:372
fmt.Errorf("Agent hint directory is required")
-> fmt.Errorf("agent hint directory is required")
Go error strings must not be capitalized. Updates the one test that
asserts on this message text. No behavior change.
Verified: make lint (gofmt + go vet + staticcheck) passes.
typefield
pushed a commit
to typefield/dingtalk-workspace-cli
that referenced
this pull request
Jul 18, 2026
…l-AI#656 merge + fix structure test After merging PR DingTalk-Real-AI#656 (split schema_catalog.json into per-product shards): - Regenerated all catalog shards via `make generate-schema` to match the current cobra tree (devapp LeafSpec migration changes). The old shards were from DingTalk-Real-AI#656's branch (older main), causing source_hash mismatch → embedded catalog load failure. - Fixed schema_catalog_structure_test.go: reference to deleted `embeddedSchemaCatalogJSON` replaced with `json.Marshal(embeddedSchemaCatalog().Snapshot)` — validates the reassembled snapshot from the new per-product shard embed. Verified: all safety tests pass, TestEmbeddedSchemaCatalogStructure passes, drift ok, make build succeeds.
typefield
pushed a commit
to typefield/dingtalk-workspace-cli
that referenced
this pull request
Jul 18, 2026
…umption separation
== LeafSpec command framework (internal/helpers/leaf.go) ==
Declarative command construction: LeafSpec/LeafFlag/NewLeafCommand with
Call (pluggable dispatch), LeafInt, PostMount (hook), Trim, Validate.
Collapses per-command hand-written required validation, alias/env fallback,
value transform, toolArgs assembly into one declarative path.
== devapp migration (28/31 commands) ==
All MCP-direct devapp leaf commands migrated to LeafSpec. Factories
(devAppCall/devAppCallCursor/devAppMeta) fold 33 repeated closures.
fakeDevAppRunner test harness asserts toolArgs for all migrated commands.
4 complex commands (delete/robot submit/result/config) kept hand-written
with documented rationale.
== Schema generation/consumption separation ==
- gen.go: isolated //go:generate pragmas from business code (single entry).
- command_meta.go: ResolveMeta(cliPath) → CommandMeta{Identity,Safety,Selection}
unified consumption API. All consumers (help/schema/agent) call one function.
- command_safety.go: SafetyForCLIPath + RenderSafetyAnnotation — safety
metadata flows from embedded catalog into --help output.
- calendar.go HelpFunc fix: delegates to root's HelpFunc at help-time
(not stale capture) so Safety annotations reach calendar commands.
- schema_catalog_structure.go: closed catalog structure validation gate.
== MCP metadata refresh tool ==
cmd/fetch_mcp_metadata: iterates 26 MCP server endpoints via tools/list,
merges with previous data for cross-server interface_ref, fills registry
stubs. make fetch-mcp-metadata target. Utility for future baseline refresh.
== PR DingTalk-Real-AI#656 merge (per-product catalog shards) ==
schema_catalog.json (single file) → schema_catalog/ (catalog.json +
tools/<product>.json shards). Reduces merge conflicts on concurrent PRs.
== AGENTS.md ==
Documents the generation/consumption split: gen.go (entry), 6 inputs →
catalog, ResolveMeta (consumption), fetch-mcp-metadata (refresh).
Verified: make policy exit 0, drift zero, all tests pass.
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
internal/cli/schema_catalog/catalog.json(global envelope, ~1 MB) plus oneinternal/cli/schema_catalog/tools/<product>.jsonshard per product (22 shards) — instead of a single 17.6 MB / 435k-lineinternal/cli/schema_catalog.json. The loader embeds the directory and reassembles the sameSchemaCatalogSnapshot; all generation-time gates,source_hashintegrity, and policy checks are unchanged. Also folds in a one-line staticcheck ST1005 fix in an unrelated file somake lintgoes fully green.source_hash). A PR touching one product now rewrites only that product's shard.Conflict surface:
tools/aitable.json(3.2 MB) +catalog.json(1 MB)tools/calendar.json(654 KB) +catalog.json— disjoint shard, mergeableWhy it's safe (storage-only):
source_hashis computed overVersion+SurfaceHash+Catalog+Tools, independent of JSON partitioning, so any missing/stale/tampered shard still fails the content-hash check exactly as before. The split happens only at the file-write step;BuildSchemaCatalogSnapshot,ValidateSchemaDeliveryInvariants, andloadSchemaCatalogSnapshotare untouched. Policyjqqueries keep their single-document interface via a newscripts/policy/with-catalog.shreassembly helper.Verification
make buildmake lint— gofmt + go vet + staticcheck all pass (includes the folded ST1005 fix inagentmetadata/metadata.go)make test— ALL PASSED (cli, contract, integration/extensions, scripts; 0 failed)make policy— ok (22 products, 564 tools; incl. command-registry, command-surface, generated-drift, schema-catalog, schema-binary, agent-examples dry-run)./scripts/policy/check-generated-drift.sh— deterministic (two runs byte-identical) + matches committed./scripts/policy/check-command-surface.sh --strict— N/A (no command path/flag/visibility change in this PR)Notes
toolsmap is partitioned per product. The 1 MB globalcatalog.jsonstill gets touched by every product PR because it holds irreducible cross-product aggregates (agent_metadata,interface_metadata, totals). It is small and its diff is localized + mergeable — the unmergeable monster is eliminated. Fully partitioning the global aggregates would require recomputing them at load time (duplicating generation logic) and is deliberately out of scope.agentmetadata/metadata.go:372error string lowercased (the only staticcheck finding onmain); CI does not enforce staticcheck today, but this keeps localmake lintgreen. Kept as a separate commit on this branch for review clarity.cmd_schema_catalog/main.go) writes shards; loader (schema_catalog.go) embeds+reassembles; newscripts/policy/with-catalog.sh; 4 policy scripts repointed;Makefile+//go:generate→ directory output; tests adapted;AGENTS.md,docs/schema-dynamic-endpoint-design.md,CHANGELOG.mdsynced; deletes oldschema_catalog.json.