feat(tests): validate the catalog corpus against musher-dev/spec - #1
Merged
Conversation
Precedes the tooling that needs them: node_modules/ and *.tsbuildinfo for the test suite, .devcontainer/.env and .task/ for the container and task runner. The .env entry matters most — the template beside it is committed, the filled-in copy never is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The seed commit left this repository with no validation of its own, on the
grounds that the platform is the sole validator. It still is — but "the
platform will reject it" is a slow way to learn that an item is malformed,
and nothing here caught it at commit time.
These tests apply the same contracts earlier. They are not a second
authority: the schemas are fetched at run time from the tip of the public
musher-dev/spec repository rather than vendored, so the corpus is judged
against the contract as it currently stands. A copy held here would drift,
and a stale copy is worse than none — it passes a corpus the live spec
would reject, silently. The repository is public, so no credential is read.
Covers the three phases component spec §7 lets a client decide offline:
parser the Musher YAML profile — one document per file, string keys,
no anchors, aliases, merge keys or tags, and the bounds
structural each document against its family's fetched JSON Schema
semantic the cross-document rules — identity agreement, reference
resolution, path containment, media, the description Markdown
profile, image pinning, endpoint resolution, connection
compatibility, parameter coverage
`capability` is deliberately absent. Whether a Compute Profile is offered,
whether a published component exists and whether a version is monotonic are
decided against the platform catalog over the network, and an implementation
MUST NOT report a rule it cannot check. An item green here can still be
rejected at sync.
rules.test.ts guards the guards: a suite that passes on a clean corpus
passes just as readily when a rule is unreachable, so each case breaks one
thing in a synthetic item and asserts the normative diagnostic fires.
Nothing is restated from the spec where it can be read from it — the media
path grammar and the component value-schema defaults are pulled out of the
fetched bundles at run time, so they cannot drift from their definition.
Run by Node's type stripping and built-in test runner: no build step, no
framework. 377 tests across the 13 items.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two workflows, for two things that fail differently. validate typechecks, lints the workflows and runs the suite on every push and pull request. It also runs on a daily schedule, which is the part that only makes sense because the schemas are not vendored: a spec change can stop accepting an item that used to validate, and without a scheduled run nothing notices until the next catalog PR happens to come along. devcontainer covers the thing nobody can check without building it — the scripts, the lockfile against devcontainer.json, and the .env template against what the tasks actually read. Path-filtered, so it stays quiet on an items-only change. Both are contents: read and take no credential. musher-dev/spec is public. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gives the repository a reproducible environment and the same checks CI runs, locally. Taskfile wraps the npm scripts rather than replacing them: `npm test` stays what CI runs and what works in a bare checkout with no Task installed, and `task` is the ergonomics layer on top. One taskfile, not the split musher-dev/spec uses, because there is one build surface here. The lefthook split is about the network. `npm test` fetches the spec bundles on every run, so it cannot be a pre-commit job — a commit made offline would fail for a reason that has nothing to do with the commit. Pre-commit holds what is local and fast (types, shell, workflows); pre-push holds the suite. commit-msg enforces Conventional Commits. .env.example is committed and .env is not, so env:check reports the keys a stale local copy is missing rather than letting a task read an unset one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validation section said this repository runs no validation of its own and that a mistake is not caught at commit time. Both stopped being true; it now says what runs, what is fetched at run time and why, and which phase is deliberately missing. The platform is still named as the sole authority — the tests are the same contracts applied early, not a second set — and the one-item-per-PR guidance stays, since that is what makes a rejection only the platform can raise easy to attribute. Two details the suite forced into the open while it was being written: blueprint component references are repo-local paths that must begin ./ and end .yaml, not bare stems. The prefix is load-bearing — a bare name is not distinguishable from the UUID a published reference uses, so without it no validator can tell which resolver the reference wanted. The example said `component: my-app`, which the structural phase rejects. A connection's two ends must agree on schema.type, and on semanticType wherever the consuming input names one. The multi-service paragraph said only that connections wire outputs to inputs, which is the shape of the thing and not the constraint on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The seed commit left this repository deliberately unvalidated — the platform
was the sole validator, and no CI ran here. The platform is still the sole
authority, but "the platform will reject it" is a slow way to learn that
an item is malformed. This adds the same contracts, applied early.
Not a second authority
Schemas are fetched at run time from the tip of the public
musher-dev/spec, never vendored. A copyheld here would be a second answer to "what is the contract" — and a stale copy
is worse than none, because it passes a corpus the live spec would reject,
silently. Nothing about the source is configurable, and if the fetch fails the
suite fails loudly, naming the URL. The repo is public, so no credential is
read anywhere in the suite.
What runs
spec.test.tsparser.test.tsparserstructural.test.tsstructuralsemantic.test.tssemanticlayout.test.tsrules.test.tscapabilityis deliberately absent. Whether a Compute Profile is offered,whether a published component exists, whether a version is monotonic — all
decided against the platform catalog over the network, and an implementation
MUST NOT report a rule it cannot check. An item green here can still be
rejected at sync.
rules.test.tsguards the guards: a suite that passes on a clean corpus passesjust as readily when a rule is silently unreachable, so each case breaks one
thing and asserts the normative diagnostic fires.
Also included
validateon push/PR plus a daily schedule (the part that only makessense because schemas aren't vendored: it catches a spec change that stops
accepting an item, in the window before the next catalog PR).
devcontaineris path-filtered and stays quiet on items-only changes.
wraps the npm scripts, so
npm teststill works in a bare checkout. Hooksplit is about the network:
npm testfetches, so it's pre-push, notpre-commit — a commit made offline shouldn't fail for an unrelated reason.
Two README corrections the suite forced into the open
./andend
.yaml, not bare stems. The prefix is load-bearing — a bare name isn'tdistinguishable from the UUID a published reference uses. The old example
(
component: my-app) is rejected by the structural phase.schema.type, and onsemanticTypewherever the consuming input names one.
Test plan
task check— the full CI set — passes locally:npm run typecheck— cleannpm test— 377 tests, 377 pass, 0 fail across all 13 itemsshellcheck(-S style) — cleanactionlint— cleanNo item YAML or media is touched; this is additive plus the README.
🤖 Generated with Claude Code