Skip to content

feat(tests): validate the catalog corpus against musher-dev/spec - #1

Merged
amiralis1365 merged 5 commits into
mainfrom
feat/corpus-validation
Aug 25, 2026
Merged

feat(tests): validate the catalog corpus against musher-dev/spec#1
amiralis1365 merged 5 commits into
mainfrom
feat/corpus-validation

Conversation

@amiralis1365

Copy link
Copy Markdown
Collaborator

The seed commit left this repository deliberately unvalidated — the platform
was the sole validator, and no CI ran here. The platform is still the sole
authority, but "the platform will reject it" is a slow way to learn that
an item is malformed. This adds the same contracts, applied early.

Not a second authority

Schemas are fetched at run time from the tip of the public
musher-dev/spec, never vendored. A copy
held here would be a second answer to "what is the contract" — and a stale copy
is worse than none, because it passes a corpus the live spec would reject,
silently. Nothing about the source is configurable, and if the fetch fails the
suite fails loudly, naming the URL. The repo is public, so no credential is
read anywhere in the suite.

What runs

File Phase Checks
spec.test.ts Bundles resolve and are self-contained. Fails first, so the corpus is never judged against a 404 page.
parser.test.ts parser The Musher YAML profile (§7.1) — one doc per file, string keys, no anchors/aliases/merge keys/tags, size and depth bounds.
structural.test.ts structural Each document against its family's fetched JSON Schema.
semantic.test.ts semantic Cross-document rules — identity, reference resolution, path containment, media, description Markdown, image pinning, endpoint resolution, connection compatibility, parameter coverage.
layout.test.ts Item folder structure.
rules.test.ts The rules themselves, against deliberately broken synthetic items.

capability is deliberately absent. Whether a Compute Profile is offered,
whether a published component exists, whether a version is monotonic — all
decided against the platform catalog over the network, and an implementation
MUST NOT report a rule it cannot check. An item green here can still be
rejected at sync.

rules.test.ts guards the guards: a suite that passes on a clean corpus passes
just as readily when a rule is silently unreachable, so each case breaks one
thing and asserts the normative diagnostic fires.

Also included

  • CIvalidate on push/PR plus a daily schedule (the part that only makes
    sense because schemas aren't vendored: it catches a spec change that stops
    accepting an item, in the window before the next catalog PR). devcontainer
    is path-filtered and stays quiet on items-only changes.
  • Dev container, Taskfile, lefthook — the same checks locally. Taskfile
    wraps the npm scripts, so npm test still works in a bare checkout. Hook
    split is about the network: npm test fetches, so it's pre-push, not
    pre-commit — a commit made offline shouldn't fail for an unrelated reason.

Two README corrections the suite forced into the open

  • Blueprint component references are repo-local paths that must begin ./ and
    end .yaml, not bare stems. The prefix is load-bearing — a bare name isn't
    distinguishable from the UUID a published reference uses. The old example
    (component: my-app) is rejected by the structural phase.
  • A connection's two ends must agree on schema.type, and on semanticType
    wherever the consuming input names one.

Test plan

task check — the full CI set — passes locally:

  • npm run typecheck — clean
  • npm test377 tests, 377 pass, 0 fail across all 13 items
  • shellcheck (-S style) — clean
  • actionlint — clean

No item YAML or media is touched; this is additive plus the README.

🤖 Generated with Claude Code

amiralis1365 and others added 5 commits August 25, 2026 12:32
Precedes the tooling that needs them: node_modules/ and *.tsbuildinfo for
the test suite, .devcontainer/.env and .task/ for the container and task
runner. The .env entry matters most — the template beside it is committed,
the filled-in copy never is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The seed commit left this repository with no validation of its own, on the
grounds that the platform is the sole validator. It still is — but "the
platform will reject it" is a slow way to learn that an item is malformed,
and nothing here caught it at commit time.

These tests apply the same contracts earlier. They are not a second
authority: the schemas are fetched at run time from the tip of the public
musher-dev/spec repository rather than vendored, so the corpus is judged
against the contract as it currently stands. A copy held here would drift,
and a stale copy is worse than none — it passes a corpus the live spec
would reject, silently. The repository is public, so no credential is read.

Covers the three phases component spec §7 lets a client decide offline:

  parser      the Musher YAML profile — one document per file, string keys,
              no anchors, aliases, merge keys or tags, and the bounds
  structural  each document against its family's fetched JSON Schema
  semantic    the cross-document rules — identity agreement, reference
              resolution, path containment, media, the description Markdown
              profile, image pinning, endpoint resolution, connection
              compatibility, parameter coverage

`capability` is deliberately absent. Whether a Compute Profile is offered,
whether a published component exists and whether a version is monotonic are
decided against the platform catalog over the network, and an implementation
MUST NOT report a rule it cannot check. An item green here can still be
rejected at sync.

rules.test.ts guards the guards: a suite that passes on a clean corpus
passes just as readily when a rule is unreachable, so each case breaks one
thing in a synthetic item and asserts the normative diagnostic fires.

Nothing is restated from the spec where it can be read from it — the media
path grammar and the component value-schema defaults are pulled out of the
fetched bundles at run time, so they cannot drift from their definition.

Run by Node's type stripping and built-in test runner: no build step, no
framework. 377 tests across the 13 items.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two workflows, for two things that fail differently.

validate typechecks, lints the workflows and runs the suite on every push
and pull request. It also runs on a daily schedule, which is the part that
only makes sense because the schemas are not vendored: a spec change can
stop accepting an item that used to validate, and without a scheduled run
nothing notices until the next catalog PR happens to come along.

devcontainer covers the thing nobody can check without building it — the
scripts, the lockfile against devcontainer.json, and the .env template
against what the tasks actually read. Path-filtered, so it stays quiet on
an items-only change.

Both are contents: read and take no credential. musher-dev/spec is public.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gives the repository a reproducible environment and the same checks CI runs,
locally.

Taskfile wraps the npm scripts rather than replacing them: `npm test` stays
what CI runs and what works in a bare checkout with no Task installed, and
`task` is the ergonomics layer on top. One taskfile, not the split
musher-dev/spec uses, because there is one build surface here.

The lefthook split is about the network. `npm test` fetches the spec bundles
on every run, so it cannot be a pre-commit job — a commit made offline would
fail for a reason that has nothing to do with the commit. Pre-commit holds
what is local and fast (types, shell, workflows); pre-push holds the suite.
commit-msg enforces Conventional Commits.

.env.example is committed and .env is not, so env:check reports the keys a
stale local copy is missing rather than letting a task read an unset one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validation section said this repository runs no validation of its own
and that a mistake is not caught at commit time. Both stopped being true;
it now says what runs, what is fetched at run time and why, and which phase
is deliberately missing. The platform is still named as the sole authority —
the tests are the same contracts applied early, not a second set — and the
one-item-per-PR guidance stays, since that is what makes a rejection only
the platform can raise easy to attribute.

Two details the suite forced into the open while it was being written:

blueprint component references are repo-local paths that must begin ./ and
end .yaml, not bare stems. The prefix is load-bearing — a bare name is not
distinguishable from the UUID a published reference uses, so without it no
validator can tell which resolver the reference wanted. The example said
`component: my-app`, which the structural phase rejects.

A connection's two ends must agree on schema.type, and on semanticType
wherever the consuming input names one. The multi-service paragraph said
only that connections wire outputs to inputs, which is the shape of the
thing and not the constraint on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@amiralis1365 amiralis1365 self-assigned this Aug 25, 2026
@amiralis1365
amiralis1365 removed the request for review from justinmerrell August 25, 2026 14:25
@amiralis1365
amiralis1365 merged commit e606ef3 into main Aug 25, 2026
4 checks passed
@amiralis1365
amiralis1365 deleted the feat/corpus-validation branch August 25, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant