Version 3 of the box format: runtime seam, the wire break, and the node and native runtimes - #8
Merged
Merged
Conversation
v3 Phase A: a target says which machine a box runs on; a runtime says what runs inside it. Those facts lived in one table, which made every target adapter a statement that a box is a Python box and would have made a second runtime a fork of it rather than one more adapter beside it. The runtime half moves to src/contract/runtimes.mjs — layout, execution kinds, discovery, shell-free argv, self-test probe — mirrored by rust/src/contract/runtimes.rs and python/src/scrollcase_consumer/_contract.py and proven against a new shared fixture, runtime-contract.json. The builder-side half moves under src/runtimes/python/: launcher repair, the pip requirements reader, and the starter files `new scroll` writes. The three places that had bypassed the adapter and hard-coded `venv/`, or re-derived the Windows standard-library path in a branch, now ask the runtime. The four Node call sites that each carried a copy of the unsupported-version message share one helper. No wire format, document, schema or existing fixture changes.
A temp workspace escaped into a commit in August, leaving a second, stale copy of src/ tracked in the tree — including a copy of the contract that no longer agreed with the real one.
…lared executables
The single format break the v3 plan allows. Every change that touches a schema, a document or a
fixture lands here, so implementing the runtimes the vocabulary now names is pure code.
- A box declares $(printf %s 'runtime: { id, version, entryPoint }') instead of leaving a reader to infer Python from a
Python-shaped entry point. The wire vocabulary names python, node and native; only python has an
adapter, and a box naming another is refused by name rather than misread.
- modelId and runtimeId are gone, replaced by an optional labels map the tool never reads.
modelCacheSubdir becomes cacheSubdir; pythonVersion and pythonEntryPoint move into the runtime
block; provenance.pythonVersion becomes provenance.runtimeVersion.
- weights is gone. Whether an asset ships inside the archive is per entry, so a box can embed a
small entry point and defer a large dataset in one build. --weights goes with it: a build-time
override of a per-asset declaration is the silent-repack bug the flag already warned about.
- The self-test generalises. selfTest.probe carries imports and commands, and the runtime is the
only thing that turns either into a command line.
- The executable bit is declared rather than inferred from venv/bin: assets and local files may ask
for it, and extraction chmods explicitly so a strict umask cannot take it away again.
Still to do in this phase: the tri-language consumers, the conformance fixture, the examples and
the docs.
The conformance fixture keeps its 81 cases and gains three: a declared-executable box that keeps its bit under umask 077, a release naming a runtime this build has no adapter for, and a runtime block that disagrees with box.json. The signed example release is re-signed rather than edited, under a fresh key, because a fixture whose ed25519 signature no longer checks would prove nothing. The twelve example scrolls migrate to v3, and model-cache/ becomes cache/ in the examples and in the two demo entrypoints that read the field by name from box.json.
The schema route moves to /schema/v3/ everywhere it is derived or asserted, and box-format.md gains a table of exactly what changed between the two versions and why. managing-weights.md becomes managing-assets.md: the guide was built around a box-wide switch that no longer exists, and its subject was never weights in the first place. It now covers per-asset embed, the executable declaration, and what a deferred asset puts in the release. The white paper drops the weights-mode glossary entry for a deferred-asset one, names the new contract exports, and records that --weights was removed rather than replaced.
The manifest types carry a runtime block, labels and a probe instead of modelId, runtimeId, pythonEntryPoint and pythonImports, and the deferred-asset list replaces the weights/assets co-requirement — the list is the whole statement now, so there is nothing left for it to disagree with. RuntimeExecution gains a Binary shape and the kind becomes <runtime>-<shape> asked of the runtime that owns it: a script belongs to python or to node depending on the box, and the shape alone cannot say which. rust/tests/fixtures/signed-release.json is re-signed under a fresh key, its private half never having been committed, and the trusted key beside it is rewritten to match.
The models carry a BoxRuntime block and a labels map instead of modelId, runtimeId and pythonEntryPoint, RequiredAsset gains the executable declaration a materializer needs, and BoxExecution grows the two shapes the format now names but no adapter answers for yet. The probe becomes SelfTestProbe with imports and commands, and self_test_argv becomes self_test_invocations: a probe may imply several commands, each with its own required exit status. resolve_environment now takes the runtime id rather than assuming one, which is the last place the package inferred Python from the shape of a box.
AGENTS.md and CONTRIBUTING.md still described a v2-only line with a weights mode. The canonical terms gain runtime and deferred asset, the layout section names src/runtimes/ and runtimes.mjs, and the silently-breaking-paths list asks about embedded versus deferred rather than a mode that no longer exists.
The last of the three consumers to hand the archive's mode to open(2), which masks it by the process umask. Under 077 a declared-executable box lost the bit in Rust and Node and kept it in Python: three implementations of one contract, disagreeing observably. The conformance case now runs under umask 077 in all three drivers. Removing the chmod from any one of them turns bin/tool from 755 into 700, which is what makes the case a guard rather than a description.
Version 3 made where an asset lives a per-entry scroll declaration with no build-time override, and the flag went with it. Every demo and example workflow still passed it, so each would have failed at the first argument the CLI no longer knows.
Phase C of the version 3 work, on the Node side. Both runtimes arrive as adapters against the seam extracted in phase A: RUNTIME_ADAPTERS now holds the same three ids RUNTIME_IDS always named, and the wire did not move for either. A native box has no interpreter, so its layout names none and none is derived; its binary is the command itself; and a command probe is its only self-test, because there is no module system to ask for an import. Each of those is refused rather than ignored where a scroll declares otherwise. A node box is a python box with a different interpreter and no trampoline to repair, so the launcher step became the runtime's answer rather than pixi.mjs importing Python's directly. One thing the wire did need. pixi.lock declares a licence per conda package but cannot see what was linked into a binary a scroll supplies: that happened before Scrollcase saw the file, and reading the binary would be guessing. So a scroll may point at a declaration the project reviewed, the build checks every path it names is a file the box really carries, and the list is signed into the release — where a licence decision can be made before an archive is downloaded — as well as written beside the derived audit under THIRD_PARTY_NOTICES/. Authoring is runtime-dispatched throughout: --runtime chooses which execution kinds are offered, which starter is written, and which pixi dependency the generated manifest declares. --python-version becomes --runtime-version.
The mirror gains both adapters and the declared bundled-licence inventory. A layout's entry point and standard library become optional, which is what a native box actually has; assert_runtime_entry_point gains its third answer; and the probe kinds a runtime can answer are derived from whether it has an import probe at all, rather than declared a second time beside it. The conformance suite gains a case for an inventory that disagrees between box.json and the release, and the case that used to prove an unimplemented runtime now proves a python box relabelled native after signing — every id the format names is implemented, so that is the refusal that still matters.
The same three changes as the Rust mirror: both adapters, an optional layout entry point and standard library, and assert_runtime_entry_point as a shared function so all three consumers refuse a relabelled box in the same words. The probe kinds are derived from whether the runtime has an import probe at all, and bundledLicenses joins the fields box.json must agree with.
Both were built, verified with --self-test and run for real on this machine, and each turned up something the unit suite could not have. The native example started as sqlite3, whose own linkage is entirely @rpath. It failed anyway: conda-forge's ncurses, three dependencies down, ships a libncurses that re-exports libtinfo through an unrewritten build-machine path. The box was correct and a package inside it was not — which is exactly the limitation the native runtime states rather than assumes, and the self-test caught it before anything was signed. The example runs zstd instead. The node example failed its self-test against this repository's own package.json: Node reads the nearest one *above* the file it is running, so a box without one asks whichever directory it was extracted into. The node runtime now writes the box its own, unless the payload already carries one.
Found by feeding this repository's own published v2 demo box to the v3 verifier: it was refused, but with "Invalid signed document: $.schemaVersion must equal 3" — true, and no use to whoever is holding a box and needs to know which version it is. The envelope schema pins the version to a const, so the generic shape error got there first; version 2 also never got the by-name guard version 1 had. The Python and Rust consumers already named it. All three now say the same thing about the same published document.
The two new examples are the only thing in CI that builds a box with no interpreter, and one whose interpreter is not Python — the paths where a Python assumption would hide, and where the unit suite's fake toolchain cannot follow.
It counted archives after a rebuild, on the reasoning that a differing archive would land beside the first under its own hash. It would not: a build clears its own object directory before writing into it, so the count is always one. Measured — changing the scroll's sourceRevision produced a completely different archive and left the count at one. It now compares the archive's name, which is its SHA-256, before and after.
assetBaseUrl said "asset" and meant nothing of the kind: a scroll's assets carry a URL each, and this value only ever built two links — the release naming the archive, and the channel naming the release. It is publishBaseUrl, and the flag is --publish-base-url. It is also optional now, everywhere. archive.url and a channel entry's releaseManifestUrl may be absent, and a build given no publish location omits both rather than refusing. The old refusal forced anyone packaging a program to run on their own machine to invent an address, while the tool declined to invent one itself on the grounds that a placeholder in a signed release is a false statement. Both could not be right. Nothing is lost but the address: an archive is verified by sha256 and size, and all three consumers resolve one beside its release document rather than by following a link. A strip-release-archive-url conformance case pins that in every language, and examples/hello-box-native now declares no publish location at all — built, verified and run that way. This commit also carries the in-progress documentation work already in the tree: the versioned docs site with its frozen v2 snapshot, the codon, dataset and transcode examples, and src/cli-docs.mjs.
# Conflicts: # CHANGELOG.md
Deploying scrollcase with
|
| Latest commit: |
f4defc5
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://66937d6f.scrollcase.pages.dev |
| Branch Preview URL: | https://v3-phase-a-runtime-seam.scrollcase.pages.dev |
Both are defects in the tests, not in the product, and both had been invisible because this branch had never been pushed and CI runs on main and pull requests only. Windows carries no POSIX modes: archiveFileMode writes 0644 for every entry of a Windows box, and an extracted file reports no mode to read back. The umask conformance case asserted 755 and 644 on every platform. It is now gated with requiresPosixModes, honoured by all three drivers exactly as requiresSymlinks already is — the case is not weaker on Windows, it is inapplicable there. Proved by forcing the gate closed: 85 run, 1 skipped. The phase C archive-executable test asserted a refusal on every target. On a Windows target there is no bit to be missing, so the build correctly does not refuse. It now asserts both halves of the rule and says which target each applies to.
clippy's unused_unit, on the one function only Windows compiles. It had never been linted: this branch reached Windows CI for the first time today. Checked for real rather than by analogy — cargo clippy --target x86_64-pc-windows-msvc compiles the cfg(not(unix)) branch on this Mac, and is what should have caught it before the push.
Restores `scrollcase/contract/browser` by replacing the stale `assertPythonEntryPoint` re-export with runtime-model exports from `runtimes.mjs` (including `assertRuntimeEntryPoint`), so the browser entry point links correctly again. Also adds a new docs guide at `/guides/migrating-from-v2`, wires it into the sidebar and v2 deprecation notice, and updates related references (Node API docs, box-format docs, white paper, changelog). The package-surface test now links each exported subpath in real Node processes via the `exports` map to catch unresolved re-exports that Vitest can miss.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Version 3 of the box format, in three phases plus two later wire changes. 19 commits.
This PR exists to run CI. Everything below has been proven on macOS only — the branch has never
been pushed until now, so Linux and Windows have been exercised by nothing at all. Phases B and C
touched packing, relocation, archive modes and launcher repair, which
AGENTS.mdnames as the firstof the paths that break silently. That is the risk this PR is here to retire.
What changed
Phase A — the runtime seam. The runtime model was extracted from the target model, so a target
adapter is no longer also a statement that a box is a Python box. Behaviour-preserving; proven
byte-identical against the previous commit with a fake toolchain.
Phase B — the wire break. A declared
runtime: { id, version, entryPoint }block,labelsreplacing
modelId/runtimeId, per-assetembedreplacing the box-wideweights, a generalisedselfTest.probe, declared executables, and/schema/v3/. A v3 verifier refuses a v1 or v2 documentby name rather than reinterpreting it; there is no dual-read path and no migration tool, because
a box is rebuilt from its scroll.
Phase C —
nodeandnative. Both arrived as adapters against the settled seam, in thecontract and in all three consumers, with authoring and the CLI dispatched on the runtime. A native
box has no interpreter: it names no entry point, its binary is the command line, and a command
probe is its only self-test.
--python-versionbecame--runtime-version.Two wire changes after the plan, both decided while the branch was unpushed so v3 stays the only
break:
bundledLicenseDeclaration/bundledLicenses, for the licences of what was linked inside abinary a scroll supplies — the half
pixi.lockcannot see, declared by the project and signed.assetBaseUrlbecamepublishBaseUrl, and publishing became optional:archive.urland achannel entry's
releaseManifestUrlmay now be absent, and a build given no publish locationomits both rather than refusing. Nothing was lost but the address — an archive is verified by
sha256and size, and all three consumers resolve one beside its release document.What was proven, and where
Green locally in all three languages: Node (558 tests), Rust (8 suites, clippy clean,
cargo package), Python (68 tests, mypy, wheel + sdist). Docs build with the postbuild verifier.Real builds on macOS:
hello-box-nodeandhello-box-nativeboth build, passverify --self-test,run, and rebuild byte-identically. A throwaway box with one embedded and one deferred asset was
built against a loopback server, and
runrefuses until the deferred file is materialised. Therepository's own published v2 demo box was fed to all three v3 verifiers.
Three defects were found by doing that work rather than by the suite:
ncursescarries an unrewritten build-machine path, which breaks a native box thatis otherwise correct. The self-test caught it before signing.
package.jsonabove the file it runs, so a box without one asks whateverdirectory it was extracted into. The node runtime now writes the box its own.
its own object directory first. It now compares the archive's name.
Deliberately not in this PR
workflow_dispatchworkflows with thedemo key, after merge —
https://scrollcase.dev/schema/v3/…currently returns 404, so a box builtnow would carry a
$schemathat points nowhere.## [Unreleased]is still open. npm, PyPI and crates.io stay where theyare until after merge.
What to look at
fixtures and the generated types.
schemaVersion1 or 2.🤖 Generated with Claude Code