Skip to content

spec: seed the 4.3 cycle with character content for the corpus form - #89

Merged
Nicholas-Keystate merged 1 commit into
Nicholas-Keystate:mainfrom
dhh1128:spec/4.3-seed-corpus-form
Aug 27, 2026
Merged

spec: seed the 4.3 cycle with character content for the corpus form#89
Nicholas-Keystate merged 1 commit into
Nicholas-Keystate:mainfrom
dhh1128:spec/4.3-seed-corpus-form

Conversation

@dhh1128

@dhh1128 dhh1128 commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

A 4.3 seed giving §15's committed corpus form some content about characters, plus the tool that enforces it.

Section 15 already makes "a canonical-form violation (the bytes fail the committed corpus form — ordering, corpus identity)" a conviction kind (L2777-2778). So the form exists and failing it is convictable. Nothing says which characters may appear — searched 2026-08-26, the ratified bytes contain no occurrence of "unicode", "utf-8", "codepoint", "charset", or any form of "normaliz". Finding #86 carries the evidence and the attacks.

The seed proposes that a domain commit its script set, with a character outside it a canonical-form violation. That is what makes the rule portable: Custos cannot say which scripts are legitimate without being wrong about Azerbaijani or Japanese, and a committed set is derivable from bytes with no Unicode data file. Independently of the set, law bytes are in NFC and exclude the two bidirectional overrides, three invisibles, unpaired surrogates, and control characters other than tab and newline. U+200B stays conditional, because Thai, Lao, Khmer and Myanmar use it to divide words. Refusal rather than repair, since a stripped override yields a clause nobody enacted.

tools/check-corpus-characters.py enforces the unconditional part over every file in spec/ and lineage/ plus the three root documents, together with an allowlist of the 10 non-ASCII characters this corpus actually uses — none of which is a letter, which is what defeats every cross-script homoglyph without naming a script. The corpus passes today: 748,199 characters, 30 files, nothing to fix. This is prevention rather than cleanup.

Confusable detection is deliberately absent. Per-word script mixing fires constantly on Japanese and misses an all-Cyrillic word that renders as Latin; the relational test needs Unicode data the standard library does not carry, and that dependency is a decision the seed does not make.

This does not discharge #86. A ruling is owed before the seed can enter. Offered to the #77 docket.

Relates to #86, #77, and #85.

…icholas-Keystate#86)

Section 15 already makes "the bytes fail the committed corpus form" a
conviction kind and glosses the form as ordering and corpus identity.
Nothing in the ratified text says which characters may appear, so a clause
can read one way to every reviewer and hash as something else.

The seed gives the form character content in a portable shape: a domain
commits its script set, and a character outside it is a canonical-form
violation. Unconditionally excluded are the two bidirectional overrides,
three invisibles, unpaired surrogates, control characters other than tab
and newline, and text not in NFC. U+200B stays conditional, because Thai,
Lao, Khmer and Myanmar use it to divide words. Refusal rather than repair,
since a stripped override yields a clause nobody enacted.

tools/check-corpus-characters.py enforces the unconditional part plus an
allowlist of the 10 non-ASCII characters this corpus actually uses, none
of which is a letter — which defeats every cross-script homoglyph without
naming a script or consulting Unicode data. The whole corpus passes today.

Confusable detection is left out on purpose. Per-word script mixing fires
on Japanese and misses an all-Cyrillic word that renders as Latin; the
relational test needs Unicode data the standard library lacks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>

@Nicholas-Keystate Nicholas-Keystate left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substance-gate review at the branch bytes.

Checker verified: check-corpus-characters.py runs green here as claimed — 31 files, 0 failures; the corpus arrives conforming, so this is prevention at its cheapest moment. The claim that the ratified bytes contain no charset/normalization language was re-verified by search.

The shape is right by this record's own laws. The committed script set is the declare-don't-mandate pattern (custody plurality, S7-2, S8-1, and the S9 coincidence ruling all take the same form): the standard mandates that a domain commits, never what. The unconditional exclusions are correctly minimal (renderer-instructions only; bidi marks and ZWSP conditionally spared for the scripts that need them — the Azerbaijani/Japanese/Thai reasoning is the right test). Refusal-not-repair follows §15's existing conviction shape: bytes silently corrected are a clause nobody enacted — that sentence is spec-grade and should survive into the successor's text.

Enforcement-grade note for the S8-1 register: this covenant is act-detectable — violation derives from committed bytes alone, no promoting instrument needed. The declaration writes itself.

One pin owed: the normative text references Normalization Form C and named codepoints; when spec text lands at 4.3, UAX #15 / the Unicode version should enter the external-pin discipline (engagement-companion route) like every other consumed external semantic — otherwise "NFC" floats on whatever Unicode revision is ambient, which is the defect class the seed itself is curing one level down.

Ruling posture: merging as seed-intake to the drafting corpus; the ruling the header confesses as owed executes with the 4.3 chain per the docket's standing pattern. Script-set placement (§18 vs founding law) stays chartered to the re-rooting decision, as the seed correctly leaves it.

@Nicholas-Keystate
Nicholas-Keystate merged commit 44632d2 into Nicholas-Keystate:main Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants