feat(tokenizer): M1.B — full WHATWG named-character-reference table#3
Merged
Conversation
Replace the ~30-entry seed table with the complete 2231-entry WHATWG list, codegen'd into NamedCharacterReferences by bin/generate-entities.php from the vendored resources/entities.json (sha256-stamped, ASCII-only output so regeneration is byte-stable under PHP-CS-Fixer). matchNamed() switches from a full-table scan to a longest-match probe over descending candidate lengths — ≤31 hash lookups per '&', O(1) in table size — so the full table doesn't blow the committed performance budget. Also implements the entity rules that only matter at full table size: the WHATWG §13.2.5.73 historical exception (a legacy semicolon-less match in an attribute value followed by '=' or an alphanumeric stays literal) and the missing-semicolon-after-character-reference parse error.
…in entity test Rector's AddArrayFunctionClosureParamTypeRector wants the TokenizerError type on the error-mapping closures. Phan flags the test's access to @internal classes (no notion of a package-owned test suite) and caps array-shape inference on the 2231-entry TABLE constant, making unsampled keys look like invalid offsets — both suppressed file-level with the reasoning inline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Internal\Tokenizer\NamedCharacterReferencesbybin/generate-entities.phpfrom the vendoredresources/entities.json(canonical spec data, sha256-stamped in the file header). Output is ASCII-only (\u{…}escapes from the spec's own codepoint lists), so regeneration is byte-stable and PHP-CS-Fixer-clean.CharacterReference::matchNamed()now probes candidate lengths downward (longest match, ≤31isset()lookups per&, bounded byMAX_LENGTH = 33) instead of scanning the table — O(1) in table size, protecting the committed ≤2× parse-time budget on entity-heavy input.=/alphanumeric stay literal (?x=1©=2keeps©, no error); semicolon-less matches elsewhere emitmissing-semicolon-after-character-reference.Test plan
CharacterReferenceTest(9 tests): generated table ≡ vendored JSON (all 2231 entries), length bounds, multi-codepoint + ligature entries, longest-match vs legacy prefix, attribute-context literal rule, missing-semicolon error in attribute and data contextsmake ci(Docker: tests 8.3/8.4/8.5, PHPStan max, Phan, Rector, CS-Fixer) green locally