feat(tests): html5lib tokenizer conformance suite (6,651/6,690 passing)#4
Merged
Merged
Conversation
…es it forced
Vendor the html5lib-tests tokenizer suites (6,806 cases; xmlViolation.test
excluded; bin/fetch-html5lib-tests.php re-downloads) and run every Data-state
case without lastStartTag through the tokenizer, comparing token sequences:
6,690 runnable, 6,651 passing, 39 skip-listed with reasons (CR/CRLF input
normalization and per-state NUL replacement — both M1.C).
Triage drove five tokenizer fixes:
- reconsume() rewound the buffer after an EOF (non-advancing) consume, so a
bare '&' looped Data -> CharacterReference until OOM; it now only undoes
an advancing consume, matching the spec's 'reconsume EOF' semantics.
- DOCTYPE public/system identifier states (spec 13.2.5.56-66) were an M1.A
shortcut to bogus-doctype + force-quirks; implemented fully (~317 cases).
- Comment-end-bang state (13.2.5.52) was missing.
- WhitespaceToken now carries decoded $data ('	' raw vs "\t" decoded).
- Numeric char refs saturate at 0x110000 instead of overflowing to float on
65-bit input; '&#X' without digits flushes its original case.
Round-trip fidelity (concat raw == input) holds throughout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
M1.B part 2: wire the html5lib-tests tokenizer suites in as a conformance backstop, and fix what they caught.
Suite: 6,806 cases vendored (
tests/fixtures/html5lib-tokenizer/, refreshable viabin/fetch-html5lib-tests.php;xmlViolation.testexcluded by design). 6,690 run (Data state, nolastStartTag); 6,651 pass; 39 skip-listed with per-entry reasons — all reduce to two M1.C work items: CR/CRLF input normalization (13) and per-state NUL → U+FFFD (26). Token sequences compared; error codes are the follow-up.Tokenizer fixes the triage forced:
&at EOF —reconsume()rewound after a non-advancing EOF consume, re-tokenizing the same&forever. Now matches spec 'reconsume EOF' semantics. (Found by case 47 of 6,690.)<!DOCTYPE html PUBLIC …>now parses identifiers, case-insensitive keywords, all quoting states, spec error codes).<!--x--!>).WhitespaceTokendecoded data —	raw vs\tdecoded were conflated; token now exposes$data.lush.Round-trip fidelity (headline invariant) green throughout.
Test plan