feat: implement full UCD 17.0#148
Open
srghma wants to merge 86 commits into
Open
Conversation
- Added Script_Extensions lookup support. - Added isKhmer, isCyrillic and other script helpers. - Verified Noncharacter and Default_Ignorable properties. Co-authored-by: srghma <219769217+srghma@users.noreply.github.com>
- Added Script_Extensions lookup and getScriptExtensions function. - Added isScript and hasScript functions. - Added boolean helper functions for all 176 Unicode scripts. - Verified Noncharacter and Default_Ignorable properties. - Updated documentation. Co-authored-by: srghma <219769217+srghma@users.noreply.github.com>
- Added support for Script_Extensions property. - Added comprehensive boolean helpers for all 176 Unicode scripts. - Added identifier properties: isIDStart, isIDContinue, isXIDStart, isXIDContinue. - Added punctuation/extender properties: isDash, isHyphen, isQuotationMark, isTerminalPunctuation, isExtender, isRegionalIndicator. - Verified and ensured consistency for Noncharacter and Default_Ignorable properties. - Updated documentation and generated lookup tables. Co-authored-by: srghma <219769217+srghma@users.noreply.github.com>
- Added Case Folding support (Full and Simple). - Added Grapheme Cluster Break and Word Break properties. - Added ID_Start, ID_Continue, XID_Start, XID_Continue properties. - Added Dash, Hyphen, Quotation_Mark, Terminal_Punctuation, Diacritic, Sentence_Terminal, Pattern_Syntax, Pattern_White_Space, Extender, and Regional_Indicator properties. - Added boolean helper functions for all 176 Unicode scripts. - Updated documentation and generated lookup tables. - Updated verification suite. Co-authored-by: srghma <219769217+srghma@users.noreply.github.com>
…essing - Added support for all 176 Unicode scripts with individual boolean helpers (isKhmer, isCyrillic, etc.). - Added Script_Extensions support. - Added full and simple Case Folding support (getCaseFold, getSimpleCaseFold). - Added Grapheme Cluster Break, Word Break, Sentence Break, and Line Break properties. - Added Identifier properties (ID_Start, ID_Continue, XID_Start, XID_Continue). - Added Emoji properties (Emoji, Emoji_Presentation, Emoji_Modifier, Emoji_Modifier_Base, Emoji_Component, Extended_Pictographic). - Added various punctuation and symbol properties (Dash, Hyphen, Quotation_Mark, Terminal_Punctuation, Diacritic, Sentence_Terminal, Pattern_Syntax, Pattern_White_Space, Extender, Regional_Indicator). - Added Grapheme_Base and Grapheme_Extend. - Updated infrastructure (makeTables.lean, TableLookup.lean) to support efficient lookups. - Updated documentation and verified with extensive tests. Co-authored-by: srghma <219769217+srghma@users.noreply.github.com>
…5838323980231 Add Script_Extensions support and script-specific helper functions
… using ts script
# Conflicts: # lakefile.lean # makeTables.lean
…odifier and Regional_Indicator are just "is btw two codes")
What was happening: - The test compares d.numeric to lookupNumericValue d.code in tests/UnicodeTableTest/Spec.lean:192. - d.numeric is built from UnicodeData.txt and augmented with DerivedNumericValues.txt in table-generators/UnicodeData/Basic.lean:246. - The old generator compressed decimal/digit sequences into ranges but stored only the first value, e.g. 0030..0039 => decimal 0. That makes 1, 2, etc. return decimal 0. - Your patch fixed some decimal ranges by reconstructing offsets, but the base list was incomplete and it did not address other Digit/Numeric range cases. I fixed it in the generator by emitting exact singleton numeric rows: table-generators/makeTablesForLookup.lean:697 | some n => t := t.push (d.code, d.code, n) Then regenerated tables. The generated Numeric_Value table is now 2023 singleton rows, so no lookup-side workaround is needed. Verification: - Ran just tables - Ran just test - Result: 54/54 tests passed, including Numeric_Value
…hape to Numeric_Value: the generator compresses ranges into a key table keyed only by the range start, so later code points in each range return #[]. I’m fixing the generator to emit per-code-point keys for script extensions, then regenerating tables and rerunning tests.
…ings like "09E8;Beng Cakm Sylo", but instead more type safe return type
Author
|
ok, I finished |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#112