Release 3.4.0: multi-language code-page correctness (#11, #12, #13) - #14
Merged
Conversation
Three related multi-language defects (issues #11, #12, #13), all found by auditing pyOpenVBA after the same bug class hit the TypeScript port. PROJECT stream (#11). It is code-page ANSI per [MS-OVBA] 2.3.1, but four sites hardcoded cp1252, so every rewrite -- add, rename, delete -- re-encoded module names with errors="replace". A cp1251 project holding МодульТест emitted Module=?????????? while the dir stream kept the real name, leaving declarations Excel cross-checks in disagreement. serialize_project_stream, parse_project_stream, and parse_projectwm now accept the project's code_page (default 1252 for standalone callers) and the save path threads it through. Vietnamese (#13). Python's charmap codecs do no composition, so 'Tiếng Việt'.encode('cp1258') destroyed every stacked-diacritic character. NFD alone does not help: cp1258 stores ệ as precomposed ê plus a combining dot-below, which is not that character's canonical decomposition. New encode_mbcs() decomposes an unmappable character and folds each combining mark back into the base until the codec accepts the result, emitting the remaining marks as combining bytes. Text that already encodes directly is returned byte-for-byte unchanged, so this can only affect characters that were becoming '?'. Codec resolution (#12). 54936 (GB18030) is the one page VBA hosts write that Python does not spell cp<N>; it now maps to the gb18030 codec. Unresolvable pages warn instead of silently falling back to latin-1. And when a module's name, stream name, or doc string disagrees between its ANSI record and its UTF-16 partner, the Unicode record -- lossless by construction -- now wins. Byte output for existing cp1252/ASCII files is unchanged: the 25-case save matrix across every live fixture hashes identically against a v3.3.0 worktree. Verified in live Excel: a cp1251 workbook whose module is NAMED МодульТест compiles and returns Привет, мир from a Cyrillic-named function.
The cross-OS language matrix earned its keep on its first CI run: five
pages (10000 Mac Roman, 20866 KOI8-R, 21866 KOI8-U, 28592 ISO-8859-2,
28595 ISO-8859-5) passed on windows-latest and failed on ubuntu.
Cause: on Windows, CPython falls through to the operating system's
code-page registry, so codecs.lookup("cp28592") succeeds there while
raising LookupError on Linux and macOS. Text in those pages therefore
decoded correctly on one platform and became latin-1 mojibake on
another -- the same silent-degradation bug class the matrix exists to
catch, in a form no single-platform test run could see.
_CODEPAGE_ALIASES now maps 30 Windows code-page identifiers to portable
Python codec names (Macintosh, KOI8, the ISO-8859 family, ISO-2022,
EUC, GB, UTF-7) and is consulted before the cp<N> spelling, so every
platform resolves identically. Two new tests assert portability against
encodings.search_function -- the pure-Python registry, identical
everywhere -- so this fails on any platform rather than only the
affected one.
'\c' is not a recognized Python escape, so the string happened to work while pyright and ruff flagged it -- and pyright runs inside the main CI test jobs, which is why they all failed while the language matrix passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11, closes #12, closes #13.
#11 — PROJECT stream hardcoded cp1252. Reproduced first: a cp1251 project with a module named
МодульТестemittedModule=??????????while the dir stream kept the real name. Four sites now take the project'scode_page. Verified in live Excel — the workbook compiles and returnsПривет, мирfrom a Cyrillic-named function in a Cyrillic-named module.#13 — cp1258 Vietnamese destroyed on encode. New
encode_mbcs()folds combining marks back into the base until the charmap accepts (ệ→ê0xEA + dot-below 0xF2). Byte-identical wherever direct encoding already worked.#12 — GB18030 alias, loud fallback, ANSI/Unicode reconciliation. All three items.
Test matrix + CI. 21 code pages × (zero-substitution, NFC round trip, full workbook cycle), plus native module names for cp1251/932/936, running in a dedicated cross-OS
languagesjob mirroring the port's CI.One correction to file on #11: the Mac Roman aside does not hold on modern Python —
cp10000resolves. 54936 was the only real gap.Verification: 462 tests + pyright strict + ruff; 25-case save matrix byte-identical against a v3.3.0 worktree; both live Excel gates green.