fix: read xUnit reports as the documents declare, not via the host locale codec - #1346
Open
ppcvote wants to merge 1 commit into
Open
fix: read xUnit reports as the documents declare, not via the host locale codec#1346ppcvote wants to merge 1 commit into
ppcvote wants to merge 1 commit into
Conversation
…cale The importers opened report files in text mode with no encoding, so CPython decoded them with locale.getpreferredencoding() before lxml could honour the file's own <?xml version="1.0" encoding="UTF-8"?> prolog. Measured on the same 241-byte UTF-8 GoogleTest report containing a test named charges_€100_fee: utf-8 name='charges_€100_fee' cp950 UnicodeDecodeError: can't decode byte 0xe2 — import dies cp1252 name='charges_€100_fee' — silent mojibake cp437 name='charges_Γé¼100_fee' — silent mojibake "rb" name='charges_€100_fee' The crash is the loud case and the rarer one. cp1252 is the default ANSI codepage on en-US Windows, covering most Windows users and the windows-latest CI leg: nothing errors, the test name is just wrong, and anything matching on it downstream silently misses. This also round-tripped the project's own output incorrectly, since exporters/testing/xml/__init__.py:344 already writes these files as UTF-8. Six sites, three of them XML fed to lxml (gtest, junit, cppunit importers) which now open "rb" so the prolog decides, and three reading JSON or plain text (importers/testplan.py, testing/cpp/gtest.py, testing/cpp/hobbestest.py) which state encoding="utf-8". cppunit_to_junit already accepted bytes and encoded str to UTF-8 itself, so the binary read also drops a redundant decode/encode round trip. The test runs each importer in a child interpreter with PYTHONUTF8=0 LC_ALL=C so it reproduces on a UTF-8 CI host, and asserts on bytes decoded as UTF-8 rather than on a string read back through the same broken default, which would round-trip the mojibake and pass either way. Verified by reverting the six sites: 3 failed / 1 passed before, 4 passed after, with the ASCII case passing either way as a control. tests/unit/testplan: 894 passed before, 898 after, same 30 pre-existing errors. Closes morganstanley#1345 Signed-off-by: ppcvote <risky9763@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1345.
The defect
The importers opened report files in text mode with no
encoding=, so CPython decoded them withlocale.getpreferredencoding(False)before lxml could honour the file's own<?xml version="1.0" encoding="UTF-8"?>prolog.Same 241-byte UTF-8 GoogleTest report, one test named
charges_€100_fee, read four ways:utf-8name='charges_€100_fee'cp950UnicodeDecodeError: can't decode byte 0xe2— import diescp1252name='charges_€100_fee'— silent mojibakecp437name='charges_Γé¼100_fee'— silent mojibake"rb"name='charges_€100_fee'The crash is the loud case and the rarer one. cp1252 is the default ANSI codepage on en-US Windows, which covers most Windows users and the
windows-latestleg of this project's CI: nothing errors, the test name in the imported report is simply wrong, and anything matching on it downstream silently misses.It also round-tripped testplan's own output incorrectly, since
exporters/testing/xml/__init__.py:344already writes these files as UTF-8.The change
Six sites, treated as two kinds rather than one:
XML handed to lxml —
importers/gtest.py,importers/junit.py,importers/cppunit.pynow open"rb", so the prolog decides. That is what lxml expects for a self-describing document.JSON or plain text —
importers/testplan.py,testing/cpp/gtest.py,testing/cpp/hobbestest.pystateencoding="utf-8". Opening these binary would change what the callers receive, so the encoding argument is the right instrument there.One incidental win:
cppunit_to_junitis already typedUnion[str, bytes]and encodesstrto UTF-8 itself, so the binary read drops a redundant decode/encode round trip rather than adding work.The test
Two deliberate choices, because the obvious version of this test is worthless:
It runs each importer in a child interpreter with
PYTHONUTF8=0 PYTHONCOERCECLOCALE=0 LC_ALL=C LANG=C. Without that it passes on a UTF-8 host whether or not the fix is present, which is precisely how this survived in CI.It asserts on bytes decoded as UTF-8, never on a string read back through the same default. A read-back assertion round-trips the mojibake and passes both ways.
Verified by reverting the six sites and re-running: 3 failed / 1 passed before — one failure per XML importer — and 4 passed after. The fourth case is ASCII and passes either way; it is there as a control, so a failure in it means the harness broke rather than the encoding handling.
While writing it I got the CppUnit fixture wrong at first: its XSLT splits names on
::, so a<Name>without the separator yields an empty name and the test failed for a reason unrelated to encoding. The fixture now carriesFees::<name>, and that is noted in a comment so the next person does not repeat it.Checks
tests/unit/testplan: 894 passed before, 898 after — exactly my four — with the same 30 pre-existing errors both times.tests/unit/testplan/importers: 11 passed. Newsfragment added as1345_changed.xunit_importers_read_utf8.rst. Commit is DCO signed off.Happy to narrow this to the three
importers/sites if you would rather keep the diff tighter, or to split the JSON ones into a separate PR.