Skip to content

fix: read xUnit reports as the documents declare, not via the host locale codec - #1346

Open
ppcvote wants to merge 1 commit into
morganstanley:mainfrom
ppcvote:fix/xunit-importer-encoding
Open

fix: read xUnit reports as the documents declare, not via the host locale codec#1346
ppcvote wants to merge 1 commit into
morganstanley:mainfrom
ppcvote:fix/xunit-importer-encoding

Conversation

@ppcvote

@ppcvote ppcvote commented Aug 20, 2026

Copy link
Copy Markdown

Closes #1345.

The defect

The importers opened report files in text mode with no encoding=, so CPython decoded them with locale.getpreferredencoding(False) before lxml could honour the file's own <?xml version="1.0" encoding="UTF-8"?> prolog.

Same 241-byte UTF-8 GoogleTest report, one test named charges_€100_fee, read four ways:

codec result
utf-8 name='charges_€100_fee'
cp950 UnicodeDecodeError: can't decode byte 0xe2 — import dies
cp1252 name='charges_€100_fee'silent mojibake
cp437 name='charges_Γé¼100_fee'silent mojibake
opened "rb" name='charges_€100_fee'

The crash is the loud case and the rarer one. cp1252 is the default ANSI codepage on en-US Windows, which covers most Windows users and the windows-latest leg of this project's CI: nothing errors, the test name in the imported report is simply wrong, and anything matching on it downstream silently misses.

It also round-tripped testplan's own output incorrectly, since exporters/testing/xml/__init__.py:344 already writes these files as UTF-8.

The change

Six sites, treated as two kinds rather than one:

XML handed to lxmlimporters/gtest.py, importers/junit.py, importers/cppunit.py now open "rb", so the prolog decides. That is what lxml expects for a self-describing document.

JSON or plain textimporters/testplan.py, testing/cpp/gtest.py, testing/cpp/hobbestest.py state encoding="utf-8". Opening these binary would change what the callers receive, so the encoding argument is the right instrument there.

One incidental win: cppunit_to_junit is already typed Union[str, bytes] and encodes str to UTF-8 itself, so the binary read drops a redundant decode/encode round trip rather than adding work.

The test

Two deliberate choices, because the obvious version of this test is worthless:

It runs each importer in a child interpreter with PYTHONUTF8=0 PYTHONCOERCECLOCALE=0 LC_ALL=C LANG=C. Without that it passes on a UTF-8 host whether or not the fix is present, which is precisely how this survived in CI.

It asserts on bytes decoded as UTF-8, never on a string read back through the same default. A read-back assertion round-trips the mojibake and passes both ways.

Verified by reverting the six sites and re-running: 3 failed / 1 passed before — one failure per XML importer — and 4 passed after. The fourth case is ASCII and passes either way; it is there as a control, so a failure in it means the harness broke rather than the encoding handling.

While writing it I got the CppUnit fixture wrong at first: its XSLT splits names on ::, so a <Name> without the separator yields an empty name and the test failed for a reason unrelated to encoding. The fixture now carries Fees::<name>, and that is noted in a comment so the next person does not repeat it.

Checks

tests/unit/testplan: 894 passed before, 898 after — exactly my four — with the same 30 pre-existing errors both times. tests/unit/testplan/importers: 11 passed. Newsfragment added as 1345_changed.xunit_importers_read_utf8.rst. Commit is DCO signed off.

Happy to narrow this to the three importers/ sites if you would rather keep the diff tighter, or to split the JSON ones into a separate PR.

…cale

The importers opened report files in text mode with no encoding, so CPython
decoded them with locale.getpreferredencoding() before lxml could honour the
file's own <?xml version="1.0" encoding="UTF-8"?> prolog.

Measured on the same 241-byte UTF-8 GoogleTest report containing a test named
charges_€100_fee:

  utf-8    name='charges_€100_fee'
  cp950    UnicodeDecodeError: can't decode byte 0xe2 — import dies
  cp1252   name='charges_€100_fee' — silent mojibake
  cp437    name='charges_Γé¼100_fee' — silent mojibake
  "rb"     name='charges_€100_fee'

The crash is the loud case and the rarer one. cp1252 is the default ANSI
codepage on en-US Windows, covering most Windows users and the windows-latest
CI leg: nothing errors, the test name is just wrong, and anything matching on
it downstream silently misses.

This also round-tripped the project's own output incorrectly, since
exporters/testing/xml/__init__.py:344 already writes these files as UTF-8.

Six sites, three of them XML fed to lxml (gtest, junit, cppunit importers)
which now open "rb" so the prolog decides, and three reading JSON or plain
text (importers/testplan.py, testing/cpp/gtest.py, testing/cpp/hobbestest.py)
which state encoding="utf-8". cppunit_to_junit already accepted bytes and
encoded str to UTF-8 itself, so the binary read also drops a redundant
decode/encode round trip.

The test runs each importer in a child interpreter with PYTHONUTF8=0 LC_ALL=C
so it reproduces on a UTF-8 CI host, and asserts on bytes decoded as UTF-8
rather than on a string read back through the same broken default, which would
round-trip the mojibake and pass either way. Verified by reverting the six
sites: 3 failed / 1 passed before, 4 passed after, with the ASCII case passing
either way as a control.

tests/unit/testplan: 894 passed before, 898 after, same 30 pre-existing errors.

Closes morganstanley#1345

Signed-off-by: ppcvote <risky9763@gmail.com>
@ppcvote
ppcvote requested a review from a team as a code owner August 20, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

xUnit report importers decode UTF-8 XML with the host locale codec: crash on DBCS Windows, silent mojibake on cp1252

1 participant