fix: repair mojibake introduced by the American English pass - #7
Merged
Conversation
That pass was scripted with PowerShell, and Get-Content -Raw decodes as the
ANSI codepage on Windows PowerShell 5.1 rather than UTF-8. Every non-ASCII
character round-tripped through cp1252 and was re-encoded, so the ellipsis
and arrow characters turned into "a-hat" sequences -- visible in the legend
("768D -> 3D") and in the projection loading message. The result is still
valid UTF-8, so it compiled, passed tests, and shipped.
cp1252 has undefined byte values that decode to U+FFFD, so the transform
loses information and cannot be reversed byte-for-byte. The five affected
files are instead restored from 4f7c116, the commit before the pass; nothing
has touched them since. Set-Content -Encoding utf8 had also added BOMs to
three further files, which are stripped here.
Adds scripts/check_encoding.py and wires it into CI as its own job. This
class of corruption is invisible to both the type-checker and the test suite
-- the only symptom is garbage in the rendered UI -- so it needs a check that
looks at the bytes. Verified it fails on reintroduced corruption and passes
on the repaired tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first CI run failed on the checker's own source. Its patterns are, by definition, the character sequences it hunts for, so spelling them literally made the file trip its own check -- and the U+FFFD constant did the same. Excluding the file from the scan would have been the easy fix, but then genuine corruption in it would go unnoticed. Instead the patterns are built from numeric code points, so the source stays pure ASCII and the checker can scan itself like any other file. Verified: source is ASCII, the code points still decode to the intended pairs, all four sample corruptions are matched, exit 1 on injected corruption and exit 0 on the clean tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
That pass was scripted with PowerShell, and Get-Content -Raw decodes as the ANSI codepage on Windows PowerShell 5.1 rather than UTF-8. Every non-ASCII character round-tripped through cp1252 and was re-encoded, so the ellipsis and arrow characters turned into "a-hat" sequences -- visible in the legend ("768D -> 3D") and in the projection loading message. The result is still valid UTF-8, so it compiled, passed tests, and shipped.
cp1252 has undefined byte values that decode to U+FFFD, so the transform loses information and cannot be reversed byte-for-byte. The five affected files are instead restored from 4f7c116, the commit before the pass; nothing has touched them since. Set-Content -Encoding utf8 had also added BOMs to three further files, which are stripped here.
Adds scripts/check_encoding.py and wires it into CI as its own job. This class of corruption is invisible to both the type-checker and the test suite -- the only symptom is garbage in the rendered UI -- so it needs a check that looks at the bytes. Verified it fails on reintroduced corruption and passes on the repaired tree.