Fix Windows console/file UTF-8 encoding crashes - #131
Conversation
Two related issues hit on Windows, found while running the full
transcribe -> pack -> render pipeline from a stock Windows terminal:
- helpers/render.py, grade.py, pack_transcripts.py, timeline_view.py,
transcribe_batch.py: several print() calls use non-ASCII characters
(mostly arrows). Windows consoles default to a non-UTF-8 codepage
(e.g. cp1252), which raises UnicodeEncodeError on those prints unless
the caller happens to set PYTHONIOENCODING=utf-8 first. Reconfigure
stdout to UTF-8 unconditionally at import time instead of relying on
that env var being set.
- helpers/render.py build_master_srt: master.srt was written without an
explicit encoding, so accented characters got mangled on platforms
where the default text encoding isn't UTF-8 (Windows again). This also
broke the downstream ffmpeg subtitles burn ("Invalid UTF-8 in decoded
subtitles text"). Write it explicitly as UTF-8.
- helpers/render.py build_final_composite: after fixing the encoding
above, ffmpeg's subtitles filter still failed on Windows
("Invalid data found when processing input") because the absolute
subtitles path used backslashes, which collide with the filter's own
escaping syntax. Normalize to forward slashes before escaping the
drive-letter colon.
There was a problem hiding this comment.
2 issues found across 5 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="helpers/grade.py">
<violation number="1" location="helpers/grade.py:40">
P3: `sys.stdout.reconfigure(encoding="utf-8")` runs unconditionally at import time and raises `AttributeError` if `sys.stdout` is not a live `TextIOWrapper` (e.g. stdout is `None` under `pythonw.exe`, or the module is imported while stdout is replaced by a non-filesystem stream such as `io.StringIO` under `contextlib.redirect_stdout`). These are standalone CLI scripts so normal console runs are unaffected, but guarding the call makes the fix robust to those environments instead of trading one crash (UnicodeEncodeError) for another (AttributeError). This applies identically to all four added calls in this batch.</violation>
</file>
<file name="helpers/render.py">
<violation number="1" location="helpers/render.py:34">
P3: `sys.stdout.reconfigure()` raises `AttributeError` when `sys.stdout` is not a `TextIOWrapper` (e.g., stdout redirected to a StringIO or a non-buffered wrapper in an embedded/hosted interpreter), and this call runs unconditionally at import time so it would abort the module before any fallback. In normal CLI usage stdout is a TextIOWrapper, so this is an edge case; guard it (or use the `errors`-safe reconfigure only when available) to keep the module importable in such environments.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| # Windows consoles default to a non-UTF-8 codepage (e.g. cp1252), which raises | ||
| # UnicodeEncodeError on the arrows this module prints. Force UTF-8 stdout | ||
| # unconditionally rather than relying on PYTHONIOENCODING being set. | ||
| sys.stdout.reconfigure(encoding="utf-8") |
There was a problem hiding this comment.
P3: sys.stdout.reconfigure(encoding="utf-8") runs unconditionally at import time and raises AttributeError if sys.stdout is not a live TextIOWrapper (e.g. stdout is None under pythonw.exe, or the module is imported while stdout is replaced by a non-filesystem stream such as io.StringIO under contextlib.redirect_stdout). These are standalone CLI scripts so normal console runs are unaffected, but guarding the call makes the fix robust to those environments instead of trading one crash (UnicodeEncodeError) for another (AttributeError). This applies identically to all four added calls in this batch.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At helpers/grade.py, line 40:
<comment>`sys.stdout.reconfigure(encoding="utf-8")` runs unconditionally at import time and raises `AttributeError` if `sys.stdout` is not a live `TextIOWrapper` (e.g. stdout is `None` under `pythonw.exe`, or the module is imported while stdout is replaced by a non-filesystem stream such as `io.StringIO` under `contextlib.redirect_stdout`). These are standalone CLI scripts so normal console runs are unaffected, but guarding the call makes the fix robust to those environments instead of trading one crash (UnicodeEncodeError) for another (AttributeError). This applies identically to all four added calls in this batch.</comment>
<file context>
@@ -34,6 +34,11 @@
+# Windows consoles default to a non-UTF-8 codepage (e.g. cp1252), which raises
+# UnicodeEncodeError on the arrows this module prints. Force UTF-8 stdout
+# unconditionally rather than relying on PYTHONIOENCODING being set.
+sys.stdout.reconfigure(encoding="utf-8")
+
</file context>
| sys.stdout.reconfigure(encoding="utf-8") | |
| if hasattr(sys.stdout, "reconfigure"): | |
| sys.stdout.reconfigure(encoding="utf-8") |
| # Windows consoles default to a non-UTF-8 codepage (e.g. cp1252), which raises | ||
| # UnicodeEncodeError on the arrows/em-dashes this module prints. Force UTF-8 | ||
| # stdout unconditionally rather than relying on PYTHONIOENCODING being set. | ||
| sys.stdout.reconfigure(encoding="utf-8") |
There was a problem hiding this comment.
P3: sys.stdout.reconfigure() raises AttributeError when sys.stdout is not a TextIOWrapper (e.g., stdout redirected to a StringIO or a non-buffered wrapper in an embedded/hosted interpreter), and this call runs unconditionally at import time so it would abort the module before any fallback. In normal CLI usage stdout is a TextIOWrapper, so this is an edge case; guard it (or use the errors-safe reconfigure only when available) to keep the module importable in such environments.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At helpers/render.py, line 34:
<comment>`sys.stdout.reconfigure()` raises `AttributeError` when `sys.stdout` is not a `TextIOWrapper` (e.g., stdout redirected to a StringIO or a non-buffered wrapper in an embedded/hosted interpreter), and this call runs unconditionally at import time so it would abort the module before any fallback. In normal CLI usage stdout is a TextIOWrapper, so this is an edge case; guard it (or use the `errors`-safe reconfigure only when available) to keep the module importable in such environments.</comment>
<file context>
@@ -28,6 +28,11 @@
+# Windows consoles default to a non-UTF-8 codepage (e.g. cp1252), which raises
+# UnicodeEncodeError on the arrows/em-dashes this module prints. Force UTF-8
+# stdout unconditionally rather than relying on PYTHONIOENCODING being set.
+sys.stdout.reconfigure(encoding="utf-8")
+
try:
</file context>
| sys.stdout.reconfigure(encoding="utf-8") | |
| if hasattr(sys.stdout, "reconfigure"): | |
| sys.stdout.reconfigure(encoding="utf-8") |
…font on Windows Three small Windows failures, none of them covered by browser-use#131 or browser-use#128. grade.py passed a temp file path straight into metadata=print:file=. Inside a filter option value the drive colon reads as an option separator, so ffmpeg reports "No option name near 'UsersRunner...'" and the analysis dies before a single frame is sampled. The path is now escaped the way render.py already escapes the subtitles path. Verified with --analyze on a real capture: the filter it computes is byte-identical to the one from a run where the path parses. render.py and timeline_view.py read the transcript and the EDL with read_text(), which decodes with the locale codepage. This does not raise on a non-ASCII transcript, which would at least be visible. On cp1251 it returns mojibake: "привет" comes back as "Рїсђрёрірес‚", and the subtitles are built from that. Both files are JSON, so UTF-8 is the only correct decoding. FONT_CANDIDATES listed macOS and Linux paths only. load_font falls back to ImageFont.load_default(), so the timeline still renders, in a bitmap font at a fixed size. Consolas and Arial appended.
…oted path Review on browser-use#135 pointed at the write side and at a gap in the escaping. The concat list is written with the locale codepage, so a segment under a path with non-ASCII characters is written in a form ffmpeg cannot open: "Impossible to open ..." and the render stops. The same list written as UTF-8 concatenates. This is the write half of the read fix already in this branch. The master SRT had the same problem with a worse ending, since it is the artifact the viewer sees: written in a codepage, libass reads it as UTF-8 and the subtitles come out as mojibake. browser-use#131 carries this one line too. Keeping it here so the branch fixes reading and writing together rather than half of each. pack_transcripts.py reads the same transcript JSON with the locale codepage. What transcribe.py writes is ASCII-escaped by json.dumps, so this one needs an externally produced transcript to bite, but the read should still name UTF-8. _esc_filter_path escaped the drive colon but not the quote, while the value it produces is wrapped in single quotes. A temp path under a user named O'Brien would close the quoted section early. render.py escapes both; now this does too.
Summary
helpers/*.pyscripts print non-ASCII characters (arrows); on a stock Windows console (non-UTF-8 codepage, e.g. cp1252) this raisesUnicodeEncodeErrorunless the caller happens to setPYTHONIOENCODING=utf-8first. Reconfigure stdout to UTF-8 unconditionally at import time inrender.py,grade.py,pack_transcripts.py,timeline_view.py, andtranscribe_batch.py.render.py'sbuild_master_srtwrotemaster.srtwithout an explicit encoding, corrupting accented characters on Windows and breaking the downstream ffmpeg subtitles burn (Invalid UTF-8 in decoded subtitles text). Now written explicitly as UTF-8.render.py'sbuild_final_composite: after the encoding fix, ffmpeg'ssubtitlesfilter still failed on Windows (Invalid data found when processing input) because the absolute subtitles path used backslashes, colliding with the filter's own escaping syntax. Normalized to forward slashes before escaping the drive-letter colon.Found and fixed while running the full transcribe → pack → render pipeline end-to-end on Windows against real footage.
Test plan
pack_transcripts.pyon a real transcript on Windows withoutPYTHONIOENCODINGset — no crash, arrow prints correctly.render.pypipeline (extract → concat → subtitles burn) on Windows —master.srtaccents intact, ffmpeg subtitles filter succeeds.reconfigure(encoding="utf-8")is a no-op there since stdout is already UTF-8).Summary by cubic
Fixes Windows-specific Unicode and path issues that caused UnicodeEncodeError in helper output and subtitle burn failures. Previously: helpers printed non-ASCII to a non-UTF-8 console and
master.srtused the default encoding/backslash paths; now: stdout is UTF-8,master.srtis UTF-8, and subtitle paths use forward slashes with an escaped drive colon.helpers/render.py,helpers/grade.py,helpers/pack_transcripts.py,helpers/timeline_view.py, andhelpers/transcribe_batch.py. Callers no longer need PYTHONIOENCODING set. No-op on macOS/Linux.master.srtwith UTF-8 encoding to preserve accents and satisfyffmpegsubtitle parsing.ffmpegby replacing backslashes with forward slashes and escaping the drive-letter colon to avoid filtergraph escaping conflicts on Windows.Written for commit 4e7cc82. Summary will update on new commits.