Browser-fetch byte download OOMs the renderer on large videos (RangeError: Invalid string length)

[bendavidsteel]

Summary

Video._browser_fetch_bytes (the fallback download path in pytok/api/video.py) can crash the Chrome renderer with an out-of-memory error on large videos. It surfaces either as a caught {"ok":false,"error":"RangeError: Invalid string length"} or, for larger files, as a hard renderer crash ("Aw, Snap! Out of Memory" + a Crashpad dump).

Observed live during a multi-hour scrape: a renderer on one account's profile crashed (Crashpad dump written) while downloading; an identical RangeError: Invalid string length was also seen in an earlier run. The scrape absorbed it (the direct httpx path handles the vast majority of downloads and the worker pool tolerates a renderer blip), so this is robustness, not a blocker.

Root cause

_browser_fetch_bytes reads the response as an arrayBuffer() (good), but then materializes the entire file as a single JavaScript string twice before returning it:

let binary = ''; const chunk = 0x8000;
for (let i = 0; i < bytes.length; i += chunk) {
  binary += String.fromCharCode.apply(null, bytes.subarray(i, i + chunk)); }
return JSON.stringify({ok: true, b64: btoa(binary)});

binary is a whole-file-sized string and btoa(binary) a ~1.33× larger one; both live in the renderer at once, plus the JSON wrapper. For large videos this hits V8's max string length (~512 MB / 2^29 chars) → RangeError: Invalid string length, or exhausts the renderer's heap → hard crash before the try/catch can even return.

Impact

• Fallback only: _httpx_fetch_bytes runs first and succeeds for almost all videos, so this fires rarely (only when httpx is rejected AND the video is large).
• When it does fire it crashes a renderer, leaving a Crashpad dump and briefly disrupting that worker's session.
• Secondary concern: shuttling the whole file base64-over-CDP is slow and memory-heavy even when it doesn't crash.

Suggested fix

Avoid holding the whole video as a single JS string. Options, roughly in order of preference:

1. Stream the body out of band via CDP rather than through page JS — e.g. Fetch/Network.getResponseBody / IO.read on the response stream, reading in chunks into Python bytes. No giant JS string at all.
2. Chunked base64 transfer: have the in-page script return base64 in bounded slices (e.g. 8–16 MB) across multiple evaluate calls (or a generator), and concatenate on the Python side, so no single JS string approaches the limit.
3. At minimum, cap the browser-fetch path to a safe size and surface a clear error above it, so it degrades predictably instead of crashing the renderer.

Repro sketch

Force the httpx path to fail (so the browser fallback is used) and download a large (hundreds-of-MB) video via video.bytes(); the renderer OOMs / throws RangeError: Invalid string length.

Refs

• pytok/api/video.py — _browser_fetch_bytes (the base64 string accumulation) and bytes() (httpx-then-browser fallback order).

🤖 Generated with Claude Code