Feature: Improve huge diffs performances using byte arenas

[clemg]

Prerequisites

• I have searched for duplicate or closed feature requests
• I have read the contributing guidelines

Proposal

Hey guys

Very cool blog post!

I think it's cool to see how much thought and effort is put in making great tools, especially on the linux (/ huge diffs) end

Personally, I experienced some crashes (OOMs) when trying on ginormous diffs

This got me wondering if there was any easy win on the matter, and I tried a few things that I'd like to share and discuss

1. Detaching parsed strings

Detaching makes sense on its own, but I don't understand why it is currently done line by line?

As I understand it, detaching only needs to break the reference from a parsed line back to the giant patch buffer so the GC can reclaim it. That reference can be broken at any granularity, so copying each file's text once is enough, and every line then slices into that small per-file copy instead of the whole patch

I tried exactly that on this branch: it produces a byte-identical parsed model (same file/line/char counts + content hash), but replaces ~22.8M detach calls with ~77k, about 300× fewer (on linux v6...v7: 76,872 files, 22.8M lines). Fewer allocations and GC pauses, so the parse step gets noticeably faster (how much depends on the CPU, since detachString does an encode/decode round-trip whose per-call cost varies a lot between machines)

2. Storing line content as a byte arena

Detaching frees the source patch, but the parsed line copies are still millions of individual js strings, and that's where a big part of the remaining memory goes:

• per-object overhead: even after detaching, every line is still its own v8 string object (tens of millions of them), each carrying V8's fixed per-object header (~16+ bytes: map pointer, hash, length) plus pointer/offset fields, on top of its actual bytes
• UTF-16 inflation: a js string flips to 2 bytes/char the moment it holds one character above U+00FF (e.g. an emoji or CJK glyph). detaching per file makes a whole file one string, so one such char taints the entire file. UTF-8 keeps ASCII at 1 byte and only spends extra on those chars
• heap fragmentation: churning tens of millions of small, varied-size allocations leaves the GC heap full of little holes between still-live objects. V8 can't hand a memory page back to the OS unless the entire page is free, so the process keeps far more RSS resident than the live data actually needs, and a GC doesn't fix it (it's also why this shows up in real browser RSS but barely in Node's (or Bun in my case) cleaner allocator)

So I tried stroring each file's lines as a single UTF-8 byte arena (Uint8Array), plus an Int32Array of per-line byte offsets so a line is just decode(bytes[offset[i]..offset[i+1]]), decoded on-demand, instead of a string[] (branch). Interestingly this makes fix #1 above redundant -> the arena copies each line into its own buffer, which already detaches it from the patch, so this branch doesn't bother detaching at all. (The deployed demo is just this + a dockerfile, ignore the docker stuff, its just clanker made for me to deploy)
It's important to use bytes and not chars, because ASCII stays 1 byte/char, it never taints a whole file to UTF-16, and it lives off the V8 heap.
Two small bits about this: the decoder uses ignoreBOM so a line that genuinely starts with a U+FEFF byte-order mark isn't silently stripped, and it keeps a plain string[] fallback for the rare files with a lone surrogate that wouldn't survive a UTF-8 round-trip

I deployed my forked main (for baseline comparison) and a byte-arena branch on the same server, to compare on identical hardware:

• main baseline: https://diffshub-baseline.clemg.fr
• perf branch (byte arena): https://diffshub-perf.clemg.fr

Results

Measuring baseline vs perf on the linux v6...v7 compare (isolated Chrome, reading the renderer's OS-level RSS; M2 MacBook Air 16gb; averaged over 3 runs):

metric	baseline	byte arena	Δ
peak renderer RSS	2752 MB	1911 MB	−31%
retained RSS (after forced GC)*	1688 MB	849 MB	−50%

* we can force GC in devtools > memory > 🗑️

The ranges across runs never overlapped: the worst byte-arena run beats the best baseline run. Parsed output is byte-identical

You can see the same thing live in Chrome's Task Manager (window -> task manager -> memory footprint; force a GC first):

The top tab is the perf branch (2.1 GB footprint, ~454 MB JS), the one right under is the baseline (3.4 GB, ~690 MB JS). However, usedJSHeapSize (and the devtools "JavaScript VM instance") under-counts this and can even read higher with the change, because the bytes move off the V8 heap into the Uint8Array. The correct number is the total renderer footprint shown here

These two changes have some drawbacks though:

• I don't like that it's a class, because a class instance is fragile across every serialization boundary. Structured clone (postMessage to the highlight worker), IndexedDB, structuredClone(), any cache -> they all drop the prototype, so you have to manually revive it at each boundary or it silently breaks (this actually bit me as a blank render when scrolling past the virtualizationin the worker path until I added a revive()). A plain { bytes, offsets } data object + free functions would probably get the same memory win without that, and could probably also keep the API non-breaking
• it changes a public type for additionLines/deletionLines, it goes from string[] to a LineList (x[i] → x.get(i), x[i] = j → x.set(i, j), etc), which is breaking for current users
• per-read decode costs microseconds, and only for the visible (virtualized) lines. Small diffs (≤ 500 lines) see a few µs of extra parse time. This could very probably just get ignored with a size threshold

Btw, the clanker helped to prototype parts of these, so just take this as an experiment. If you'd like I can tidy things up to make a proper version later

So yeah, just sharing thoughts and the experiment, feel free to try it and see how it goes for you!

---

More raw numbers (isolated Chrome over CDP, OS-level RSS, local builds)

Same measurement as above, against local builds instead of the deployments. Linux peak RSS is noisy run-to-run (the stream is long and the peak lands at different GC moments), so the retained-after-GC row is the stabler measure; every run in both setups points the same way

linux v6.0...v7.0 (678 MB patch, 76,872 files, ~22.8M lines):

metric	baseline	byte arena	Δ
peak renderer RSS	2436 MB	1258 MB	−48%
retained RSS (after forced GC)	1482 MB	627 MB	−58%
stream + parse span	62.1 s	54.3 s	−13%

bun v1.2.15...v1.3.14* (76 MB patch, ~2M lines):

metric	baseline	byte arena	Δ
peak renderer RSS	744 MB	616 MB	−17%
retained RSS (after forced GC)	557 MB	468 MB	−16%
stream + parse span	5.7 s	5.1 s	−10%

*: why this specifically? bun PR from the landing page is big but I like them bigger

(the "span" includes the throttled download, identical for both arms, so it understates the parse-CPU difference)

Motivation and context

Context: I want to discuss possible improvements over diffs and ultimately diffshub.com. I'm not sure if issues is the place for this as the contributing guidelines are not saying where discussions should precisely take place. Lmk if we should move, happy to transfer

[amadeus]

Hey hey, this is dope, thanks for the writeup and research!

To answer a few questions

Detaching makes sense on its own, but I don't understand why it is currently done line by line?

A lot of this was landed on after creating a bunch of benchmarks that balanced speed vs memory usage, with the AI iteratively looping and trying out different things.

It's possible I might be misunderstanding your post, but it does look like your updated changes do increase memory in Chrome when I GC after the linux diff fully loads:

(I assume this variant is the byte arena version tho? I am curious if some of your ideas on detaching larger sub strings could increase parsing performance tho)

[clemg]

Thanks for looking at it!

You're right that the heap goes up there, and it's expected because the "select JS VM instance" / heap-snapshot number includes ArrayBuffer backing stores, so it's now counting the arena's UTF-8 bytes + the per-line offset arrays. The arena doesn't really shrink live bytes per se, what it removes is fragmentation. With 22.8M tiny string objects, V8 can't return a page to the OS unless it's completely empty, so the survivors pin a lot of committed-but-unused pages, and the GC doesnt clear that
One contiguous Uint8Array per file doesn't fragment

That gap I was mentioning only shows in the "total renderer footprint" not the per-isolate heap, and it shows the whole tab's process memory (js heap + those off heap ArrayBuffers + DOM + GPU + everything the OS actually hands the tab). The per-isolate "VM instance" heap you're seeing is just one slice of it. Could you check (in chrome) Window > task manager > Memory footprint on the same pages to compare again? You'll be able to see the total memory used for everything on both pages, and (hopefully) it should be much lower like on my screenshot above

[amadeus]

Word, yeah that looks a lot better under that view, very interesting!

I'm down for you to start a branch with this where we can maybe start playing around with things. One thing to keep in mind, we are currently beta testing an editor feature ( #749 ), so that may need additional thinking around how to manage seal and all that.

[clemg]

Good you mention #749 , didn't knew about it

I'll take some time to check out what's on there and cleaning my stuff so it's easier to play with the test branch and see if its a win

Btw, while having a byte arena style storage would probably reduce the memory usage significantly and remove the need to detach per file (as in the first point in my first msg), I feel like detaching per file is an easy win for a little cost in the meantime

Here is the total diff to detach per file instead of per line, and here is a link to test this online

See the following image on Ghostty's huge PR from the landing page (pink on the left is the first line in the task manager and blue on the right is the second). This might require additional testing and benchmarking, but I'm always getting better memory usage with this!

Let me know your thoughts on this :p

[clemg]

Hey there, quick update on this

Here is a new branch about everything I've talked about here above

I've rewritten it to be cleaner, and to keep the current API as similar as possible. I think it's better that my first showcase code-wise because it's much more readable, because I figured out some stuff along the way

All in all, this branch (clemg/diff-byte-arena) - on my hardware - is pretty much consistently using about 1.8go to 2go of ram on a huge compare like linux v6..v7 (after a GC pass), which is roughly half of the 3.6go it currently uses on diffshub.com (try it for yourself!)

I also tested this branch with the #749 related stuff and the editor, and couldn't find any issue with it. I played with the examples on /docs and everything seems fine to me so far

How it works now (also explained in the code):

Instead of keeping one JS string per diff line (on linux v6..v7 again that's 22.8M of them, each carrying an object header, each one flipping its whole file to UTF-16 on the first non-ascii char, all of it fragmenting the heap so the tab never hands memory back), each side of a file (before/after) now packs every line into a single UTF-8 byte buffer plus a small offsets table, and decodes a line on demand as a slice of that buffer.

So the line bytes live off the JS heap, ascii stays one byte/char, and because the viewer already virtualizes rendeing, only the lines actually on screen are ever decoded back into strings (at least in the viewer, not in the editor afaiu). The thing that made this pass so much more readable (to me at least) was keeping it plain data rather than a class: it survices the structured-clone hop to the highlight workers with no "revive" step, and .length stays a normal field so the API barely moved (content reads just go through tiny helpers like lineAt(i) and joinLines(lines) instead of lines[i] and lines.join(), I'm not sure what to do about this..).

The handflul of inputs that can't round-trip through UTF-8 (a lone surrogate) or aren't worth encoding (synthetic merge-conflict diffs) transparently fall back to a plain string array, and the parsed model comes out byte-for-byte identical to the current implementation with detachString, with parsing even a touch faster since it encodes once up front instead of detaching every line

Note: I'm not really sure if you guys do test on various hardware for perf or if so, how you do it. It might be interesting to benchmark more and play with the current branch to find out new things

Disclosure: the tests in this branch are fully clanker made. they make sense, but I haven't checked them in depth

[amadeus]

Awesome! Can you setup this branch as a PR where we can track progress and stuff

[clemg]

Heyy there!

Quick update about everything above:

After thinking again about the byte arena thing, I thought: why would we even bother about parsing the patch in the byte arena in the first place? The whole point of the byte arena is having everything stored contiguously in the memory, so why not just send all the patch directly into it and parse on demand?

And parsing seems fast enough to do it lazily: the byte parser chews through the whole 678MB linux patch in ~4s on my machine. This is fast, but it could still be too slow as is if the machine is slow or if the patch is made out of big files only

so whats up

So here is a quick experiment on this branch about this. It's somewhat vibecoded, I guided my good friend claude through this to have a working proof of concept and to see if it would be worth it to try things like this

I had a few other ideas around this also, like packing the hunk objects tighter or deduping the context lines that are stored on both sides, but I couldn't really do anything significant in either of those lol
So for this clemg/diff-parse-bytes branch experiemnt, the patch bytes now go straight into the arenas as they arrive (the parser only peeks at the tiny diff --git / @@ headers to build the structure), and a line only becomes a real JS string when it scrolls into view. No string churn while streaming, which means that according to my testing parsing gets ~2x faster than the current byte arena branch I proposed, and most importantly the peak usage of RAM and RAM usage after settling both drop again (I get consistent -25% peak on the linux v6..v7 comparison, and the parsed model stays byte-for-byte identical to what the string parser produces)

Bonus: the lone-surrogate fallback isn't even needed on this path. A lone surrogate can't exist in UTF-8 bytes, and invalid bytes just sit in the arena and decode to U+FFFD when read, the exact text the string pipeline produced

Btw this is just for telling you about the new branch. I don't think it's even near ready to review, but that may be a good track to explore if the byte arena ends up landing in main! This is a working PoC just to prove the point

Btw 2: the editor isn't impacted: only the streamed viewer path changes, the string parser stays untouched for everything else, and the editor reads lines through the same joinLines/lineAt helpers as the byte arena branch

The experiment

To compare all of this fairly I set up a small bench: diffshub-bench.clemg.fr
The server clones and builds the branches itself (current main @ pierrecomputer/pierre, beta-1.3 @ pierrecomputer/pierre, byte arena @ clemg/pierre, and this new clemg/diff-parse-bytes branch @ clemg/pierre), runs your picks of the versions * diffs matrix one at a time, each in a fresh headless Chrome driven over CDP - streamed live to the page so you can watch it parse - then reads the renderer's real memory from the OS before and after a forced GC. Same hardware for every branch, so the numbers are actually comparable. Every finished run lands in the sortable history at the bottom; the peak and after-GC columns are pretty much the whole story :p

You can always try the new branch on your hardware too, and in fact it should be much faster because my poor server is already struggling and has like the worst of the worst CPU there is

Here are the results you can see on the bench page (all on the linux v6.0..v7.0 compare, 678MB):

version	parse CPU	parse wall	peak MB	settled MB	after-GC MB	JS heap MB
main	127.0s	164.1s	3,882	3,188	3,183	673
beta-1.3	111.9s	147.6s	4,397	3,185	3,179	672
byte-arena	89.1s	118.8s	2,735	1,819	1,759	437
parse-bytes	37.7s	65.4s	2,104	1,720	1,714	429

[clemg]

btw the diff seems huge but it's in huge part just moved code

hope I'm not harassing you 😁

[amadeus]

No worries, these posts are great! I'm just really busy, thus the slow responses. This new version is also very interesting to me as well. My initial thoughts would be, if we are going to ship a major change like this, ideally we just ship the final form of it, and not different versions spread across different releases that could cause various API thrash. Mostly because it does change some of the shape of FileHunkMetadata and i'd rather not do two major release of that.

[amadeus]

I think probably the most important thing for me to play around with is getting this working inside the diffshub environment so can play around with different scenarios, so maybe a separate PR for this to track that work a bit differently even if it's similar? Then i can use the generated preview links to test things out.

Also a note just in case, i did ship some fixes to patch parsing yesterday that would need to be included in any PRs/updates if not already

[clemg]

I'm just really busy, thus the slow responses

No worries! everyone is busy and this is not urgent, hope you enjoyed your trip :)

I'm not sure that I got the first point of your 2nd message right. Note that both branches from @clemg/pierre are working with diffshub very fine, you can try this locally if you'd like to (I agree its annoying to do it 1 by 1 tho)

I also agree on the shipping problem and doing only one change. I'll try proposing a clean(er) way of doing this new parsing method, on top of the draft PR I opened, in the next few days hopefully

With that built on top of the beta branch and what you included, it should be fairly easy to try all of this in just one go and decide where to go from there
Let me know if that would work with you and you plans

[amadeus]

I'm not sure that I got the first point of your 2nd message right.

what i mean by this is just simply that by having PRs opened against this repo it allows me to kick off/approve preview builds from the PR jobs, and also at the same time check the specific code/changes more easily. No rush tho!