Follow-up from #241's review rounds, with the numbers measured there. The native decoder buffers a whole solid block before draining: peak RSS on a 400 MB block is 997 MB on the native path against 147 MB flat on the TypeScript path. The description of #241 states the trade honestly; this issue tracks removing it.
Two independent steps, in order of effort:
-
Skip the defensive copy on the finish() path. queue() in src/ipc/workers/nativeLzma2.ts copies every buffer via Uint8Array.from; on finish() the library is done with its buffer and the copy is pure overhead. Measured: dropping it takes the 400 MB peak from 997 MB to 579 MB. The copy is probably load-bearing on the update() path, where the library may reuse the buffer, so the skip must be finish()-only and tested on both paths.
-
Adopt the streaming variant. @napi-rs/lzma ships decompressStream in the same package, which would replace the buffering model entirely and bound memory to a chunk. This rewrites the adapter that three review rounds just verified byte-identical across the 27 fixtures, so it wants the full dual-path digest harness rerun, not a light touch.
Worth knowing for whoever picks it up: the real Vintage Story installer is one solid block of hundreds of megabytes, so today an install transiently costs roughly 2.5x the block size in RAM. Fine on a desktop, unkind on a small machine.
Follow-up from #241's review rounds, with the numbers measured there. The native decoder buffers a whole solid block before draining: peak RSS on a 400 MB block is 997 MB on the native path against 147 MB flat on the TypeScript path. The description of #241 states the trade honestly; this issue tracks removing it.
Two independent steps, in order of effort:
Skip the defensive copy on the finish() path. queue() in src/ipc/workers/nativeLzma2.ts copies every buffer via Uint8Array.from; on finish() the library is done with its buffer and the copy is pure overhead. Measured: dropping it takes the 400 MB peak from 997 MB to 579 MB. The copy is probably load-bearing on the update() path, where the library may reuse the buffer, so the skip must be finish()-only and tested on both paths.
Adopt the streaming variant. @napi-rs/lzma ships decompressStream in the same package, which would replace the buffering model entirely and bound memory to a chunk. This rewrites the adapter that three review rounds just verified byte-identical across the 27 fixtures, so it wants the full dual-path digest harness rerun, not a light touch.
Worth knowing for whoever picks it up: the real Vintage Story installer is one solid block of hundreds of megabytes, so today an install transiently costs roughly 2.5x the block size in RAM. Fine on a desktop, unkind on a small machine.