Skip to content

perf: fast native UTF-16 source copy into Wasm memory on Node#224

Merged
guybedford merged 1 commit into
mainfrom
perf/node-utf16-copy
Jul 12, 2026
Merged

perf: fast native UTF-16 source copy into Wasm memory on Node#224
guybedford merged 1 commit into
mainfrom
perf/node-utf16-copy

Conversation

@guybedford

Copy link
Copy Markdown
Owner

Copying the JS source into Wasm linear memory is ~25% of total parse time, and it was done with a per-character charCodeAt loop (~1.2 GB/s). On Node we can instead encode the string as UTF-16 straight into memory natively via Buffer (~15 GB/s, ~13x), with no other lexer changes since the lexer already indexes UTF-16 code units.

Warm parse of the full sample corpus (3057 KiB, avg of 25 runs):

  • before: ~11ms
  • after: ~8ms (~1.35x, ~27% faster end-to-end)
// before
copyLE(source, new Uint16Array(wasm.memory.buffer, addr, len));
// after
Buffer.from(wasm.memory.buffer, addr, len * 2).write(source, 'utf16le');

The whole gain comes from a single capability: encoding a JS string as UTF-16 directly into linear memory. Node exposes it natively; the browser has no equivalent (TextEncoder is UTF-8 only), so the portable charCodeAt path is retained as a fallback and the same speedup is left on the table anywhere a native UTF-16 encode-into-memory becomes available.

The fast path is gated strictly to the real Node Buffer on little-endian hosts (every platform Node runs on); browser bundles and big-endian hosts keep the existing code-unit copy. Existing tests pass unchanged, including the non-ASCII sample files.

@guybedford
guybedford force-pushed the perf/node-utf16-copy branch from cf4df7c to db32979 Compare July 8, 2026 00:16

@BridgeAR BridgeAR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@guybedford
guybedford merged commit b8f9d89 into main Jul 12, 2026
1 check passed
@guybedford
guybedford deleted the perf/node-utf16-copy branch July 12, 2026 01:34
mergify Bot added a commit to ArcadeData/arcadedb that referenced this pull request Jul 12, 2026
…2.3.1 in /studio [skip ci]

Bumps [es-module-lexer](https://github.com/guybedford/es-module-lexer) from 2.3.0 to 2.3.1.
Release notes

*Sourced from [es-module-lexer's releases](https://github.com/guybedford/es-module-lexer/releases).*

> 2.3.1
> -----
>
> What's Changed
> --------------
>
> * perf: fast native UTF-16 source copy into Wasm memory on Node by [`@​guybedford`](https://github.com/guybedford) in [guybedford/es-module-lexer#224](https://redirect.github.com/guybedford/es-module-lexer/pull/224)
> * perf(lexer): gate keywordStart on the next byte for i/c tokens by [`@​BridgeAR`](https://github.com/BridgeAR) in [guybedford/es-module-lexer#223](https://redirect.github.com/guybedford/es-module-lexer/pull/223)
>
> **Full Changelog**: <guybedford/es-module-lexer@2.3.0...2.3.1>


Commits

* [`2b2e620`](guybedford/es-module-lexer@2b2e620) 2.3.1
* [`8c21b74`](guybedford/es-module-lexer@8c21b74) perf(lexer): gate keywordStart on the next byte for i/c tokens ([#223](https://redirect.github.com/guybedford/es-module-lexer/issues/223))
* [`b8f9d89`](guybedford/es-module-lexer@b8f9d89) perf: fast native UTF-16 source copy into Wasm memory on Node ([#224](https://redirect.github.com/guybedford/es-module-lexer/issues/224))
* See full diff in [compare view](guybedford/es-module-lexer@2.3.0...2.3.1)
  
[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility\_score?dependency-name=es-module-lexer&package-manager=npm\_and\_yarn&previous-version=2.3.0&new-version=2.3.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
  
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot show  ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants