Skip to content

perf(lexer): gate keywordStart on the next byte for i/c tokens#223

Merged
guybedford merged 3 commits into
guybedford:mainfrom
BridgeAR:BridgeAR/2026-07-07-lexer-keyword-fastpath
Jul 12, 2026
Merged

perf(lexer): gate keywordStart on the next byte for i/c tokens#223
guybedford merged 3 commits into
guybedford:mainfrom
BridgeAR:BridgeAR/2026-07-07-lexer-keyword-fastpath

Conversation

@BridgeAR

@BridgeAR BridgeAR commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

The main character loop calls keywordStart() (and through it a second boundary helper) on every i and c token it reaches. On the angular/d3/rollup/magic-string corpus the byte after the token rules it out ~98% of the time for i and ~95% for c, so those calls are almost always wasted. Checking *(pos + 1) against the keyword's second letter (m for import, l for class) first short-circuits the calls with a single load+compare. memcmp already requires that same byte, so the guard is behaviour-preserving.

Why

The per-character loop is the dominant cost (it runs once per source code unit; the boundary helpers are called <0.05x per code unit), so cutting a function call out of the two hottest keyword cases moves the needle where micro-tuning the helpers themselves does not.

Measured on the same corpus the README benchmarks (~3M code units, Node 24 / V8, 150 iters x 13 trials, drop best+worst, run against the built wasm):

  • parse time: -1.4% mean (every trial negative), parse output byte-identical
  • wasm: -229 bytes / -129 instructions

Drive-by

  • isBrOrWs: write the 9..13 range as (char16_t)(c - 9) < 5, one unsigned compare instead of two. The wasm backend keeps the shorter form at every inlined copy of the helper; this accounts for most of the byte/instruction reduction above.

The main character loop calls keywordStart() (and through it a second
boundary helper) on every `i` and `c` it reaches, but over real-world
sources the next byte rules the token out ~98% of the time for `i` and
~95% for `c`. Checking `*(pos + 1)` against the keyword's second letter
first short-circuits those calls with a single load+compare; memcmp
already requires that same byte, so the guard changes nothing observable.

Measured on the angular/d3/rollup/magic-string corpus (~3M code units,
Node 24 / V8, n=150 iters x 13 trials, drop best+worst): -1.4% mean parse
time, parse output byte-identical. Wasm shrinks 229 bytes / 129
instructions (the drive-by below covers most of the byte delta).

Drive-by fix:

* isBrOrWs: write 9..13 as `(char16_t)(c - 9) < 5`, one unsigned compare
  instead of two, shrinking every inlined copy of the helper.
@BridgeAR
BridgeAR marked this pull request as ready for review July 8, 2026 09:02

@guybedford guybedford left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good thanks, although seems wasteful to do the double compare if we can avoid it?

Comment thread src/lexer.c Outdated
Comment thread src/lexer.c Outdated
The pos[1]=='m' gate already fixes the first byte, so the follow-up
memcmp only needs to check "port". Replace MPORT with PORT and start the
compare at pos+2, dropping the redundant re-read of pos[1] (-6 wasm
instructions).
Same as the import gate: pos[1]=='l' already fixes the first byte, so the
memcmp only needs "ass". Slice &LASS[1] (LASS stays for the two
non-fast-path class checks). -4 bytes, no extra instructions.
@guybedford
guybedford enabled auto-merge (squash) July 12, 2026 02:03
@guybedford
guybedford merged commit 8c21b74 into guybedford:main Jul 12, 2026
1 check passed
mergify Bot added a commit to ArcadeData/arcadedb that referenced this pull request Jul 12, 2026
…2.3.1 in /studio [skip ci]

Bumps [es-module-lexer](https://github.com/guybedford/es-module-lexer) from 2.3.0 to 2.3.1.
Release notes

*Sourced from [es-module-lexer's releases](https://github.com/guybedford/es-module-lexer/releases).*

> 2.3.1
> -----
>
> What's Changed
> --------------
>
> * perf: fast native UTF-16 source copy into Wasm memory on Node by [`@​guybedford`](https://github.com/guybedford) in [guybedford/es-module-lexer#224](https://redirect.github.com/guybedford/es-module-lexer/pull/224)
> * perf(lexer): gate keywordStart on the next byte for i/c tokens by [`@​BridgeAR`](https://github.com/BridgeAR) in [guybedford/es-module-lexer#223](https://redirect.github.com/guybedford/es-module-lexer/pull/223)
>
> **Full Changelog**: <guybedford/es-module-lexer@2.3.0...2.3.1>


Commits

* [`2b2e620`](guybedford/es-module-lexer@2b2e620) 2.3.1
* [`8c21b74`](guybedford/es-module-lexer@8c21b74) perf(lexer): gate keywordStart on the next byte for i/c tokens ([#223](https://redirect.github.com/guybedford/es-module-lexer/issues/223))
* [`b8f9d89`](guybedford/es-module-lexer@b8f9d89) perf: fast native UTF-16 source copy into Wasm memory on Node ([#224](https://redirect.github.com/guybedford/es-module-lexer/issues/224))
* See full diff in [compare view](guybedford/es-module-lexer@2.3.0...2.3.1)
  
[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility\_score?dependency-name=es-module-lexer&package-manager=npm\_and\_yarn&previous-version=2.3.0&new-version=2.3.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
  
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot show  ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants