Skip to content

perf(lexer): skip identifier and number runs#227

Merged
guybedford merged 2 commits into
guybedford:mainfrom
BridgeAR:BridgeAR/2026-07-12-skip-token-runs
Jul 13, 2026
Merged

perf(lexer): skip identifier and number runs#227
guybedford merged 2 commits into
guybedford:mainfrom
BridgeAR:BridgeAR/2026-07-12-skip-token-runs

Conversation

@BridgeAR

Copy link
Copy Markdown
Contributor

Summary

Skip the remainder of identifier and number runs after their first token dispatch, avoiding repeated switch dispatch and lastTokenPos stores.

On the 3,057 KiB sample corpus, Wasm improves from 15,356.31 to 13,010.46 us per sweep on Node 18.20.8 (-15.28%) and from 11,724.34 to 9,856.69 us on Node 24.18.0 (-15.93%). The asm.js build improves from 15,934.34 to 12,945.57 us on Node 24.18.0 (-18.76%). The full Wasm binary grows by 379 raw / 105 gzip bytes.

Test plan

  • Run the full Wasm and asm.js test matrix, including minimal builds
  • Compare baseline and candidate metadata over 100,000 generated sources

On the 3,057 KiB sample corpus, Wasm drops from 15,356.31 to 13,010.46 us per sweep on Node 18.20.8 (-15.28%) and from 11,724.34 to 9,856.69 us on Node 24.18.0 (-15.93%). The asm.js build drops from 15,934.34 to 12,945.57 us on Node 24.18.0 (-18.76%).

The fast path adds 379 raw / 105 gzip bytes to the full Wasm binary. Measurements used 1,500 warmup sweeps followed by 9 interleaved trials of 50 sweeps, dropping the best and worst trials.
Comment thread src/lexer.c
Comment thread src/lexer.c Outdated
@guybedford
guybedford enabled auto-merge (squash) July 13, 2026 00:58
@guybedford
guybedford merged commit 2abc49e into guybedford:main Jul 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants