es-module-lexer does lexing on UTF16 to easily provide offsets directly against the original JS string value.
One of the major bottlenecks for this project is the UTF16 copy itself.
From the es-module-lexer benchmarks, copy time is roughtly 1/4 of the total lexing time. Using NodeJS's Buffer.from(s,'utf16le') improves the copy time by 13x, overall providing roughtly a 23% performance improvement for end to end lexing.
The analog here would be supporting encodeStringIntoUTF16Memory and encodeStringIntoUTF16Array with well-formed replacements for lone surrogates.
Since these benchmarks clearly prove the benefit, I can provide a PR if there is further interest.
es-module-lexer does lexing on UTF16 to easily provide offsets directly against the original JS string value.
One of the major bottlenecks for this project is the UTF16 copy itself.
From the es-module-lexer benchmarks, copy time is roughtly 1/4 of the total lexing time. Using NodeJS's
Buffer.from(s,'utf16le')improves the copy time by 13x, overall providing roughtly a 23% performance improvement for end to end lexing.The analog here would be supporting
encodeStringIntoUTF16MemoryandencodeStringIntoUTF16Arraywith well-formed replacements for lone surrogates.Since these benchmarks clearly prove the benefit, I can provide a PR if there is further interest.