Follow-up from #45, which fixed the broader non-ASCII collision. Narrow, fails safe, not urgent.
The problem
normalizeWord (src/narration.ts) keeps combining marks via \p{M}, which is deliberate and correct — Devanagari matras and Thai tone marks are load-bearing, and dropping them just relocates the collision #45 fixed.
But toLowerCase() on U+0130 (LATIN CAPITAL LETTER I WITH DOT ABOVE) emits i + U+0307 COMBINING DOT ABOVE, and \p{M} retains that mark. So a Turkish anchor doesn't match an ASCII transcript token:
narration.atWord('scene', 'İstanbul') // → null, even when Whisper emitted "Istanbul"The obvious fix does not work
Reordering to .toLowerCase().normalize('NFC') is the natural suggestion. It changes nothing:
anchor İstanbul → 0130 0073 0074 ...
NFC → lower (current) → 0069 0307 0073 0074 ...
lower → NFC (proposed) → 0069 0307 0073 0074 ... ← identical
transcript "Istanbul" → 0069 0073 0074 ...
i + U+0307 has no precomposed form in Unicode — there is no "i with dot above" character, because the dot is inherent to lowercase i — so NFC has nothing to compose and the ordering is irrelevant.
What an actual fix costs
Any real fix has to special-case U+0307, and that trades directly against the behavior #45 exists to provide. Some options, none free:
- Strip
U+0307 only when it directly follows i. Narrow and targeted, but it's a hardcoded Unicode carve-out in a function whose whole point is being general.
- Use
Intl.Collator with sensitivity: 'base' for comparison instead of normalize-and-strip. Handles this correctly and generally, but changes matching semantics repo-wide and is markedly slower per comparison — atWord runs per word per scene.
- Full case-folding (
toLocaleLowerCase('tr')) — wrong, since it fixes Turkish by breaking every other locale's I.
Option 2 is probably right if this is worth fixing, but it deserves a benchmark first.
Why it is low priority
It fails safe. atWord returns null, and the documented caller pattern is narration.atWord(scene, word) ?? fallback (see demos/showcase.demo.ts), so the demo falls back to its timeout rather than misbehaving. That is strictly better than the pre-#45 behavior, which confidently returned the first non-Latin token's timestamp.
Found while reviewing #45; not a regression from it.
Follow-up from #45, which fixed the broader non-ASCII collision. Narrow, fails safe, not urgent.
The problem
normalizeWord(src/narration.ts) keeps combining marks via\p{M}, which is deliberate and correct — Devanagari matras and Thai tone marks are load-bearing, and dropping them just relocates the collision #45 fixed.But
toLowerCase()onU+0130(LATIN CAPITAL LETTER I WITH DOT ABOVE) emitsi+U+0307COMBINING DOT ABOVE, and\p{M}retains that mark. So a Turkish anchor doesn't match an ASCII transcript token:The obvious fix does not work
Reordering to
.toLowerCase().normalize('NFC')is the natural suggestion. It changes nothing:i+U+0307has no precomposed form in Unicode — there is no "i with dot above" character, because the dot is inherent to lowercasei— so NFC has nothing to compose and the ordering is irrelevant.What an actual fix costs
Any real fix has to special-case
U+0307, and that trades directly against the behavior #45 exists to provide. Some options, none free:U+0307only when it directly followsi. Narrow and targeted, but it's a hardcoded Unicode carve-out in a function whose whole point is being general.Intl.Collatorwithsensitivity: 'base'for comparison instead of normalize-and-strip. Handles this correctly and generally, but changes matching semantics repo-wide and is markedly slower per comparison —atWordruns per word per scene.toLocaleLowerCase('tr')) — wrong, since it fixes Turkish by breaking every other locale'sI.Option 2 is probably right if this is worth fixing, but it deserves a benchmark first.
Why it is low priority
It fails safe.
atWordreturnsnull, and the documented caller pattern isnarration.atWord(scene, word) ?? fallback(seedemos/showcase.demo.ts), so the demo falls back to its timeout rather than misbehaving. That is strictly better than the pre-#45 behavior, which confidently returned the first non-Latin token's timestamp.Found while reviewing #45; not a regression from it.