From fa28d8080a023d8f2902e709551e8a518f81335d Mon Sep 17 00:00:00 2001 From: Dmitry Bondarenko Date: Tue, 7 Jul 2026 22:53:55 +0600 Subject: [PATCH 1/4] Plan --- PLAN.md | 137 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 137 insertions(+) create mode 100644 PLAN.md diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..7ed323b --- /dev/null +++ b/PLAN.md @@ -0,0 +1,137 @@ +# Migrate svelte-demo rendering to ProseMirror (LIX-MDSP-20) + +## Context + +The demo at `demo/svelte-demo` currently renders the parser's streaming output with an ad-hoc, hand-written rendering layer inside `src/routes/+page.svelte` (~898 lines): a reactive block-grouping state machine (`parsedBlocks`), manual open-span tracking, table reconstruction (`buildTableRows`), and a giant `{#each}/{#if}` markup tree with Tailwind span classes. This is the "legacy state-machine code" to replace. + +Goal: replace that rendering layer with ProseMirror, following the architecture of the Lixpi main repo (workspace-local shallow clone currently available at `/tmp/claude-1000/-home-dima-Desktop-markdown-stream-parser/8b99adf2-a3b8-4241-898c-740929163041/scratchpad/lixpi-ref`; do not rely on this path outside this workspace): a framework-free ProseMirror module (schema + pure stream-assembly functions) driving an `EditorView` via transactions. + +**Key adaptation vs Lixpi:** Lixpi's `packages/lixpi/prosemirror/src/stream-assembly.ts` consumes the OLD parser segment shape (`{segment, styles[], type, isBlockDefining}`). This repo's tree-sitter parser emits a new offset-based `Chunk` shape (`src/tree-sitter/types.ts`): `{text, offset, length, block:{type, level?, language?, list?, table?}, opening/closing/contained spans, backtrackOffset?, recovery?}` wrapped in `StreamingChunk` (`START_STREAM | STREAMING | END_STREAM`). Lixpi's schema also lacks list/table nodes, which this parser emits. So we replicate the *architecture*, not the code verbatim. + +**User-approved decisions:** +- **Rebuild projection** strategy: keep a chunk buffer; on each chunk rebuild the whole doc via a pure `chunks → doc` function and dispatch one replace transaction. Backtracking = filter buffer; reset/replay = clear buffer. Demo-scale docs make O(n) rebuild imperceptible. +- **Location**: `demo/svelte-demo/src/lib/prosemirror/` (framework-free TS, promotable to a package later; repo stays single-package). +- Hand-written NodeSpecs for lists/tables (no `prosemirror-schema-list`/`prosemirror-tables` — those provide editing commands we don't need for a read-only view; Lixpi hand-writes all specs too). +- All commands run inside docker container `lixpi-markdown-stream-parser-demo`. + +## New files (all under `demo/svelte-demo/src/lib/prosemirror/`) + +### 1. `schema.ts` +Adapt Lixpi's `base-schema.ts` (scratchpad ref above), extend with lists/tables, drop lixpi-only nodes. Read-only view ⇒ `parseDOM` optional. + +Nodes: +- `doc` (`block+`), `paragraph` (`inline*` → `['p', 0]`), `heading` (attr `level`, → `h1..h6`), `code_block` (attr `language`, `content:'text*'`, `marks:''`, `code:true`, → `['pre', {'data-language': language}, ['code', 0]]`), `blockquote` (`block+`), `text` +- `bullet_list` (`list_item+` → `['ul', 0]`), `ordered_list` (attr `order` → `['ol', {start}, 0]`), `list_item` (attr `task: null|{checked}`, `content:'block+'`, → `['li', {'data-task': checked|unchecked}, 0]`) +- `table` (`table_row+` → `['table', ['tbody', 0]]`), `table_row` (`(table_header_cell|table_cell)+` → `['tr', 0]`), `table_header_cell`/`table_cell` (attr `align`, `content:'inline*'`, → `['th'|'td', {style:'text-align: ...'}, 0]`) +- `image` — **inline** (`inline:true`, group `inline`, attrs `src`, `alt`) since the parser emits images as inline spans + +Marks (per Lixpi's `createStreamingMark` mapping): `strong`, `em`, `code`, `strikethrough` (→ ``), `link` (attr `href`, `inclusive:false`, render with `rel="noopener noreferrer"`). + +Security: parser span metadata is untrusted text. Before creating link marks or image nodes, sanitize URLs with a single helper used by stream assembly: +- Links: allow `http:`, `https:`, and `mailto:` only; reject empty, malformed, `javascript:`, `vbscript:`, and `data:` URLs. +- Images: allow `http:`, `https:`, root-relative/path-relative URLs, and safe `data:image/*` URLs only; reject protocol-relative URLs, scriptable URLs, and non-image data URLs. +- Rejected links/images render as their plain covered text, not as clickable links or image nodes. + +### 2. `stream-assembly.ts` +Pure, no DOM/Svelte. Import types from `../../../../../src/markdown-stream-parser.ts` (same relative-source style `+page.svelte` already uses). + +```ts +// backtrack semantics: if chunk.backtrackOffset set, drop buffered chunks with +// (offset + length) > backtrackOffset, then append (proven predicate, +page.svelte:180-188) +applyStreamingChunkToBuffer(buffer: Chunk[], chunk: Chunk): Chunk[] + +// shared predicate so editor buffer and +page.svelte debug parsedSegments cannot drift +isChunkBeforeBacktrack(chunk: Chunk, backtrackOffset: number): boolean + +sanitizeLinkHref(rawHref: string): string | null +sanitizeImageSrc(rawSrc: string): string | null + +// port of the parsedBlocks state machine (+page.svelte:382-441): boundaries on +// block.type change (except same-tableId cells), tableId change, heading level change, +// list newline heuristic; ADD: list depth/type change also starts a new group +groupChunksIntoBlocks(chunks: Chunk[]): Chunk[][] + +// slice chunk text at span boundaries (absolute UTF-16 offsets); active spans = +// carried-open ∪ opening ∪ covering-contained − closed; image spans → inline image +// node replacing covered text; others → marks (bold→strong, italic→em, code→code, +// strikethrough→strikethrough, link→link{href:sanitizedUrl}). Rejected unsafe URLs +// render as plain covered text. Skip empty runs. +buildInlineContent(schema: Schema, blockChunks: Chunk[]): Node[] + +// fold groups into nodes: paragraph/heading{level}/code_block{language}/ +// blockquote(paragraph)/list depth-stack (nested bullet_list/ordered_list, ordinal→order, +// task attr)/table grouping by tableId with rowIndex/columnIndex ordering + cellId dedup +// (port buildTableRows, +page.svelte:334-373). Empty buffer → doc(paragraph). +// try/catch per block → fallback plain paragraph so mid-stream states never throw. +buildDocFromChunks(schema: Schema, chunks: Chunk[]): Node +``` + +Notes: trim one trailing `\n` per non-code block group; tolerate partial table rows mid-stream; open link/image spans have no url/src until closed — render as plain text until closure (rebuild fixes retroactively). + +### 3. `editor.ts` +```ts +createStreamRenderer(mount: HTMLElement): StreamRenderer +// StreamRenderer: { handleStreamingChunk(parsed: StreamingChunk): void; reset(): void; destroy(): void } +``` +- `new EditorView(mount, { state: EditorState.create({schema, doc: emptyDoc}), editable: () => false })` +- Private non-reactive `buffer: Chunk[]`. On `STREAMING`: update buffer, `nextDoc = buildDocFromChunks(...)`, skip if `nextDoc.eq(state.doc)`, else `dispatch(tr.replaceWith(0, doc.content.size, nextDoc.content))` +- On `START_STREAM`: internal `reset()` (restart per stream, not append across runs). `console.warn` on `recovery.type === 'window_overflow'`. + +### 4. `ProseMirrorRenderer.svelte` +Mirrors Lixpi's `ProseMirror.svelte` mount pattern: `bind:this={mountEl}` div with `class="prose prose-sm max-w-none"`, `onMount` → `createStreamRenderer`, `onDestroy` → `destroy()`. Exports `handleStreamingChunk` / `reset` for `bind:this` use from the page. + +### 5. `prosemirror.css` +- `.ProseMirror { outline: none; word-wrap: break-word; }` (no global `pre-wrap`; trim newlines in assembly instead) +- Task-list checkboxes via `li[data-task]::before` (☐/☑), code-block language badge via `pre[data-language]::after`, table `th/td` borders to match old look. + +## Modified files + +### `demo/svelte-demo/src/routes/+page.svelte` +- Replace the `{#each parsedBlocks ...}` markup (lines ~596–829) with `` in the same card div. +- Subscription callback (~135–198): add `pmRenderer?.handleStreamingChunk(parsed)`; keep `parsedSegments` accumulation + backtrack filtering (feeds debug columns) and the backtrack `console.warn`. +- `resetParser()` (~302): add `pmRenderer?.reset()`. +- Delete dead code: `parsedBlocks` reactive block, `buildTableRows`, `getTableAlignClass`, `getTableCellAlignClass`, `isTableCellBlockType`, `getSpanClasses`, `hasCodeStyle`, `getActiveSpanTypes`, `updateOpenSpans` + `openSpans` state, `TableCellGroup`/`TableRowGroup`/`TableAlign` types. +- Keep: example picker, delay slider, play/pause/step/reset, and all debug columns (Current Token, Parsed Chunks JSON, raw tokens, concatenated txt). + +### `demo/svelte-demo/package.json` and `demo/svelte-demo/pnpm-lock.yaml` +Add direct dependencies actually imported by the implementation: `prosemirror-model`, `prosemirror-state`, and `prosemirror-view`. Add `prosemirror-transform` only if implementation code imports it directly. Add `vitest` as a devDependency plus a `test` script because the demo package currently has `check` but no test runner. + +### `demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts` +Add focused unit tests for the pure assembly layer, including URL sanitization and shared backtrack behavior. + +### `demo/svelte-demo/src/app.css` +Add `@import './lib/prosemirror/prosemirror.css';` (Tailwind v4 CSS-first; typography plugin already loaded). + +## Implementation order + +1. `docker compose up -d`; then `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo add prosemirror-model prosemirror-state prosemirror-view` (add `prosemirror-transform` only if directly imported) +2. Add the demo test runner: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo add -D vitest`, then add a `test` script. +3. Implement `schema.ts` +4. Implement `stream-assembly.ts` (port grouping/table logic from `+page.svelte`) +5. Add `stream-assembly.test.ts` for the pure assembly layer. +6. Implement `editor.ts` + `ProseMirrorRenderer.svelte` + `prosemirror.css` + `app.css` import +7. Wire into `+page.svelte`, delete legacy rendering +8. Run tests and typecheck: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run test` and `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run check` + +## Verification (all inside the container) + +1. Add focused unit tests for `stream-assembly.ts`: backtrack filtering shared by buffer/debug paths, nested lists, ordered-list start attrs, task lists, tables, link/image URL sanitization and rejection, open/closed/contained spans, zero-length text skipping, and malformed mid-stream states falling back instead of throwing. +2. Run the unit tests inside the container. +3. `docker exec -d lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run dev` (predev regenerates the manifest; binds 0.0.0.0:5173 → host 5173). Open `http://localhost:5173`. +4. Exercise examples from `demo/svelte-demo/static/llm-streams-examples/`: + - headings/paragraphs/bold/italic/lists: `claude-3.5-1-quantum-physics`, `gpt-4.o-history-of-cats` + - fenced code blocks + language: `claude-3.7-happy-number-5-programs`, `gpt-4.5-cat-coding` + - **backtracking**: `claude-3.7-markdown-with-nested-code-block`, `test-error-recovery` — watch for `⚠️ BACKTRACK` console warning; PM doc must self-correct with no stale/duplicated text + - strikethrough: `test-strikethrough`; find table/task-list examples via `grep -l '|' static/llm-streams-examples/*.txt` and `grep -l '\- \['` +5. Controls: full play to END_STREAM; pause + single-step (doc updates chunk-by-chunk); reset mid-stream (doc clears); replay after completion (restarts, doesn't append); switch example mid-stream. +6. Debug columns still behave identically. +7. `pnpm --dir demo/svelte-demo run check` passes. + +## Risks / notes + +- Backtrack filter predicate `chunk.offset + chunk.length <= backtrackOffset` is the proven one from the legacy code — expose it once and reuse it for both the ProseMirror chunk buffer and debug `parsedSegments`. +- ProseMirror `toDOM` specs are nested arrays; strings like `table>tbody` are documentation shorthand only and must not be used as literal tag names. +- Sanitization belongs before ProseMirror mark/node creation; do not rely on DOM escaping to make `href`/`src` safe. Implement it as exported pure helpers so behavior is unit-testable without a browser. +- Legacy list-item newline grouping heuristic is imperfect; keep for parity plus the added depth/type boundary rule. +- ProseMirror text nodes can't be empty — skip zero-length runs. +- `+page.svelte` uses Svelte legacy syntax under Svelte 5 (`$:`/`on:click`) — keep new code consistent (onMount/bind:this), don't convert the page to runes. From 05415e30187ed42bd16b798d480b34316696debe Mon Sep 17 00:00:00 2001 From: Dmitry Bondarenko Date: Wed, 8 Jul 2026 22:53:21 +0600 Subject: [PATCH 2/4] Moves demo to ProseMirror --- PLAN.md | 58 +- demo/svelte-demo/package.json | 9 +- demo/svelte-demo/pnpm-lock.yaml | 603 ++++++++++++++++++ demo/svelte-demo/src/app.css | 1 + .../prosemirror/ProseMirrorRenderer.svelte | 30 + .../svelte-demo/src/lib/prosemirror/editor.ts | 51 ++ .../src/lib/prosemirror/prosemirror.css | 72 +++ .../svelte-demo/src/lib/prosemirror/schema.ts | 163 +++++ .../lib/prosemirror/stream-assembly.test.ts | 158 +++++ .../src/lib/prosemirror/stream-assembly.ts | 465 ++++++++++++++ .../stream-examples.integration.test.ts | 143 +++++ demo/svelte-demo/src/routes/+page.svelte | 461 +------------ demo/svelte-demo/tsconfig.json | 1 + demo/svelte-demo/vitest.config.ts | 9 + src/tree-sitter-markdown-stream-parser.ts | 6 +- src/tree-sitter/inline-detection.ts | 3 + src/tree-sitter/segment-generator.ts | 189 +++--- vitest.config.ts | 1 + 18 files changed, 1861 insertions(+), 562 deletions(-) create mode 100644 demo/svelte-demo/src/lib/prosemirror/ProseMirrorRenderer.svelte create mode 100644 demo/svelte-demo/src/lib/prosemirror/editor.ts create mode 100644 demo/svelte-demo/src/lib/prosemirror/prosemirror.css create mode 100644 demo/svelte-demo/src/lib/prosemirror/schema.ts create mode 100644 demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts create mode 100644 demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts create mode 100644 demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts create mode 100644 demo/svelte-demo/vitest.config.ts diff --git a/PLAN.md b/PLAN.md index 7ed323b..d6ce584 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,10 +1,32 @@ # Migrate svelte-demo rendering to ProseMirror (LIX-MDSP-20) +## Completed follow-up review findings (2026-07-08) + +The plan below has been implemented (new module in `demo/svelte-demo/src/lib/prosemirror/`, legacy rendering removed from `+page.svelte`). The post-implementation review items were applied inside the docker container `lixpi-markdown-stream-parser-demo`. + +### 1. Fixed the demo test run +Added `demo/svelte-demo/vitest.config.ts` so the demo runner includes only `src/**/*.test.ts` and excludes generated `.svelte-kit/**` output. `pnpm --dir demo/svelte-demo run test` now passes with the real ProseMirror unit/integration tests only. + +### 2. Restored strict demo typechecking +Restored `"strict": true` in `demo/svelte-demo/tsconfig.json` and kept `"allowImportingTsExtensions": true`. Strict-mode errors came from the root parser source imported by the demo, so nullable tree-sitter parse results now have explicit guards in `src/tree-sitter/inline-detection.ts` and `src/tree-sitter/segment-generator.ts`. `pnpm --dir demo/svelte-demo run check` now reports 0 errors. + +### 3. Styled open spans during streaming +`buildInlineContent` now tracks open non-link/image spans across block chunks and applies live marks for bold/italic/code/strikethrough until a matching closing span or the current buffer end. Link/image behavior stays closed-only for URL metadata safety. Added unit tests for a multi-chunk bold span and an unclosed bold span at the end of the current buffer. + +### 4. Reviewed out-of-scope edits +- Kept the type-only `Edit` cast in `src/tree-sitter-markdown-stream-parser.ts` because current `web-tree-sitter` types require `editPoint`/`editRange` even though the runtime accepts the existing edit shape. Removing it breaks strict demo typechecking. +- Reverted the root Vitest expansion so root tests stay scoped to `src/**/*.test.ts`; demo tests run through `pnpm --dir demo/svelte-demo run test`. + +### 5. Verified examples and controls path +The dev server is running at `http://localhost:5173` and responds with HTTP 200. The container has no Chromium/Firefox/Playwright/Puppeteer binary, so real browser automation could not be executed there. Added `stream-examples.integration.test.ts` to replay the real JSON token streams through the parser and ProseMirror assembly, covering heading/list/emphasis examples, code blocks, backtracking self-correction with no stale active text, strikethrough, tables, task-list metadata, reset, replay after completion, and switching examples at the buffer level. + +Everything else was verified as conforming: module layering and function signatures (including the `buildContentFromChunks(schema, chunks): Fragment` core required for Lixpi portability), schema node/mark parity with Lixpi, URL sanitization rules and plain-text fallback, editor wiring, legacy-code deletion in `+page.svelte` with the shared `isChunkBeforeBacktrack` used by both debug and renderer paths, dependency set, CSS, and `svelte-check` (0 errors). + ## Context The demo at `demo/svelte-demo` currently renders the parser's streaming output with an ad-hoc, hand-written rendering layer inside `src/routes/+page.svelte` (~898 lines): a reactive block-grouping state machine (`parsedBlocks`), manual open-span tracking, table reconstruction (`buildTableRows`), and a giant `{#each}/{#if}` markup tree with Tailwind span classes. This is the "legacy state-machine code" to replace. -Goal: replace that rendering layer with ProseMirror, following the architecture of the Lixpi main repo (workspace-local shallow clone currently available at `/tmp/claude-1000/-home-dima-Desktop-markdown-stream-parser/8b99adf2-a3b8-4241-898c-740929163041/scratchpad/lixpi-ref`; do not rely on this path outside this workspace): a framework-free ProseMirror module (schema + pure stream-assembly functions) driving an `EditorView` via transactions. +Goal: replace that rendering layer with ProseMirror, following the architecture of the Lixpi main repo (local checkout at `/home/dima/Desktop/lixpi`; reference files in `packages/lixpi/prosemirror/src/`): a framework-free ProseMirror module (schema + pure stream-assembly functions) driving an `EditorView` via transactions. **Key adaptation vs Lixpi:** Lixpi's `packages/lixpi/prosemirror/src/stream-assembly.ts` consumes the OLD parser segment shape (`{segment, styles[], type, isBlockDefining}`). This repo's tree-sitter parser emits a new offset-based `Chunk` shape (`src/tree-sitter/types.ts`): `{text, offset, length, block:{type, level?, language?, list?, table?}, opening/closing/contained spans, backtrackOffset?, recovery?}` wrapped in `StreamingChunk` (`START_STREAM | STREAMING | END_STREAM`). Lixpi's schema also lacks list/table nodes, which this parser emits. So we replicate the *architecture*, not the code verbatim. @@ -17,11 +39,11 @@ Goal: replace that rendering layer with ProseMirror, following the architecture ## New files (all under `demo/svelte-demo/src/lib/prosemirror/`) ### 1. `schema.ts` -Adapt Lixpi's `base-schema.ts` (scratchpad ref above), extend with lists/tables, drop lixpi-only nodes. Read-only view ⇒ `parseDOM` optional. +Adapt Lixpi's `base-schema.ts` (`/home/dima/Desktop/lixpi/packages/lixpi/prosemirror/src/base-schema.ts`), extend with lists/tables, drop lixpi-only nodes. Read-only view ⇒ `parseDOM` optional. Nodes: - `doc` (`block+`), `paragraph` (`inline*` → `['p', 0]`), `heading` (attr `level`, → `h1..h6`), `code_block` (attr `language`, `content:'text*'`, `marks:''`, `code:true`, → `['pre', {'data-language': language}, ['code', 0]]`), `blockquote` (`block+`), `text` -- `bullet_list` (`list_item+` → `['ul', 0]`), `ordered_list` (attr `order` → `['ol', {start}, 0]`), `list_item` (attr `task: null|{checked}`, `content:'block+'`, → `['li', {'data-task': checked|unchecked}, 0]`) +- `bullet_list` (`list_item+` → `['ul', 0]`), `ordered_list` (attr `order` → `['ol', {start}, 0]`), `list_item` (attr `task: null|{checked}`, `content:'block+'`, → `['li', {'data-task': 'checked'|'unchecked'}, 0]`; attribute omitted entirely for non-task items) - `table` (`table_row+` → `['table', ['tbody', 0]]`), `table_row` (`(table_header_cell|table_cell)+` → `['tr', 0]`), `table_header_cell`/`table_cell` (attr `align`, `content:'inline*'`, → `['th'|'td', {style:'text-align: ...'}, 0]`) - `image` — **inline** (`inline:true`, group `inline`, attrs `src`, `alt`) since the parser emits images as inline spans @@ -82,13 +104,13 @@ Mirrors Lixpi's `ProseMirror.svelte` mount pattern: `bind:this={mountEl}` div wi ### 5. `prosemirror.css` - `.ProseMirror { outline: none; word-wrap: break-word; }` (no global `pre-wrap`; trim newlines in assembly instead) -- Task-list checkboxes via `li[data-task]::before` (☐/☑), code-block language badge via `pre[data-language]::after`, table `th/td` borders to match old look. +- Task-list checkboxes via `li[data-task="unchecked"]::before` (☐) / `li[data-task="checked"]::before` (☑) plus `list-style: none` on task items, code-block language badge via `pre[data-language]::after { content: attr(data-language) }`, table `th/td` borders to match old look. ## Modified files ### `demo/svelte-demo/src/routes/+page.svelte` - Replace the `{#each parsedBlocks ...}` markup (lines ~596–829) with `` in the same card div. -- Subscription callback (~135–198): add `pmRenderer?.handleStreamingChunk(parsed)`; keep `parsedSegments` accumulation + backtrack filtering (feeds debug columns) and the backtrack `console.warn`. +- Subscription callback (~135–198): add `pmRenderer?.handleStreamingChunk(parsed)`; keep `parsedSegments` accumulation + backtrack filtering (feeds debug columns) and the backtrack `console.warn`. Replace the page's inline backtrack predicate (`seg.chunk.offset + seg.chunk.length <= chunk.backtrackOffset`, line ~183) with the shared `isChunkBeforeBacktrack` import so debug and renderer paths cannot drift. - `resetParser()` (~302): add `pmRenderer?.reset()`. - Delete dead code: `parsedBlocks` reactive block, `buildTableRows`, `getTableAlignClass`, `getTableCellAlignClass`, `isTableCellBlockType`, `getSpanClasses`, `hasCodeStyle`, `getActiveSpanTypes`, `updateOpenSpans` + `openSpans` state, `TableCellGroup`/`TableRowGroup`/`TableAlign` types. - Keep: example picker, delay slider, play/pause/step/reset, and all debug columns (Current Token, Parsed Chunks JSON, raw tokens, concatenated txt). @@ -97,7 +119,7 @@ Mirrors Lixpi's `ProseMirror.svelte` mount pattern: `bind:this={mountEl}` div wi Add direct dependencies actually imported by the implementation: `prosemirror-model`, `prosemirror-state`, and `prosemirror-view`. Add `prosemirror-transform` only if implementation code imports it directly. Add `vitest` as a devDependency plus a `test` script because the demo package currently has `check` but no test runner. ### `demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts` -Add focused unit tests for the pure assembly layer, including URL sanitization and shared backtrack behavior. +Add focused unit tests for the pure assembly layer: backtrack filtering shared by buffer/debug paths, nested lists, ordered-list start attrs, task lists, tables, link/image URL sanitization and rejection, open/closed/contained spans, zero-length text skipping, and malformed mid-stream states falling back instead of throwing. Test style reference: `/home/dima/Desktop/lixpi/packages/lixpi/prosemirror/src/stream-assembly.test.ts`. ### `demo/svelte-demo/src/app.css` Add `@import './lib/prosemirror/prosemirror.css';` (Tailwind v4 CSS-first; typography plugin already loaded). @@ -105,7 +127,7 @@ Add `@import './lib/prosemirror/prosemirror.css';` (Tailwind v4 CSS-first; typog ## Implementation order 1. `docker compose up -d`; then `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo add prosemirror-model prosemirror-state prosemirror-view` (add `prosemirror-transform` only if directly imported) -2. Add the demo test runner: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo add -D vitest`, then add a `test` script. +2. Add the demo test runner: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo add -D vitest`, then add a `"test": "vitest run"` script (non-watch mode; `pnpm add` will also trigger the demo's `postinstall` WASM copy — expected). 3. Implement `schema.ts` 4. Implement `stream-assembly.ts` (port grouping/table logic from `+page.svelte`) 5. Add `stream-assembly.test.ts` for the pure assembly layer. @@ -115,17 +137,25 @@ Add `@import './lib/prosemirror/prosemirror.css';` (Tailwind v4 CSS-first; typog ## Verification (all inside the container) -1. Add focused unit tests for `stream-assembly.ts`: backtrack filtering shared by buffer/debug paths, nested lists, ordered-list start attrs, task lists, tables, link/image URL sanitization and rejection, open/closed/contained spans, zero-length text skipping, and malformed mid-stream states falling back instead of throwing. -2. Run the unit tests inside the container. -3. `docker exec -d lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run dev` (predev regenerates the manifest; binds 0.0.0.0:5173 → host 5173). Open `http://localhost:5173`. -4. Exercise examples from `demo/svelte-demo/static/llm-streams-examples/`: +1. Unit tests from implementation step 5 pass: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run test`. +2. `docker exec -d lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run dev` (predev regenerates the manifest; binds 0.0.0.0:5173 → host 5173). Open `http://localhost:5173`. +3. Exercise examples from `demo/svelte-demo/static/llm-streams-examples/`: - headings/paragraphs/bold/italic/lists: `claude-3.5-1-quantum-physics`, `gpt-4.o-history-of-cats` - fenced code blocks + language: `claude-3.7-happy-number-5-programs`, `gpt-4.5-cat-coding` - **backtracking**: `claude-3.7-markdown-with-nested-code-block`, `test-error-recovery` — watch for `⚠️ BACKTRACK` console warning; PM doc must self-correct with no stale/duplicated text - strikethrough: `test-strikethrough`; find table/task-list examples via `grep -l '|' static/llm-streams-examples/*.txt` and `grep -l '\- \['` -5. Controls: full play to END_STREAM; pause + single-step (doc updates chunk-by-chunk); reset mid-stream (doc clears); replay after completion (restarts, doesn't append); switch example mid-stream. -6. Debug columns still behave identically. -7. `pnpm --dir demo/svelte-demo run check` passes. +4. Controls: full play to END_STREAM; pause + single-step (doc updates chunk-by-chunk); reset mid-stream (doc clears); replay after completion (restarts, doesn't append); switch example mid-stream. +5. Debug columns still behave identically. +6. `pnpm --dir demo/svelte-demo run check` passes. + +## Portability to main Lixpi + +This parser is a tool for main Lixpi, which still consumes the deprecated legacy parser shape (`{segment, styles[], isBlockDefining}`). This demo's module is the reference implementation for the new `Chunk` shape → ProseMirror mapping, so preserve these guarantees during implementation: + +- `stream-assembly.ts` stays framework-free and schema-parameterized (no import of the demo schema instance) so it runs against Lixpi's `createProseMirrorSchema(...)` schemas unchanged. +- Node/mark names stay identical to Lixpi's `base-schema.ts` (`paragraph`, `heading`, `code_block`, `blockquote`, `strong`, `em`, `code`, `strikethrough`, `link`); new list/table NodeSpecs are plain spec objects portable into Lixpi's `node-specs.ts`. +- `buildDocFromChunks` must be a thin wrapper over a `buildContentFromChunks(schema, chunks): Fragment` core. The demo replaces the whole doc; main Lixpi will instead rebuild the content of a target node (e.g. `aiResponseMessage`) and emit one bounded ReplaceStep per update — compatible with its `HeadlessProseMirrorEngine` + step-publishing pipeline without whole-doc steps. +- URL sanitizers remain pure/exported for reuse in Lixpi's server-side assembler. ## Risks / notes diff --git a/demo/svelte-demo/package.json b/demo/svelte-demo/package.json index 2d1da56..a24ef4d 100644 --- a/demo/svelte-demo/package.json +++ b/demo/svelte-demo/package.json @@ -7,6 +7,7 @@ "serve": "http-server ./build -p 5173", "build-and-serve": "pnpm run build && pnpm run serve", "preview": "vite preview", + "test": "vitest run", "prepare": "svelte-kit sync || echo ''", "prepack": "svelte-kit sync && svelte-package && publint", "check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json", @@ -54,6 +55,7 @@ "ts-node": "^10.9.2", "typescript": "^5.0.0", "vite": "^6.2.6", + "vitest": "^2.0.0", "web-tree-sitter": "*" }, "keywords": [ @@ -65,6 +67,9 @@ ] }, "dependencies": { - "@lixpi/markdown-stream-parser": "0.0.3-31" + "@lixpi/markdown-stream-parser": "0.0.3-31", + "prosemirror-model": "^1.25.10", + "prosemirror-state": "^1.4.4", + "prosemirror-view": "^1.42.0" } -} \ No newline at end of file +} diff --git a/demo/svelte-demo/pnpm-lock.yaml b/demo/svelte-demo/pnpm-lock.yaml index 71cdb65..69d19ec 100644 --- a/demo/svelte-demo/pnpm-lock.yaml +++ b/demo/svelte-demo/pnpm-lock.yaml @@ -11,6 +11,15 @@ importers: '@lixpi/markdown-stream-parser': specifier: 0.0.3-31 version: 0.0.3-31 + prosemirror-model: + specifier: ^1.25.10 + version: 1.25.10 + prosemirror-state: + specifier: ^1.4.4 + version: 1.4.4 + prosemirror-view: + specifier: ^1.42.0 + version: 1.42.0 devDependencies: '@sveltejs/adapter-auto': specifier: ^6.0.0 @@ -60,6 +69,9 @@ importers: vite: specifier: ^6.2.6 version: 6.3.3(@types/node@22.15.3)(jiti@2.4.2)(lightningcss@1.29.2) + vitest: + specifier: ^2.0.0 + version: 2.1.9(@types/node@22.15.3)(lightningcss@1.29.2) web-tree-sitter: specifier: '*' version: 0.25.10 @@ -74,102 +86,204 @@ packages: resolution: {integrity: sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==} engines: {node: '>=12'} + '@esbuild/aix-ppc64@0.21.5': + resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==} + engines: {node: '>=12'} + cpu: [ppc64] + os: [aix] + '@esbuild/aix-ppc64@0.25.3': resolution: {integrity: sha512-W8bFfPA8DowP8l//sxjJLSLkD8iEjMc7cBVyP+u4cEv9sM7mdUCkgsj+t0n/BWPFtv7WWCN5Yzj0N6FJNUUqBQ==} engines: {node: '>=18'} cpu: [ppc64] os: [aix] + '@esbuild/android-arm64@0.21.5': + resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==} + engines: {node: '>=12'} + cpu: [arm64] + os: [android] + '@esbuild/android-arm64@0.25.3': resolution: {integrity: sha512-XelR6MzjlZuBM4f5z2IQHK6LkK34Cvv6Rj2EntER3lwCBFdg6h2lKbtRjpTTsdEjD/WSe1q8UyPBXP1x3i/wYQ==} engines: {node: '>=18'} cpu: [arm64] os: [android] + '@esbuild/android-arm@0.21.5': + resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==} + engines: {node: '>=12'} + cpu: [arm] + os: [android] + '@esbuild/android-arm@0.25.3': resolution: {integrity: sha512-PuwVXbnP87Tcff5I9ngV0lmiSu40xw1At6i3GsU77U7cjDDB4s0X2cyFuBiDa1SBk9DnvWwnGvVaGBqoFWPb7A==} engines: {node: '>=18'} cpu: [arm] os: [android] + '@esbuild/android-x64@0.21.5': + resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==} + engines: {node: '>=12'} + cpu: [x64] + os: [android] + '@esbuild/android-x64@0.25.3': resolution: {integrity: sha512-ogtTpYHT/g1GWS/zKM0cc/tIebFjm1F9Aw1boQ2Y0eUQ+J89d0jFY//s9ei9jVIlkYi8AfOjiixcLJSGNSOAdQ==} engines: {node: '>=18'} cpu: [x64] os: [android] + '@esbuild/darwin-arm64@0.21.5': + resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==} + engines: {node: '>=12'} + cpu: [arm64] + os: [darwin] + '@esbuild/darwin-arm64@0.25.3': resolution: {integrity: sha512-eESK5yfPNTqpAmDfFWNsOhmIOaQA59tAcF/EfYvo5/QWQCzXn5iUSOnqt3ra3UdzBv073ykTtmeLJZGt3HhA+w==} engines: {node: '>=18'} cpu: [arm64] os: [darwin] + '@esbuild/darwin-x64@0.21.5': + resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==} + engines: {node: '>=12'} + cpu: [x64] + os: [darwin] + '@esbuild/darwin-x64@0.25.3': resolution: {integrity: sha512-Kd8glo7sIZtwOLcPbW0yLpKmBNWMANZhrC1r6K++uDR2zyzb6AeOYtI6udbtabmQpFaxJ8uduXMAo1gs5ozz8A==} engines: {node: '>=18'} cpu: [x64] os: [darwin] + '@esbuild/freebsd-arm64@0.21.5': + resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==} + engines: {node: '>=12'} + cpu: [arm64] + os: [freebsd] + '@esbuild/freebsd-arm64@0.25.3': resolution: {integrity: sha512-EJiyS70BYybOBpJth3M0KLOus0n+RRMKTYzhYhFeMwp7e/RaajXvP+BWlmEXNk6uk+KAu46j/kaQzr6au+JcIw==} engines: {node: '>=18'} cpu: [arm64] os: [freebsd] + '@esbuild/freebsd-x64@0.21.5': + resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==} + engines: {node: '>=12'} + cpu: [x64] + os: [freebsd] + '@esbuild/freebsd-x64@0.25.3': resolution: {integrity: sha512-Q+wSjaLpGxYf7zC0kL0nDlhsfuFkoN+EXrx2KSB33RhinWzejOd6AvgmP5JbkgXKmjhmpfgKZq24pneodYqE8Q==} engines: {node: '>=18'} cpu: [x64] os: [freebsd] + '@esbuild/linux-arm64@0.21.5': + resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==} + engines: {node: '>=12'} + cpu: [arm64] + os: [linux] + '@esbuild/linux-arm64@0.25.3': resolution: {integrity: sha512-xCUgnNYhRD5bb1C1nqrDV1PfkwgbswTTBRbAd8aH5PhYzikdf/ddtsYyMXFfGSsb/6t6QaPSzxtbfAZr9uox4A==} engines: {node: '>=18'} cpu: [arm64] os: [linux] + '@esbuild/linux-arm@0.21.5': + resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==} + engines: {node: '>=12'} + cpu: [arm] + os: [linux] + '@esbuild/linux-arm@0.25.3': resolution: {integrity: sha512-dUOVmAUzuHy2ZOKIHIKHCm58HKzFqd+puLaS424h6I85GlSDRZIA5ycBixb3mFgM0Jdh+ZOSB6KptX30DD8YOQ==} engines: {node: '>=18'} cpu: [arm] os: [linux] + '@esbuild/linux-ia32@0.21.5': + resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==} + engines: {node: '>=12'} + cpu: [ia32] + os: [linux] + '@esbuild/linux-ia32@0.25.3': resolution: {integrity: sha512-yplPOpczHOO4jTYKmuYuANI3WhvIPSVANGcNUeMlxH4twz/TeXuzEP41tGKNGWJjuMhotpGabeFYGAOU2ummBw==} engines: {node: '>=18'} cpu: [ia32] os: [linux] + '@esbuild/linux-loong64@0.21.5': + resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==} + engines: {node: '>=12'} + cpu: [loong64] + os: [linux] + '@esbuild/linux-loong64@0.25.3': resolution: {integrity: sha512-P4BLP5/fjyihmXCELRGrLd793q/lBtKMQl8ARGpDxgzgIKJDRJ/u4r1A/HgpBpKpKZelGct2PGI4T+axcedf6g==} engines: {node: '>=18'} cpu: [loong64] os: [linux] + '@esbuild/linux-mips64el@0.21.5': + resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==} + engines: {node: '>=12'} + cpu: [mips64el] + os: [linux] + '@esbuild/linux-mips64el@0.25.3': resolution: {integrity: sha512-eRAOV2ODpu6P5divMEMa26RRqb2yUoYsuQQOuFUexUoQndm4MdpXXDBbUoKIc0iPa4aCO7gIhtnYomkn2x+bag==} engines: {node: '>=18'} cpu: [mips64el] os: [linux] + '@esbuild/linux-ppc64@0.21.5': + resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==} + engines: {node: '>=12'} + cpu: [ppc64] + os: [linux] + '@esbuild/linux-ppc64@0.25.3': resolution: {integrity: sha512-ZC4jV2p7VbzTlnl8nZKLcBkfzIf4Yad1SJM4ZMKYnJqZFD4rTI+pBG65u8ev4jk3/MPwY9DvGn50wi3uhdaghg==} engines: {node: '>=18'} cpu: [ppc64] os: [linux] + '@esbuild/linux-riscv64@0.21.5': + resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==} + engines: {node: '>=12'} + cpu: [riscv64] + os: [linux] + '@esbuild/linux-riscv64@0.25.3': resolution: {integrity: sha512-LDDODcFzNtECTrUUbVCs6j9/bDVqy7DDRsuIXJg6so+mFksgwG7ZVnTruYi5V+z3eE5y+BJZw7VvUadkbfg7QA==} engines: {node: '>=18'} cpu: [riscv64] os: [linux] + '@esbuild/linux-s390x@0.21.5': + resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==} + engines: {node: '>=12'} + cpu: [s390x] + os: [linux] + '@esbuild/linux-s390x@0.25.3': resolution: {integrity: sha512-s+w/NOY2k0yC2p9SLen+ymflgcpRkvwwa02fqmAwhBRI3SC12uiS10edHHXlVWwfAagYSY5UpmT/zISXPMW3tQ==} engines: {node: '>=18'} cpu: [s390x] os: [linux] + '@esbuild/linux-x64@0.21.5': + resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==} + engines: {node: '>=12'} + cpu: [x64] + os: [linux] + '@esbuild/linux-x64@0.25.3': resolution: {integrity: sha512-nQHDz4pXjSDC6UfOE1Fw9Q8d6GCAd9KdvMZpfVGWSJztYCarRgSDfOVBY5xwhQXseiyxapkiSJi/5/ja8mRFFA==} engines: {node: '>=18'} @@ -182,6 +296,12 @@ packages: cpu: [arm64] os: [netbsd] + '@esbuild/netbsd-x64@0.21.5': + resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==} + engines: {node: '>=12'} + cpu: [x64] + os: [netbsd] + '@esbuild/netbsd-x64@0.25.3': resolution: {integrity: sha512-i5Hm68HXHdgv8wkrt+10Bc50zM0/eonPb/a/OFVfB6Qvpiirco5gBA5bz7S2SHuU+Y4LWn/zehzNX14Sp4r27g==} engines: {node: '>=18'} @@ -194,30 +314,60 @@ packages: cpu: [arm64] os: [openbsd] + '@esbuild/openbsd-x64@0.21.5': + resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==} + engines: {node: '>=12'} + cpu: [x64] + os: [openbsd] + '@esbuild/openbsd-x64@0.25.3': resolution: {integrity: sha512-fpqctI45NnCIDKBH5AXQBsD0NDPbEFczK98hk/aa6HJxbl+UtLkJV2+Bvy5hLSLk3LHmqt0NTkKNso1A9y1a4w==} engines: {node: '>=18'} cpu: [x64] os: [openbsd] + '@esbuild/sunos-x64@0.21.5': + resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==} + engines: {node: '>=12'} + cpu: [x64] + os: [sunos] + '@esbuild/sunos-x64@0.25.3': resolution: {integrity: sha512-ROJhm7d8bk9dMCUZjkS8fgzsPAZEjtRJqCAmVgB0gMrvG7hfmPmz9k1rwO4jSiblFjYmNvbECL9uhaPzONMfgA==} engines: {node: '>=18'} cpu: [x64] os: [sunos] + '@esbuild/win32-arm64@0.21.5': + resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==} + engines: {node: '>=12'} + cpu: [arm64] + os: [win32] + '@esbuild/win32-arm64@0.25.3': resolution: {integrity: sha512-YWcow8peiHpNBiIXHwaswPnAXLsLVygFwCB3A7Bh5jRkIBFWHGmNQ48AlX4xDvQNoMZlPYzjVOQDYEzWCqufMQ==} engines: {node: '>=18'} cpu: [arm64] os: [win32] + '@esbuild/win32-ia32@0.21.5': + resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==} + engines: {node: '>=12'} + cpu: [ia32] + os: [win32] + '@esbuild/win32-ia32@0.25.3': resolution: {integrity: sha512-qspTZOIGoXVS4DpNqUYUs9UxVb04khS1Degaw/MnfMe7goQ3lTfQ13Vw4qY/Nj0979BGvMRpAYbs/BAxEvU8ew==} engines: {node: '>=18'} cpu: [ia32] os: [win32] + '@esbuild/win32-x64@0.21.5': + resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==} + engines: {node: '>=12'} + cpu: [x64] + os: [win32] + '@esbuild/win32-x64@0.25.3': resolution: {integrity: sha512-ICgUR+kPimx0vvRzf+N/7L7tVSQeE3BYY+NhHRHXS1kBuPO7z2+7ea2HbhDyZdTephgvNvKrlDDKUexuCVBVvg==} engines: {node: '>=18'} @@ -522,6 +672,35 @@ packages: '@types/node@22.15.3': resolution: {integrity: sha512-lX7HFZeHf4QG/J7tBZqrCAXwz9J5RD56Y6MpP0eJkka8p+K0RY/yBTW7CYFJ4VGCclxqOLKmiGP5juQc6MKgcw==} + '@vitest/expect@2.1.9': + resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==} + + '@vitest/mocker@2.1.9': + resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==} + peerDependencies: + msw: ^2.4.9 + vite: ^5.0.0 + peerDependenciesMeta: + msw: + optional: true + vite: + optional: true + + '@vitest/pretty-format@2.1.9': + resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==} + + '@vitest/runner@2.1.9': + resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==} + + '@vitest/snapshot@2.1.9': + resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==} + + '@vitest/spy@2.1.9': + resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==} + + '@vitest/utils@2.1.9': + resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==} + acorn-walk@8.3.4: resolution: {integrity: sha512-ueEepnujpqee2o5aIYnvHU6C0A42MNdsIDeqy5BydrkuC5R1ZuUFnm27EeFJGoEHJQgn3uleRvmTXaJgfXbt4g==} engines: {node: '>=0.4.0'} @@ -542,6 +721,10 @@ packages: resolution: {integrity: sha512-COROpnaoap1E2F000S62r6A60uHZnmlvomhfyT2DlTcrY1OrBKn2UhH7qn5wTC9zMvD0AY7csdPSNwKP+7WiQw==} engines: {node: '>= 0.4'} + assertion-error@2.0.1: + resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==} + engines: {node: '>=12'} + async@3.2.6: resolution: {integrity: sha512-htCUDlxyyCLMgaM3xXg0C0LW2xqfuQ6p05pCEIsXuyQ+a1koYKTuBMzRNwmybfLgvJDMd0r1LTn4+E0Ti6C2AA==} @@ -553,6 +736,10 @@ packages: resolution: {integrity: sha512-NF+epuEdnUYVlGuhaxbbq+dvJttwLnGY+YixlXlME5KpQ5W3CnXA5cVTneY3SPbPDRkcjMbifrwmFYcClgOZeg==} engines: {node: '>= 0.8'} + cac@6.7.14: + resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==} + engines: {node: '>=8'} + call-bind-apply-helpers@1.0.2: resolution: {integrity: sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==} engines: {node: '>= 0.4'} @@ -561,10 +748,18 @@ packages: resolution: {integrity: sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==} engines: {node: '>= 0.4'} + chai@5.3.3: + resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==} + engines: {node: '>=18'} + chalk@4.1.2: resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} engines: {node: '>=10'} + check-error@2.1.3: + resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==} + engines: {node: '>= 16'} + chokidar@4.0.3: resolution: {integrity: sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==} engines: {node: '>= 14.16.0'} @@ -608,6 +803,10 @@ packages: dedent-js@1.0.1: resolution: {integrity: sha512-OUepMozQULMLUmhxS95Vudo0jb0UchLimi3+pQ2plj61Fcy8axbP9hbiD4Sz6DPqn6XG3kfmziVfQ1rSys5AJQ==} + deep-eql@5.0.2: + resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==} + engines: {node: '>=6'} + deepmerge@4.3.1: resolution: {integrity: sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A==} engines: {node: '>=0.10.0'} @@ -639,10 +838,18 @@ packages: resolution: {integrity: sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==} engines: {node: '>= 0.4'} + es-module-lexer@1.7.0: + resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==} + es-object-atoms@1.1.1: resolution: {integrity: sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==} engines: {node: '>= 0.4'} + esbuild@0.21.5: + resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==} + engines: {node: '>=12'} + hasBin: true + esbuild@0.25.3: resolution: {integrity: sha512-qKA6Pvai73+M2FtftpNKRxJ78GIjmFXFxd/1DVBqGo/qNhLSfv+G12n9pNoWdytJC8U00TrViOwpjT0zgqQS8Q==} engines: {node: '>=18'} @@ -654,9 +861,16 @@ packages: esrap@1.4.6: resolution: {integrity: sha512-F/D2mADJ9SHY3IwksD4DAXjTt7qt7GWUf3/8RhCNWmC/67tyb55dpimHmy7EplakFaflV0R/PC+fdSPqrRHAQw==} + estree-walker@3.0.3: + resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==} + eventemitter3@4.0.7: resolution: {integrity: sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw==} + expect-type@1.4.0: + resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==} + engines: {node: '>=12.0.0'} + fdir@6.4.4: resolution: {integrity: sha512-1NZP+GK4GfuAv3PqKvxQRDMjdSRZjnkq7KfhlNrCNNlZ0ygQFpebfrnfnq/W7fpUnAv9aGWmY1zKx7FYL3gwhg==} peerDependencies: @@ -820,6 +1034,9 @@ packages: lodash.merge@4.6.2: resolution: {integrity: sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ==} + loupe@3.2.1: + resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==} + lower-case@2.0.2: resolution: {integrity: sha512-7fm3l3NAF9WfN6W3JOmf5drwpVqX78JtoGJ3A6W0a6ZnldM41w2fV5D490psKFTpMds8TJse/eHLFFsNHHjHgg==} @@ -872,12 +1089,22 @@ packages: resolution: {integrity: sha512-ur5UIdyw5Y7yEj9wLzhqXiy6GZ3Mwx0yGI+5sMn2r0N0v3cKJvUmFH5yPP+WXh9e0xfyzyJX95D8l088DNFj7A==} hasBin: true + orderedmap@2.1.1: + resolution: {integrity: sha512-TvAWxi0nDe1j/rtMcWcIj94+Ffe6n7zhow33h40SKxmsmozs6dz/e+EajymfoFcHd7sxNn8yHM8839uixMOV6g==} + package-manager-detector@1.2.0: resolution: {integrity: sha512-PutJepsOtsqVfUsxCzgTTpyXmiAgvKptIgY4th5eq5UXXFhj5PxfQ9hnGkypMeovpAvVshFRItoFHYO18TCOqA==} pascal-case@3.1.2: resolution: {integrity: sha512-uWlGT3YSnK9x3BQJaOdcZwrnV6hPpd8jFH1/ucpiLRPh/2zCVJKS19E4GvYHvaCcACn3foXZ0cLB9Wrx1KGe5g==} + pathe@1.1.2: + resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==} + + pathval@2.0.1: + resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==} + engines: {node: '>= 14.16'} + picocolors@1.1.1: resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} @@ -897,6 +1124,18 @@ packages: resolution: {integrity: sha512-dle9A3yYxlBSrt8Fu+IpjGT8SY8hN0mlaA6GY8t0P5PjIOZemULz/E2Bnm/2dcUOena75OTNkHI76uZBNUUq3A==} engines: {node: ^10 || ^12 || >=14} + prosemirror-model@1.25.10: + resolution: {integrity: sha512-9n6rH4DbJU1eH4SxLt6Y0HhJIo6cZsb7DJ/30uob1hOKPeO6TAaMWI2tc7kwR92BjfPOU2fFHWbZLovLi3XQfA==} + + prosemirror-state@1.4.4: + resolution: {integrity: sha512-6jiYHH2CIGbCfnxdHbXZ12gySFY/fz/ulZE333G6bPqIZ4F+TXo9ifiR86nAHpWnfoNjOb3o5ESi7J8Uz1jXHw==} + + prosemirror-transform@1.12.0: + resolution: {integrity: sha512-GxboyN4AMIsoHNtz5uf2r2Ru551i5hWeCMD6E2Ib4Eogqoub0NflniaBPVQ4MrGE5yZ8JV9tUHg9qcZTTrcN4w==} + + prosemirror-view@1.42.0: + resolution: {integrity: sha512-N54DF3OXNWDuP81G1kbfCys8ZzIjuL1VnvJ2mk5STSu/fNxWIcX/EutQLA3s9KR/2wVhgDi4hzBB/1fINVxk0A==} + publint@0.3.12: resolution: {integrity: sha512-1w3MMtL9iotBjm1mmXtG3Nk06wnq9UhGNRpQ2j6n1Zq7YAD6gnxMMZMIxlRPAydVjVbjSm+n0lhwqsD1m4LD5w==} engines: {node: '>=18'} @@ -955,6 +1194,9 @@ packages: resolution: {integrity: sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==} engines: {node: '>= 0.4'} + siginfo@2.0.0: + resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==} + sirv@3.0.1: resolution: {integrity: sha512-FoqMu0NCGBLCcAkS1qA+XJIQTR6/JHfQXl+uGteNCQ76T91DMUjPa9xfmeqMY3z80nLSg9yQmNjK0Px6RWsH/A==} engines: {node: '>=18'} @@ -963,6 +1205,12 @@ packages: resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} engines: {node: '>=0.10.0'} + stackback@0.0.2: + resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==} + + std-env@3.10.0: + resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==} + supports-color@7.2.0: resolution: {integrity: sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==} engines: {node: '>=8'} @@ -992,10 +1240,28 @@ packages: resolution: {integrity: sha512-GNzQvQTOIP6RyTfE2Qxb8ZVlNmw0n88vp1szwWRimP02mnTsx3Wtn5qRdqY9w2XduFNUgvOwhNnQsjwCp+kqaQ==} engines: {node: '>=6'} + tinybench@2.9.0: + resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==} + + tinyexec@0.3.2: + resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==} + tinyglobby@0.2.13: resolution: {integrity: sha512-mEwzpUgrLySlveBwEVDMKk5B57bhLPYovRfPAXD5gA/98Opn0rCDj3GtLwFvCvH5RK9uPCExUROW5NjDwvqkxw==} engines: {node: '>=12.0.0'} + tinypool@1.1.1: + resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==} + engines: {node: ^18.0.0 || >=20.0.0} + + tinyrainbow@1.2.0: + resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==} + engines: {node: '>=14.0.0'} + + tinyspy@3.0.2: + resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==} + engines: {node: '>=14.0.0'} + totalist@3.0.1: resolution: {integrity: sha512-sf4i37nQ2LBx4m3wB74y+ubopq6W/dIzXg0FDGjsYnZHVa1Da8FH853wlL2gtUhg+xJXjfk3kUZS3BRoQeoQBQ==} engines: {node: '>=6'} @@ -1038,6 +1304,42 @@ packages: v8-compile-cache-lib@3.0.1: resolution: {integrity: sha512-wa7YjyUGfNZngI/vtK0UHAN+lgDCxBPCylVXGp0zu59Fz5aiGtNXaq3DhIov063MorB+VfufLh3JlF2KdTK3xg==} + vite-node@2.1.9: + resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==} + engines: {node: ^18.0.0 || >=20.0.0} + hasBin: true + + vite@5.4.21: + resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==} + engines: {node: ^18.0.0 || >=20.0.0} + hasBin: true + peerDependencies: + '@types/node': ^18.0.0 || >=20.0.0 + less: '*' + lightningcss: ^1.21.0 + sass: '*' + sass-embedded: '*' + stylus: '*' + sugarss: '*' + terser: ^5.4.0 + peerDependenciesMeta: + '@types/node': + optional: true + less: + optional: true + lightningcss: + optional: true + sass: + optional: true + sass-embedded: + optional: true + stylus: + optional: true + sugarss: + optional: true + terser: + optional: true + vite@6.3.3: resolution: {integrity: sha512-5nXH+QsELbFKhsEfWLkHrvgRpTdGJzqOZ+utSdmPTvwHmvU6ITTm3xx+mRusihkcI8GeC7lCDyn3kDtiki9scw==} engines: {node: ^18.0.0 || ^20.0.0 || >=22.0.0} @@ -1086,6 +1388,31 @@ packages: vite: optional: true + vitest@2.1.9: + resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==} + engines: {node: ^18.0.0 || >=20.0.0} + hasBin: true + peerDependencies: + '@edge-runtime/vm': '*' + '@types/node': ^18.0.0 || >=20.0.0 + '@vitest/browser': 2.1.9 + '@vitest/ui': 2.1.9 + happy-dom: '*' + jsdom: '*' + peerDependenciesMeta: + '@edge-runtime/vm': + optional: true + '@types/node': + optional: true + '@vitest/browser': + optional: true + '@vitest/ui': + optional: true + happy-dom: + optional: true + jsdom: + optional: true + web-tree-sitter@0.25.10: resolution: {integrity: sha512-Y09sF44/13XvgVKgO2cNDw5rGk6s26MgoZPXLESvMXeefBf7i6/73eFurre0IsTW6E14Y0ArIzhUMmjoc7xyzA==} peerDependencies: @@ -1098,6 +1425,11 @@ packages: resolution: {integrity: sha512-p41ogyeMUrw3jWclHWTQg1k05DSVXPLcVxRTYsXUk+ZooOCZLcoYgPZ/HL/D/N+uQPOtcp1me1WhBEaX02mhWg==} engines: {node: '>=12'} + why-is-node-running@2.3.0: + resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==} + engines: {node: '>=8'} + hasBin: true + yn@3.1.1: resolution: {integrity: sha512-Ux4ygGWsu2c7isFWe8Yu1YluJmqVhxqK2cLXNQA5AcC3QfbGNpM7fu0Y8b/z16pXLnFxZYvWhd3fhBY9DLmC6Q==} engines: {node: '>=6'} @@ -1116,78 +1448,147 @@ snapshots: dependencies: '@jridgewell/trace-mapping': 0.3.9 + '@esbuild/aix-ppc64@0.21.5': + optional: true + '@esbuild/aix-ppc64@0.25.3': optional: true + '@esbuild/android-arm64@0.21.5': + optional: true + '@esbuild/android-arm64@0.25.3': optional: true + '@esbuild/android-arm@0.21.5': + optional: true + '@esbuild/android-arm@0.25.3': optional: true + '@esbuild/android-x64@0.21.5': + optional: true + '@esbuild/android-x64@0.25.3': optional: true + '@esbuild/darwin-arm64@0.21.5': + optional: true + '@esbuild/darwin-arm64@0.25.3': optional: true + '@esbuild/darwin-x64@0.21.5': + optional: true + '@esbuild/darwin-x64@0.25.3': optional: true + '@esbuild/freebsd-arm64@0.21.5': + optional: true + '@esbuild/freebsd-arm64@0.25.3': optional: true + '@esbuild/freebsd-x64@0.21.5': + optional: true + '@esbuild/freebsd-x64@0.25.3': optional: true + '@esbuild/linux-arm64@0.21.5': + optional: true + '@esbuild/linux-arm64@0.25.3': optional: true + '@esbuild/linux-arm@0.21.5': + optional: true + '@esbuild/linux-arm@0.25.3': optional: true + '@esbuild/linux-ia32@0.21.5': + optional: true + '@esbuild/linux-ia32@0.25.3': optional: true + '@esbuild/linux-loong64@0.21.5': + optional: true + '@esbuild/linux-loong64@0.25.3': optional: true + '@esbuild/linux-mips64el@0.21.5': + optional: true + '@esbuild/linux-mips64el@0.25.3': optional: true + '@esbuild/linux-ppc64@0.21.5': + optional: true + '@esbuild/linux-ppc64@0.25.3': optional: true + '@esbuild/linux-riscv64@0.21.5': + optional: true + '@esbuild/linux-riscv64@0.25.3': optional: true + '@esbuild/linux-s390x@0.21.5': + optional: true + '@esbuild/linux-s390x@0.25.3': optional: true + '@esbuild/linux-x64@0.21.5': + optional: true + '@esbuild/linux-x64@0.25.3': optional: true '@esbuild/netbsd-arm64@0.25.3': optional: true + '@esbuild/netbsd-x64@0.21.5': + optional: true + '@esbuild/netbsd-x64@0.25.3': optional: true '@esbuild/openbsd-arm64@0.25.3': optional: true + '@esbuild/openbsd-x64@0.21.5': + optional: true + '@esbuild/openbsd-x64@0.25.3': optional: true + '@esbuild/sunos-x64@0.21.5': + optional: true + '@esbuild/sunos-x64@0.25.3': optional: true + '@esbuild/win32-arm64@0.21.5': + optional: true + '@esbuild/win32-arm64@0.25.3': optional: true + '@esbuild/win32-ia32@0.21.5': + optional: true + '@esbuild/win32-ia32@0.25.3': optional: true + '@esbuild/win32-x64@0.21.5': + optional: true + '@esbuild/win32-x64@0.25.3': optional: true @@ -1436,6 +1837,46 @@ snapshots: dependencies: undici-types: 6.21.0 + '@vitest/expect@2.1.9': + dependencies: + '@vitest/spy': 2.1.9 + '@vitest/utils': 2.1.9 + chai: 5.3.3 + tinyrainbow: 1.2.0 + + '@vitest/mocker@2.1.9(vite@5.4.21(@types/node@22.15.3)(lightningcss@1.29.2))': + dependencies: + '@vitest/spy': 2.1.9 + estree-walker: 3.0.3 + magic-string: 0.30.17 + optionalDependencies: + vite: 5.4.21(@types/node@22.15.3)(lightningcss@1.29.2) + + '@vitest/pretty-format@2.1.9': + dependencies: + tinyrainbow: 1.2.0 + + '@vitest/runner@2.1.9': + dependencies: + '@vitest/utils': 2.1.9 + pathe: 1.1.2 + + '@vitest/snapshot@2.1.9': + dependencies: + '@vitest/pretty-format': 2.1.9 + magic-string: 0.30.17 + pathe: 1.1.2 + + '@vitest/spy@2.1.9': + dependencies: + tinyspy: 3.0.2 + + '@vitest/utils@2.1.9': + dependencies: + '@vitest/pretty-format': 2.1.9 + loupe: 3.2.1 + tinyrainbow: 1.2.0 + acorn-walk@8.3.4: dependencies: acorn: 8.14.1 @@ -1450,6 +1891,8 @@ snapshots: aria-query@5.3.2: {} + assertion-error@2.0.1: {} + async@3.2.6: {} axobject-query@4.1.0: {} @@ -1458,6 +1901,8 @@ snapshots: dependencies: safe-buffer: 5.1.2 + cac@6.7.14: {} + call-bind-apply-helpers@1.0.2: dependencies: es-errors: 1.3.0 @@ -1468,11 +1913,21 @@ snapshots: call-bind-apply-helpers: 1.0.2 get-intrinsic: 1.3.0 + chai@5.3.3: + dependencies: + assertion-error: 2.0.1 + check-error: 2.1.3 + deep-eql: 5.0.2 + loupe: 3.2.1 + pathval: 2.0.1 + chalk@4.1.2: dependencies: ansi-styles: 4.3.0 supports-color: 7.2.0 + check-error@2.1.3: {} + chokidar@4.0.3: dependencies: readdirp: 4.1.2 @@ -1499,6 +1954,8 @@ snapshots: dedent-js@1.0.1: {} + deep-eql@5.0.2: {} + deepmerge@4.3.1: {} detect-libc@2.0.4: {} @@ -1522,10 +1979,38 @@ snapshots: es-errors@1.3.0: {} + es-module-lexer@1.7.0: {} + es-object-atoms@1.1.1: dependencies: es-errors: 1.3.0 + esbuild@0.21.5: + optionalDependencies: + '@esbuild/aix-ppc64': 0.21.5 + '@esbuild/android-arm': 0.21.5 + '@esbuild/android-arm64': 0.21.5 + '@esbuild/android-x64': 0.21.5 + '@esbuild/darwin-arm64': 0.21.5 + '@esbuild/darwin-x64': 0.21.5 + '@esbuild/freebsd-arm64': 0.21.5 + '@esbuild/freebsd-x64': 0.21.5 + '@esbuild/linux-arm': 0.21.5 + '@esbuild/linux-arm64': 0.21.5 + '@esbuild/linux-ia32': 0.21.5 + '@esbuild/linux-loong64': 0.21.5 + '@esbuild/linux-mips64el': 0.21.5 + '@esbuild/linux-ppc64': 0.21.5 + '@esbuild/linux-riscv64': 0.21.5 + '@esbuild/linux-s390x': 0.21.5 + '@esbuild/linux-x64': 0.21.5 + '@esbuild/netbsd-x64': 0.21.5 + '@esbuild/openbsd-x64': 0.21.5 + '@esbuild/sunos-x64': 0.21.5 + '@esbuild/win32-arm64': 0.21.5 + '@esbuild/win32-ia32': 0.21.5 + '@esbuild/win32-x64': 0.21.5 + esbuild@0.25.3: optionalDependencies: '@esbuild/aix-ppc64': 0.25.3 @@ -1560,8 +2045,14 @@ snapshots: dependencies: '@jridgewell/sourcemap-codec': 1.5.0 + estree-walker@3.0.3: + dependencies: + '@types/estree': 1.0.7 + eventemitter3@4.0.7: {} + expect-type@1.4.0: {} + fdir@6.4.4(picomatch@4.0.2): optionalDependencies: picomatch: 4.0.2 @@ -1703,6 +2194,8 @@ snapshots: lodash.merge@4.6.2: {} + loupe@3.2.1: {} + lower-case@2.0.2: dependencies: tslib: 2.8.1 @@ -1738,6 +2231,8 @@ snapshots: opener@1.5.2: {} + orderedmap@2.1.1: {} + package-manager-detector@1.2.0: {} pascal-case@3.1.2: @@ -1745,6 +2240,10 @@ snapshots: no-case: 3.0.4 tslib: 2.8.1 + pathe@1.1.2: {} + + pathval@2.0.1: {} + picocolors@1.1.1: {} picomatch@4.0.2: {} @@ -1767,6 +2266,26 @@ snapshots: picocolors: 1.1.1 source-map-js: 1.2.1 + prosemirror-model@1.25.10: + dependencies: + orderedmap: 2.1.1 + + prosemirror-state@1.4.4: + dependencies: + prosemirror-model: 1.25.10 + prosemirror-transform: 1.12.0 + prosemirror-view: 1.42.0 + + prosemirror-transform@1.12.0: + dependencies: + prosemirror-model: 1.25.10 + + prosemirror-view@1.42.0: + dependencies: + prosemirror-model: 1.25.10 + prosemirror-state: 1.4.4 + prosemirror-transform: 1.12.0 + publint@0.3.12: dependencies: '@publint/pack': 0.1.2 @@ -1850,6 +2369,8 @@ snapshots: side-channel-map: 1.0.1 side-channel-weakmap: 1.0.2 + siginfo@2.0.0: {} + sirv@3.0.1: dependencies: '@polka/url': 1.0.0-next.29 @@ -1858,6 +2379,10 @@ snapshots: source-map-js@1.2.1: {} + stackback@0.0.2: {} + + std-env@3.10.0: {} + supports-color@7.2.0: dependencies: has-flag: 4.0.0 @@ -1902,11 +2427,21 @@ snapshots: tapable@2.2.1: {} + tinybench@2.9.0: {} + + tinyexec@0.3.2: {} + tinyglobby@0.2.13: dependencies: fdir: 6.4.4(picomatch@4.0.2) picomatch: 4.0.2 + tinypool@1.1.1: {} + + tinyrainbow@1.2.0: {} + + tinyspy@3.0.2: {} + totalist@3.0.1: {} ts-node@10.9.2(@types/node@22.15.3)(typescript@5.8.3): @@ -1943,6 +2478,34 @@ snapshots: v8-compile-cache-lib@3.0.1: {} + vite-node@2.1.9(@types/node@22.15.3)(lightningcss@1.29.2): + dependencies: + cac: 6.7.14 + debug: 4.4.0 + es-module-lexer: 1.7.0 + pathe: 1.1.2 + vite: 5.4.21(@types/node@22.15.3)(lightningcss@1.29.2) + transitivePeerDependencies: + - '@types/node' + - less + - lightningcss + - sass + - sass-embedded + - stylus + - sugarss + - supports-color + - terser + + vite@5.4.21(@types/node@22.15.3)(lightningcss@1.29.2): + dependencies: + esbuild: 0.21.5 + postcss: 8.5.3 + rollup: 4.40.1 + optionalDependencies: + '@types/node': 22.15.3 + fsevents: 2.3.3 + lightningcss: 1.29.2 + vite@6.3.3(@types/node@22.15.3)(jiti@2.4.2)(lightningcss@1.29.2): dependencies: esbuild: 0.25.3 @@ -1961,12 +2524,52 @@ snapshots: optionalDependencies: vite: 6.3.3(@types/node@22.15.3)(jiti@2.4.2)(lightningcss@1.29.2) + vitest@2.1.9(@types/node@22.15.3)(lightningcss@1.29.2): + dependencies: + '@vitest/expect': 2.1.9 + '@vitest/mocker': 2.1.9(vite@5.4.21(@types/node@22.15.3)(lightningcss@1.29.2)) + '@vitest/pretty-format': 2.1.9 + '@vitest/runner': 2.1.9 + '@vitest/snapshot': 2.1.9 + '@vitest/spy': 2.1.9 + '@vitest/utils': 2.1.9 + chai: 5.3.3 + debug: 4.4.0 + expect-type: 1.4.0 + magic-string: 0.30.17 + pathe: 1.1.2 + std-env: 3.10.0 + tinybench: 2.9.0 + tinyexec: 0.3.2 + tinypool: 1.1.1 + tinyrainbow: 1.2.0 + vite: 5.4.21(@types/node@22.15.3)(lightningcss@1.29.2) + vite-node: 2.1.9(@types/node@22.15.3)(lightningcss@1.29.2) + why-is-node-running: 2.3.0 + optionalDependencies: + '@types/node': 22.15.3 + transitivePeerDependencies: + - less + - lightningcss + - msw + - sass + - sass-embedded + - stylus + - sugarss + - supports-color + - terser + web-tree-sitter@0.25.10: {} whatwg-encoding@2.0.0: dependencies: iconv-lite: 0.6.3 + why-is-node-running@2.3.0: + dependencies: + siginfo: 2.0.0 + stackback: 0.0.2 + yn@3.1.1: {} zimmerframe@1.1.2: {} diff --git a/demo/svelte-demo/src/app.css b/demo/svelte-demo/src/app.css index cd67023..eede70c 100644 --- a/demo/svelte-demo/src/app.css +++ b/demo/svelte-demo/src/app.css @@ -1,3 +1,4 @@ @import 'tailwindcss'; +@import './lib/prosemirror/prosemirror.css'; @plugin '@tailwindcss/forms'; @plugin '@tailwindcss/typography'; diff --git a/demo/svelte-demo/src/lib/prosemirror/ProseMirrorRenderer.svelte b/demo/svelte-demo/src/lib/prosemirror/ProseMirrorRenderer.svelte new file mode 100644 index 0000000..0cd432d --- /dev/null +++ b/demo/svelte-demo/src/lib/prosemirror/ProseMirrorRenderer.svelte @@ -0,0 +1,30 @@ + + +
diff --git a/demo/svelte-demo/src/lib/prosemirror/editor.ts b/demo/svelte-demo/src/lib/prosemirror/editor.ts new file mode 100644 index 0000000..5bccee9 --- /dev/null +++ b/demo/svelte-demo/src/lib/prosemirror/editor.ts @@ -0,0 +1,51 @@ +import { EditorState } from 'prosemirror-state' +import { EditorView } from 'prosemirror-view' +import type { Chunk, StreamingChunk } from '../../../../../src/markdown-stream-parser.ts' +import { createEmptyDoc, schema } from './schema.ts' +import { applyStreamingChunkToBuffer, buildDocFromChunks } from './stream-assembly.ts' + +export type StreamRenderer = { + handleStreamingChunk(parsed: StreamingChunk): void + reset(): void + destroy(): void +} + +export function createStreamRenderer(mount: HTMLElement): StreamRenderer { + let buffer: Chunk[] = [] + const view = new EditorView(mount, { + state: EditorState.create({ schema, doc: createEmptyDoc() }), + editable: () => false, + }) + + function reset(): void { + buffer = [] + const nextDoc = createEmptyDoc() + view.updateState(EditorState.create({ schema, doc: nextDoc })) + } + + return { + handleStreamingChunk(parsed: StreamingChunk): void { + if (parsed.status === 'START_STREAM') { + reset() + return + } + if (parsed.status === 'END_STREAM') return + + const chunk = parsed.chunk + if (chunk.recovery?.type === 'window_overflow') { + console.warn('Recovery exceeded windowSize; only the bounded suffix was replaced.', chunk.recovery) + } + + buffer = applyStreamingChunkToBuffer(buffer, chunk) + const nextDoc = buildDocFromChunks(schema, buffer) + if (nextDoc.eq(view.state.doc)) return + + const tr = view.state.tr.replaceWith(0, view.state.doc.content.size, nextDoc.content) + view.dispatch(tr) + }, + reset, + destroy(): void { + view.destroy() + }, + } +} diff --git a/demo/svelte-demo/src/lib/prosemirror/prosemirror.css b/demo/svelte-demo/src/lib/prosemirror/prosemirror.css new file mode 100644 index 0000000..cd098f5 --- /dev/null +++ b/demo/svelte-demo/src/lib/prosemirror/prosemirror.css @@ -0,0 +1,72 @@ +.ProseMirror { + outline: none; + word-wrap: break-word; +} + +.ProseMirror p { + margin: 0.5rem 0; +} + +.ProseMirror blockquote { + border-left: 4px solid #60a5fa; + color: #374151; + font-style: italic; + margin: 0.75rem 0; + padding-left: 0.75rem; +} + +.ProseMirror pre { + background: #f3f4f6; + border-radius: 0.25rem; + color: #1f2937; + font-size: 0.875rem; + overflow-x: auto; + padding: 0.5rem; + position: relative; +} + +.ProseMirror pre[data-language]::after { + color: #6b7280; + content: attr(data-language); + font-family: ui-sans-serif, system-ui, sans-serif; + font-size: 0.75rem; + position: absolute; + right: 0.5rem; + top: 0.35rem; +} + +.ProseMirror table { + border-collapse: collapse; + font-size: 0.875rem; + margin: 0.75rem 0; +} + +.ProseMirror th, +.ProseMirror td { + border: 1px solid #d1d5db; + min-width: 6rem; + padding: 0.25rem 0.5rem; +} + +.ProseMirror th { + background: #f3f4f6; + font-weight: 600; +} + +.ProseMirror li[data-task] { + list-style: none; +} + +.ProseMirror li[data-task]::before { + display: inline-block; + margin-left: -1.25rem; + width: 1.25rem; +} + +.ProseMirror li[data-task="unchecked"]::before { + content: "☐"; +} + +.ProseMirror li[data-task="checked"]::before { + content: "☑"; +} diff --git a/demo/svelte-demo/src/lib/prosemirror/schema.ts b/demo/svelte-demo/src/lib/prosemirror/schema.ts new file mode 100644 index 0000000..7aacb57 --- /dev/null +++ b/demo/svelte-demo/src/lib/prosemirror/schema.ts @@ -0,0 +1,163 @@ +import { Schema, type DOMOutputSpec, type MarkSpec, type NodeSpec } from 'prosemirror-model' + +const paragraphDOM: DOMOutputSpec = ['p', 0] +const blockquoteDOM: DOMOutputSpec = ['blockquote', 0] +const bulletListDOM: DOMOutputSpec = ['ul', 0] +const tableDOM: DOMOutputSpec = ['table', ['tbody', 0]] +const tableRowDOM: DOMOutputSpec = ['tr', 0] +const emDOM: DOMOutputSpec = ['em', 0] +const strongDOM: DOMOutputSpec = ['strong', 0] +const codeDOM: DOMOutputSpec = ['code', 0] +const strikethroughDOM: DOMOutputSpec = ['s', 0] + +export const nodes = { + doc: { + content: 'block+', + } as NodeSpec, + + paragraph: { + content: 'inline*', + group: 'block', + toDOM() { return paragraphDOM }, + } as NodeSpec, + + heading: { + attrs: { level: { default: 1 } }, + content: 'inline*', + group: 'block', + defining: true, + toDOM(node) { + const level = Math.min(6, Math.max(1, Number(node.attrs.level) || 1)) + return [`h${level}`, 0] + }, + } as NodeSpec, + + code_block: { + attrs: { language: { default: '' } }, + content: 'text*', + marks: '', + group: 'block', + code: true, + defining: true, + toDOM(node) { + const attrs: Record = {} + if (node.attrs.language) attrs['data-language'] = node.attrs.language + return ['pre', attrs, ['code', 0]] + }, + } as NodeSpec, + + blockquote: { + content: 'block+', + group: 'block', + defining: true, + toDOM() { return blockquoteDOM }, + } as NodeSpec, + + bullet_list: { + content: 'list_item+', + group: 'block', + toDOM() { return bulletListDOM }, + } as NodeSpec, + + ordered_list: { + attrs: { order: { default: 1 } }, + content: 'list_item+', + group: 'block', + toDOM(node) { + const order = Number(node.attrs.order) || 1 + return order === 1 ? ['ol', 0] : ['ol', { start: order }, 0] + }, + } as NodeSpec, + + list_item: { + attrs: { task: { default: null } }, + content: 'block+', + defining: true, + toDOM(node) { + const task = node.attrs.task as { checked: boolean } | null + if (!task) return ['li', 0] + return ['li', { 'data-task': task.checked ? 'checked' : 'unchecked' }, 0] + }, + } as NodeSpec, + + table: { + content: 'table_row+', + group: 'block', + toDOM() { return tableDOM }, + } as NodeSpec, + + table_row: { + content: '(table_header_cell|table_cell)+', + toDOM() { return tableRowDOM }, + } as NodeSpec, + + table_header_cell: { + attrs: { align: { default: null } }, + content: 'inline*', + toDOM(node) { return createTableCellDOM('th', node.attrs.align) }, + } as NodeSpec, + + table_cell: { + attrs: { align: { default: null } }, + content: 'inline*', + toDOM(node) { return createTableCellDOM('td', node.attrs.align) }, + } as NodeSpec, + + image: { + inline: true, + group: 'inline', + attrs: { + src: {}, + alt: { default: null }, + }, + draggable: false, + toDOM(node) { + const attrs: Record = { src: node.attrs.src } + if (node.attrs.alt) attrs.alt = node.attrs.alt + return ['img', attrs] + }, + } as NodeSpec, + + text: { + group: 'inline', + } as NodeSpec, +} + +export const marks = { + link: { + attrs: { href: {} }, + inclusive: false, + toDOM(node) { + return ['a', { href: node.attrs.href, rel: 'noopener noreferrer' }, 0] + }, + } as MarkSpec, + + em: { + toDOM() { return emDOM }, + } as MarkSpec, + + strong: { + toDOM() { return strongDOM }, + } as MarkSpec, + + code: { + toDOM() { return codeDOM }, + } as MarkSpec, + + strikethrough: { + toDOM() { return strikethroughDOM }, + } as MarkSpec, +} + +function createTableCellDOM(tag: 'th' | 'td', align: unknown): DOMOutputSpec { + if (align === 'left' || align === 'center' || align === 'right') { + return [tag, { style: `text-align: ${align}` }, 0] + } + return [tag, 0] +} + +export const schema = new Schema({ nodes, marks }) + +export function createEmptyDoc() { + return schema.nodes.doc.create(null, schema.nodes.paragraph.create()) +} diff --git a/demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts new file mode 100644 index 0000000..99f953b --- /dev/null +++ b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts @@ -0,0 +1,158 @@ +import { describe, expect, it } from 'vitest' +import type { Chunk } from '../../../../../src/markdown-stream-parser.ts' +import { schema } from './schema.ts' +import { + applyStreamingChunkToBuffer, + buildDocFromChunks, + buildInlineContent, + groupChunksIntoBlocks, + isChunkBeforeBacktrack, + sanitizeImageSrc, + sanitizeLinkHref, +} from './stream-assembly.ts' + +function chunk(overrides: Partial & Pick): Chunk { + return { + text: overrides.text, + offset: overrides.offset, + length: overrides.length ?? overrides.text.length, + block: overrides.block ?? { type: 'paragraph' }, + opening: overrides.opening ?? [], + closing: overrides.closing ?? [], + contained: overrides.contained ?? [], + backtrackOffset: overrides.backtrackOffset, + recovery: overrides.recovery, + original: overrides.original, + } +} + +describe('stream assembly helpers', () => { + it('shares the legacy backtrack predicate', () => { + const first = chunk({ text: 'hello', offset: 0 }) + const stale = chunk({ text: ' world', offset: 5 }) + const replacement = chunk({ text: ' there', offset: 5, backtrackOffset: 5 }) + + expect(isChunkBeforeBacktrack(first, 5)).toBe(true) + expect(isChunkBeforeBacktrack(stale, 5)).toBe(false) + expect(applyStreamingChunkToBuffer([first, stale], replacement)).toEqual([first, replacement]) + }) + + it('sanitizes link and image urls', () => { + expect(sanitizeLinkHref('https://example.com')).toBe('https://example.com') + expect(sanitizeLinkHref('mailto:test@example.com')).toBe('mailto:test@example.com') + expect(sanitizeLinkHref('javascript:alert(1)')).toBeNull() + expect(sanitizeImageSrc('/asset.png')).toBe('/asset.png') + expect(sanitizeImageSrc('data:image/png;base64,AAAA')).toBe('data:image/png;base64,AAAA') + expect(sanitizeImageSrc('//example.com/image.png')).toBeNull() + expect(sanitizeImageSrc('data:text/html;base64,AAAA')).toBeNull() + }) + + it('builds marks and rejects unsafe links', () => { + const safe = chunk({ + text: 'safe bad', + offset: 0, + contained: [ + { type: 'link', offset: 0, length: 4, url: 'https://example.com' }, + { type: 'link', offset: 5, length: 3, url: 'javascript:alert(1)' }, + ], + }) + + const nodes = buildInlineContent(schema, [safe]) + expect(nodes.map(node => node.text).join('')).toBe('safe bad') + expect(nodes[0].marks[0]?.type.name).toBe('link') + expect(nodes[nodes.length - 1].marks).toHaveLength(0) + }) + + it('styles a span that opens in one chunk and closes in a later chunk', () => { + const nodes = buildInlineContent(schema, [ + chunk({ + text: 'bo', + offset: 0, + opening: [{ type: 'bold', openOffset: 0 }], + }), + chunk({ + text: 'ld', + offset: 2, + }), + chunk({ + text: ' text', + offset: 4, + closing: [{ type: 'bold', offset: 0, length: 4 }], + }), + ]) + + const textNodes = nodes.filter(node => node.isText) + expect(textNodes.map(node => node.text).join('')).toBe('bold text') + expect(textNodes.slice(0, 2).every(node => node.marks.some(mark => mark.type.name === 'strong'))).toBe(true) + expect(textNodes[textNodes.length - 1].marks.some(mark => mark.type.name === 'strong')).toBe(false) + }) + + it('styles an unclosed span through the end of the current buffer', () => { + const nodes = buildInlineContent(schema, [ + chunk({ + text: 'plain ', + offset: 0, + }), + chunk({ + text: 'bold', + offset: 6, + opening: [{ type: 'bold', openOffset: 6 }], + }), + ]) + + expect(nodes.map(node => node.text).join('')).toBe('plain bold') + expect(nodes[0].marks.some(mark => mark.type.name === 'strong')).toBe(false) + expect(nodes[nodes.length - 1].marks.some(mark => mark.type.name === 'strong')).toBe(true) + }) + + it('creates image nodes only for safe image sources', () => { + const nodes = buildInlineContent(schema, [ + chunk({ + text: 'logo evil', + offset: 0, + contained: [ + { type: 'image', offset: 0, length: 4, src: '/logo.png', alt: 'Logo' }, + { type: 'image', offset: 5, length: 4, src: 'javascript:alert(1)', alt: 'Bad' }, + ], + }), + ]) + + expect(nodes[0].type.name).toBe('image') + expect(nodes[0].attrs.src).toBe('/logo.png') + expect(nodes[nodes.length - 1].text).toBe('evil') + }) + + it('groups list items by depth and type boundaries', () => { + const blocks = groupChunksIntoBlocks([ + chunk({ text: 'one\n', offset: 0, block: { type: 'list_item', list: { type: 'unordered', depth: 0, marker: '-' } } }), + chunk({ text: 'two', offset: 4, block: { type: 'list_item', list: { type: 'unordered', depth: 0, marker: '-' } } }), + chunk({ text: 'nested', offset: 7, block: { type: 'list_item', list: { type: 'unordered', depth: 1, marker: '-' } } }), + ]) + + expect(blocks).toHaveLength(3) + }) + + it('builds lists, task items, tables, and code blocks', () => { + const doc = buildDocFromChunks(schema, [ + chunk({ text: 'Task', offset: 0, block: { type: 'list_item', list: { type: 'unordered', depth: 0, marker: '-', task: { checked: true } } } }), + chunk({ text: 'Head', offset: 4, block: { type: 'table_header_cell', table: { tableId: 't1', rowIndex: 0, columnIndex: 0, cellId: 't1:0:0' } } }), + chunk({ text: 'Cell', offset: 8, block: { type: 'table_cell', table: { tableId: 't1', rowIndex: 1, columnIndex: 0, cellId: 't1:1:0' } } }), + chunk({ text: 'const x = 1\n', offset: 12, block: { type: 'code_block', language: 'ts' } }), + ]) + + expect(doc.child(0).type.name).toBe('bullet_list') + expect(doc.child(0).child(0).attrs.task).toEqual({ checked: true }) + expect(doc.child(1).type.name).toBe('table') + expect(doc.child(2).type.name).toBe('code_block') + expect(doc.child(2).attrs.language).toBe('ts') + }) + + it('falls back instead of throwing for malformed block states', () => { + const doc = buildDocFromChunks(schema, [ + chunk({ text: '', offset: 0, block: { type: 'table_cell' } }), + ]) + + expect(doc.type.name).toBe('doc') + expect(doc.childCount).toBeGreaterThan(0) + }) +}) diff --git a/demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts new file mode 100644 index 0000000..7cb8532 --- /dev/null +++ b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts @@ -0,0 +1,465 @@ +import { Fragment, type Mark, type Node as ProseMirrorNode, type Schema } from 'prosemirror-model' +import type { Chunk, ClosedSpan, OpenSpan, Span, SpanType } from '../../../../../src/markdown-stream-parser.ts' + +type BlockType = Chunk['block']['type'] +type ListMetadata = NonNullable +type ListFrame = { + type: ListMetadata['type'] + depth: number + order: number + items: ProseMirrorNode[] +} +type CellGroup = { + cellId: string + columnIndex: number + type: 'table_header_cell' | 'table_cell' + align?: 'left' | 'center' | 'right' + chunks: Chunk[] +} +type MarkRange = { + type: SpanType + start: number + end: number +} + +export function isChunkBeforeBacktrack(chunk: Chunk, backtrackOffset: number): boolean { + return chunk.offset + chunk.length <= backtrackOffset +} + +export function applyStreamingChunkToBuffer(buffer: Chunk[], chunk: Chunk): Chunk[] { + const next = chunk.backtrackOffset === undefined + ? buffer + : buffer.filter(bufferedChunk => isChunkBeforeBacktrack(bufferedChunk, chunk.backtrackOffset!)) + return [...next, chunk] +} + +export function sanitizeLinkHref(rawHref: string): string | null { + const href = rawHref.trim() + if (!href) return null + + try { + const url = new URL(href) + return url.protocol === 'http:' || url.protocol === 'https:' || url.protocol === 'mailto:' + ? href + : null + } catch { + return null + } +} + +export function sanitizeImageSrc(rawSrc: string): string | null { + const src = rawSrc.trim() + if (!src || src.startsWith('//')) return null + if (src.startsWith('/') || src.startsWith('./') || src.startsWith('../')) return src + + try { + const url = new URL(src) + if (url.protocol === 'http:' || url.protocol === 'https:') return src + if (url.protocol === 'data:' && /^data:image\/[a-z0-9.+-]+;base64,/i.test(src)) return src + return null + } catch { + return null + } +} + +export function groupChunksIntoBlocks(chunks: Chunk[]): Chunk[][] { + const blocks: Chunk[][] = [] + let currentBlock: Chunk[] = [] + let lastBlockType: BlockType | undefined + let lastBlockLevel: number | undefined + let lastTableId: string | undefined + let lastListDepth: number | undefined + let lastListType: ListMetadata['type'] | undefined + let lastOffset = -1 + + for (const chunk of chunks) { + const blockType = chunk.block.type + const blockLevel = chunk.block.level + const tableId = chunk.block.table?.tableId + const listDepth = chunk.block.list?.depth + const listType = chunk.block.list?.type + let isNewBlock = false + + if ( + blockType !== lastBlockType + && !(isTableCellBlockType(blockType) && isTableCellBlockType(lastBlockType) && tableId === lastTableId) + ) { + isNewBlock = true + } else if (tableId !== lastTableId) { + isNewBlock = true + } else if (blockType === 'heading' && blockLevel !== lastBlockLevel) { + isNewBlock = true + } else if (blockType === 'list_item' && (listDepth !== lastListDepth || listType !== lastListType)) { + isNewBlock = true + } else if (blockType === 'list_item' && lastOffset >= 0) { + const previousChunk = currentBlock[currentBlock.length - 1] + if (previousChunk?.text.endsWith('\n') && chunk.text.trim().length > 0) { + isNewBlock = true + } + } + + if (isNewBlock && currentBlock.length > 0) { + blocks.push(currentBlock) + currentBlock = [] + } + + currentBlock.push(chunk) + lastBlockType = blockType + lastBlockLevel = blockLevel + lastTableId = tableId + lastListDepth = listDepth + lastListType = listType + lastOffset = chunk.offset + chunk.length + } + + if (currentBlock.length > 0) blocks.push(currentBlock) + return blocks +} + +export function buildInlineContent(schema: Schema, blockChunks: Chunk[]): ProseMirrorNode[] { + const nodes: ProseMirrorNode[] = [] + const chunks = trimTrailingNewline(blockChunks) + const closedSpans = chunks.flatMap(chunk => [...chunk.contained, ...chunk.closing]) + const imageSpans = dedupeClosedSpans(closedSpans.filter((span): span is ClosedSpan & { type: 'image'; src: string; alt?: string } => span.type === 'image')) + const markRanges = buildMarkRanges(chunks, closedSpans) + const textRuns = buildTextRuns(chunks, closedSpans, markRanges, imageSpans) + + for (const run of textRuns) { + if (run.image) { + const src = sanitizeImageSrc(run.image.src) + if (src) { + nodes.push(schema.nodes.image.create({ src, alt: run.image.alt ?? null })) + continue + } + } + + if (!run.text) continue + const marks = createMarksForRange(schema, closedSpans, markRanges, run.start, run.end) + nodes.push(marks.length > 0 ? schema.text(run.text, marks) : schema.text(run.text)) + } + + return nodes +} + +export function buildContentFromChunks(schema: Schema, chunks: Chunk[]): Fragment { + const blocks = groupChunksIntoBlocks(chunks) + const nodes: ProseMirrorNode[] = [] + let listFrames: ListFrame[] = [] + + function flushLists(toDepth = -1): void { + while (listFrames.length > 0 && listFrames[listFrames.length - 1].depth > toDepth) { + const frame = listFrames.pop()! + const listNode = createListNode(schema, frame) + if (listFrames.length > 0) { + const parent = listFrames[listFrames.length - 1] + const lastItem = parent.items.pop() + if (lastItem) { + parent.items.push(appendBlockToListItem(schema, lastItem, listNode)) + } else { + parent.items.push(schema.nodes.list_item.create(null, listNode)) + } + } else { + nodes.push(listNode) + } + } + } + + for (let index = 0; index < blocks.length; index++) { + const block = blocks[index] + const firstChunk = block[0] + const nextBlock = blocks[index + 1] + + try { + if (firstChunk?.block.type === 'list_item' && firstChunk.block.list) { + const list = firstChunk.block.list + flushLists(list.depth - 1) + let frame = listFrames[listFrames.length - 1] + if (!frame || frame.depth !== list.depth || frame.type !== list.type) { + frame = { + type: list.type, + depth: list.depth, + order: list.ordinal ?? 1, + items: [], + } + listFrames.push(frame) + } + frame.items.push(createListItemNode(schema, block, list)) + if (nextBlock?.[0]?.block.list === undefined) flushLists() + continue + } + + flushLists() + + if (isTableCellBlockType(firstChunk?.block.type)) { + const tableBlocks = [block] + const tableId = firstChunk?.block.table?.tableId + while (blocks[index + 1]?.[0]?.block.table?.tableId === tableId) { + tableBlocks.push(blocks[++index]) + } + nodes.push(createTableNode(schema, tableBlocks.flat())) + continue + } + + nodes.push(createBlockNode(schema, block)) + } catch { + flushLists() + nodes.push(schema.nodes.paragraph.create(null, createPlainTextContent(schema, block))) + } + } + + flushLists() + return nodes.length > 0 ? Fragment.fromArray(nodes) : Fragment.from(schema.nodes.paragraph.create()) +} + +export function buildDocFromChunks(schema: Schema, chunks: Chunk[]): ProseMirrorNode { + return schema.nodes.doc.create(null, buildContentFromChunks(schema, chunks)) +} + +function createBlockNode(schema: Schema, block: Chunk[]): ProseMirrorNode { + const first = block[0] + const inlineContent = buildInlineContent(schema, block) + + switch (first?.block.type) { + case 'heading': + return schema.nodes.heading.create({ level: first.block.level ?? 1 }, inlineContent) + case 'code_block': + return schema.nodes.code_block.create( + { language: first.block.language ?? '' }, + createTextNodeOrNull(schema, trimTrailingNewline(block).map(chunk => chunk.text).join('')), + ) + case 'blockquote': + return schema.nodes.blockquote.create(null, schema.nodes.paragraph.create(null, inlineContent)) + case 'paragraph': + case undefined: + default: + return schema.nodes.paragraph.create(null, inlineContent) + } +} + +function createListItemNode(schema: Schema, block: Chunk[], list: ListMetadata): ProseMirrorNode { + const paragraph = schema.nodes.paragraph.create(null, buildInlineContent(schema, block)) + return schema.nodes.list_item.create({ task: list.task ?? null }, paragraph) +} + +function createListNode(schema: Schema, frame: ListFrame): ProseMirrorNode { + const type = frame.type === 'ordered' ? schema.nodes.ordered_list : schema.nodes.bullet_list + return frame.type === 'ordered' + ? type.create({ order: frame.order }, frame.items) + : type.create(null, frame.items) +} + +function appendBlockToListItem(schema: Schema, item: ProseMirrorNode, block: ProseMirrorNode): ProseMirrorNode { + return schema.nodes.list_item.create(item.attrs, item.content.append(Fragment.from(block))) +} + +function createTableNode(schema: Schema, chunks: Chunk[]): ProseMirrorNode { + const rows = buildTableRows(chunks) + const rowNodes = rows.map(row => { + const cellNodes = row.cells.map(cell => { + const cellType = cell.type === 'table_header_cell' + ? schema.nodes.table_header_cell + : schema.nodes.table_cell + return cellType.create({ align: cell.align ?? null }, buildInlineContent(schema, cell.chunks)) + }) + return schema.nodes.table_row.create(null, cellNodes) + }) + return schema.nodes.table.create(null, rowNodes) +} + +function buildTableRows(chunks: Chunk[]): Array<{ rowIndex: number; cells: CellGroup[] }> { + const rows = new Map>() + + for (const chunk of chunks) { + const table = chunk.block.table + if (!table) continue + + let row = rows.get(table.rowIndex) + if (!row) { + row = new Map() + rows.set(table.rowIndex, row) + } + + let cell = row.get(table.cellId) + if (!cell) { + cell = { + cellId: table.cellId, + columnIndex: table.columnIndex, + type: chunk.block.type === 'table_header_cell' ? 'table_header_cell' : 'table_cell', + align: table.align, + chunks: [], + } + row.set(table.cellId, cell) + } + + cell.chunks.push(chunk) + } + + return [...rows.entries()] + .sort((a, b) => a[0] - b[0]) + .map(([rowIndex, cells]) => ({ + rowIndex, + cells: [...cells.values()].sort((a, b) => a.columnIndex - b.columnIndex), + })) +} + +function buildTextRuns( + chunks: Chunk[], + spans: ClosedSpan[], + markRanges: MarkRange[], + imageSpans: Array, +): Array<{ start: number; end: number; text: string; image?: Span & { type: 'image' } }> { + const runs: Array<{ start: number; end: number; text: string; image?: Span & { type: 'image' } }> = [] + + for (const chunk of chunks) { + const chunkStart = chunk.offset + const chunkEnd = chunk.offset + chunk.length + const boundaries = new Set([chunkStart, chunkEnd]) + + for (const span of spans) { + const spanStart = span.offset + const spanEnd = span.offset + span.length + if (spanStart > chunkStart && spanStart < chunkEnd) boundaries.add(spanStart) + if (spanEnd > chunkStart && spanEnd < chunkEnd) boundaries.add(spanEnd) + } + for (const range of markRanges) { + if (range.start > chunkStart && range.start < chunkEnd) boundaries.add(range.start) + if (range.end > chunkStart && range.end < chunkEnd) boundaries.add(range.end) + } + + const sortedBoundaries = [...boundaries].sort((a, b) => a - b) + for (let index = 0; index < sortedBoundaries.length - 1; index++) { + const start = sortedBoundaries[index] + const end = sortedBoundaries[index + 1] + if (start === end) continue + const image = imageSpans.find(span => span.offset === start && span.offset + span.length === end) + runs.push({ + start, + end, + text: chunk.text.slice(start - chunk.offset, end - chunk.offset), + image, + }) + } + } + + return runs +} + +function buildMarkRanges(chunks: Chunk[], spans: ClosedSpan[]): MarkRange[] { + const ranges: MarkRange[] = [] + const openSpans: OpenSpan[] = [] + const blockEnd = chunks.reduce((max, chunk) => Math.max(max, chunk.offset + chunk.length), 0) + + for (const span of spans) { + if (span.type === 'link' || span.type === 'image') continue + ranges.push({ type: span.type, start: span.offset, end: span.offset + span.length }) + } + + for (const chunk of chunks) { + for (const span of chunk.opening) { + if (span.type === 'link' || span.type === 'image') continue + openSpans.push(span) + } + + for (const span of chunk.closing) { + if (span.type === 'link' || span.type === 'image') continue + const openIndex = openSpans.findIndex(openSpan => + openSpan.type === span.type && openSpan.openOffset === span.offset + ) + if (openIndex === -1) continue + + ranges.push({ + type: span.type, + start: openSpans[openIndex].openOffset, + end: span.offset + span.length, + }) + openSpans.splice(openIndex, 1) + } + } + + for (const span of openSpans) { + if (span.openOffset < blockEnd) { + ranges.push({ type: span.type, start: span.openOffset, end: blockEnd }) + } + } + + return ranges +} + +function createMarksForRange(schema: Schema, spans: ClosedSpan[], markRanges: MarkRange[], start: number, end: number): Mark[] { + const marks: Mark[] = [] + const activeTypes = new Set() + + for (const span of dedupeClosedSpans(spans)) { + if (span.type === 'image') continue + if (span.offset >= end || span.offset + span.length <= start) continue + + if (span.type === 'link') { + const href = sanitizeLinkHref(span.url) + if (href) marks.push(schema.marks.link.create({ href })) + } + } + + for (const range of markRanges) { + if (range.type === 'link' || range.type === 'image') continue + if (range.start >= end || range.end <= start) continue + + if (activeTypes.has(range.type)) continue + activeTypes.add(range.type) + const mark = createMarkForSpanType(schema, range.type) + if (mark) marks.push(mark) + } + + return marks +} + +function createMarkForSpanType(schema: Schema, type: SpanType): Mark | null { + switch (type) { + case 'bold': + return schema.marks.strong.create() + case 'italic': + return schema.marks.em.create() + case 'code': + return schema.marks.code.create() + case 'strikethrough': + return schema.marks.strikethrough.create() + default: + return null + } +} + +function dedupeClosedSpans(spans: T[]): T[] { + const seen = new Set() + return spans.filter(span => { + const key = `${span.type}:${span.offset}:${span.length}:${'url' in span ? span.url : ''}:${'src' in span ? span.src : ''}` + if (seen.has(key)) return false + seen.add(key) + return true + }) +} + +function trimTrailingNewline(chunks: Chunk[]): Chunk[] { + if (chunks.length === 0) return chunks + const trimmed = [...chunks] + const last = trimmed[trimmed.length - 1] + if (!last.text.endsWith('\n')) return trimmed + + trimmed[trimmed.length - 1] = { + ...last, + text: last.text.slice(0, -1), + length: Math.max(0, last.length - 1), + } + return trimmed +} + +function createTextNodeOrNull(schema: Schema, text: string): ProseMirrorNode | null { + return text ? schema.text(text) : null +} + +function createPlainTextContent(schema: Schema, block: Chunk[]): ProseMirrorNode[] { + const text = trimTrailingNewline(block).map(chunk => chunk.text).join('') + return text ? [schema.text(text)] : [] +} + +function isTableCellBlockType(blockType: string | undefined): boolean { + return blockType === 'table_header_cell' || blockType === 'table_cell' +} diff --git a/demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts b/demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts new file mode 100644 index 0000000..4ed0819 --- /dev/null +++ b/demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts @@ -0,0 +1,143 @@ +import { readFileSync } from 'node:fs' +import { dirname, join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' +import { MarkdownStreamParser, type Chunk, type StreamingChunk } from '../../../../../src/markdown-stream-parser.ts' +import { schema } from './schema.ts' +import { applyStreamingChunkToBuffer, buildDocFromChunks } from './stream-assembly.ts' + +const __filename = fileURLToPath(import.meta.url) +const __dirname = dirname(__filename) +const repoRoot = join(__dirname, '../../../../..') +const examplesDir = join(repoRoot, 'demo/svelte-demo/static/llm-streams-examples') +const wasmDir = join(repoRoot, 'demo/svelte-demo/static') + +type ParsedExample = { + chunks: Chunk[] + activeChunks: Chunk[] + doc: ReturnType +} + +function readExampleTokens(name: string): string[] { + return JSON.parse(readFileSync(join(examplesDir, `${name}.json`), 'utf8')) as string[] +} + +async function parseExample(name: string, limit = Number.POSITIVE_INFINITY): Promise { + MarkdownStreamParser.configureWasmPath(join(wasmDir, 'tree-sitter-markdown.wasm')) + const parser = await MarkdownStreamParser.getInstance(`pm-integration-${name}-${limit}`) + const chunks: Chunk[] = [] + let activeChunks: Chunk[] = [] + + parser.subscribeToTokenParse((parsed: StreamingChunk) => { + if (parsed.status !== 'STREAMING') return + chunks.push(parsed.chunk) + activeChunks = applyStreamingChunkToBuffer(activeChunks, parsed.chunk) + }) + + parser.startParsing() + for (const token of readExampleTokens(name).slice(0, limit)) { + parser.parseToken(token) + } + parser.stopParsing() + MarkdownStreamParser.removeInstance(`pm-integration-${name}-${limit}`) + + return { + chunks, + activeChunks, + doc: buildDocFromChunks(schema, activeChunks), + } +} + +function hasNode(doc: ParsedExample['doc'], type: string): boolean { + let found = false + doc.descendants(node => { + if (node.type.name === type) found = true + }) + return found +} + +function hasMark(doc: ParsedExample['doc'], type: string): boolean { + let found = false + doc.descendants(node => { + if (node.marks.some(mark => mark.type.name === type)) found = true + }) + return found +} + +function taskAttrs(doc: ParsedExample['doc']): unknown[] { + const attrs: unknown[] = [] + doc.descendants(node => { + if (node.type.name === 'list_item' && node.attrs.task) attrs.push(node.attrs.task) + }) + return attrs +} + +function compactText(text: string): string { + return text.replace(/\s+/g, '') +} + +describe('real stream examples to ProseMirror documents', () => { + it('renders heading, list, emphasis, and code examples into document structure', async () => { + const quantum = await parseExample('claude-3.5-1-quantum-physics') + const history = await parseExample('gpt-4.o-history-of-cats') + const code = await parseExample('claude-3.7-happy-number-5-programs') + + expect(hasNode(quantum.doc, 'heading')).toBe(true) + expect(hasNode(history.doc, 'bullet_list') || hasNode(history.doc, 'ordered_list')).toBe(true) + expect(hasMark(history.doc, 'strong') || hasMark(history.doc, 'em')).toBe(true) + expect(hasNode(code.doc, 'code_block')).toBe(true) + }) + + it('self-corrects backtracking streams without stale active text', async () => { + const nestedCode = await parseExample('claude-3.7-markdown-with-nested-code-block') + const errorRecovery = await parseExample('test-error-recovery') + + expect(nestedCode.chunks.some(chunk => chunk.backtrackOffset !== undefined)).toBe(true) + expect(errorRecovery.chunks.some(chunk => chunk.backtrackOffset !== undefined)).toBe(true) + expect(compactText(nestedCode.doc.textContent)).toBe(compactText(nestedCode.activeChunks.map(chunk => chunk.text).join(''))) + expect(compactText(errorRecovery.doc.textContent)).toBe(compactText(errorRecovery.activeChunks.map(chunk => chunk.text).join(''))) + expect(hasNode(errorRecovery.doc, 'table')).toBe(true) + expect(hasNode(errorRecovery.doc, 'code_block')).toBe(true) + }) + + it('renders strikethrough, tables, and task lists', async () => { + const strikethrough = await parseExample('test-strikethrough') + const table = await parseExample('test-error-recovery') + const taskDoc = buildDocFromChunks(schema, [ + { + text: 'Completed\n', + offset: 0, + length: 10, + block: { type: 'list_item', list: { type: 'unordered', depth: 0, marker: '-', task: { checked: true } } }, + opening: [], + closing: [], + contained: [], + }, + { + text: 'Incomplete', + offset: 10, + length: 10, + block: { type: 'list_item', list: { type: 'unordered', depth: 0, marker: '-', task: { checked: false } } }, + opening: [], + closing: [], + contained: [], + }, + ]) + + expect(hasMark(strikethrough.doc, 'strikethrough')).toBe(true) + expect(hasNode(table.doc, 'table')).toBe(true) + expect(taskAttrs(taskDoc)).toEqual([{ checked: true }, { checked: false }]) + }) + + it('supports reset, replay after completion, and switching examples at the buffer level', async () => { + const partial = await parseExample('gpt-4.5-cat-coding', 5) + const replayA = await parseExample('test-strikethrough') + const replayB = await parseExample('test-strikethrough') + const switched = await parseExample('test-error-recovery', 3) + + expect(partial.activeChunks.length).toBeGreaterThan(0) + expect(buildDocFromChunks(schema, []).textContent).toBe('') + expect(replayA.doc.eq(replayB.doc)).toBe(true) + expect(switched.doc.textContent).not.toBe(partial.doc.textContent) + }) +}) diff --git a/demo/svelte-demo/src/routes/+page.svelte b/demo/svelte-demo/src/routes/+page.svelte index e7cfa81..ae70bb3 100644 --- a/demo/svelte-demo/src/routes/+page.svelte +++ b/demo/svelte-demo/src/routes/+page.svelte @@ -1,27 +1,13 @@
@@ -598,233 +384,8 @@ class="md:col-span-2 bg-white rounded shadow p-4 min-h-[400px] flex flex-col" >

Parsed Stream

-
- {#each parsedBlocks as block} - {@const blockType = block[0]?.block.type} - {@const blockLevel = block[0]?.block.level} - {@const blockLanguage = block[0]?.block.language} - {@const hasTableCells = - blockType === "table_header_cell" || blockType === "table_cell"} - {@const tableRows = hasTableCells ? buildTableRows(block) : []} - -
- {#if blockType === "heading"} - {#if blockLevel === 1} -

- {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} -

- {:else if blockLevel === 2} -

- {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} -

- {:else if blockLevel === 3} -

- {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} -

- {:else if blockLevel === 4} -

- {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} -

- {:else if blockLevel === 5} -
- {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} -
- {:else if blockLevel === 6} -
- {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} -
- {:else} - - {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} - - {/if} - {:else if blockType === "code_block"} - 
{#each block as chunk}{chunk.text}{/each}
- {#if blockLanguage} - {blockLanguage} - {/if} - {:else if blockType === "blockquote"} - - {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} - - {:else if blockType === "list_item"} - {@const list = block[0]?.block.list} - - - {#if list?.type === "ordered"} - {list.ordinal ?? ""}{list.marker} - {:else} - • - {/if} - - {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} - - {:else if blockType === "table_header_cell" || blockType === "table_cell"} -
- - {#each tableRows as row} - - {#each row.cells as cell} - - {#each cell.chunks as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} - - {/each} - - {/each} -
-
- {:else} - - - {#each block as chunk} - {@const styles = [ - ...chunk.contained.map((s) => s.type), - ...chunk.opening.map((s) => s.type), - ]} - {#if hasCodeStyle(styles)} - {chunk.text} - {:else} - {chunk.text} - {/if} - {/each} - - {/if} -
- {/each} +
+
diff --git a/demo/svelte-demo/tsconfig.json b/demo/svelte-demo/tsconfig.json index 6f788f1..5622bac 100644 --- a/demo/svelte-demo/tsconfig.json +++ b/demo/svelte-demo/tsconfig.json @@ -5,6 +5,7 @@ "checkJs": true, "esModuleInterop": true, "forceConsistentCasingInFileNames": true, + "allowImportingTsExtensions": true, "resolveJsonModule": true, "skipLibCheck": true, "sourceMap": true, diff --git a/demo/svelte-demo/vitest.config.ts b/demo/svelte-demo/vitest.config.ts new file mode 100644 index 0000000..1dace44 --- /dev/null +++ b/demo/svelte-demo/vitest.config.ts @@ -0,0 +1,9 @@ +import { defineConfig } from 'vitest/config' + +export default defineConfig({ + test: { + environment: 'node', + include: ['src/**/*.test.ts'], + exclude: ['.svelte-kit/**', 'dist/**', 'node_modules/**'], + }, +}) diff --git a/src/tree-sitter-markdown-stream-parser.ts b/src/tree-sitter-markdown-stream-parser.ts index f440876..0867fdc 100644 --- a/src/tree-sitter-markdown-stream-parser.ts +++ b/src/tree-sitter-markdown-stream-parser.ts @@ -1,4 +1,4 @@ -import { Parser, Language, type Tree, type Node } from 'web-tree-sitter' +import { Parser, Language, type Tree, type Node, type Edit } from 'web-tree-sitter' import TokensStreamBuffer from './tokens-stream-buffer.ts' import type { Chunk, @@ -353,14 +353,14 @@ export class MarkdownStreamParser { // For proper incremental parsing, tell tree-sitter what changed if (this.currentTree) { - this.currentTree.edit({ + this.currentTree.edit(({ startIndex: oldLength, oldEndIndex: oldLength, newEndIndex: this.content.length, startPosition: oldEndPosition, oldEndPosition, newEndPosition: this.endPosition - }) + }) as Edit) } // Parse the updated content diff --git a/src/tree-sitter/inline-detection.ts b/src/tree-sitter/inline-detection.ts index 1723e40..3996a6a 100644 --- a/src/tree-sitter/inline-detection.ts +++ b/src/tree-sitter/inline-detection.ts @@ -263,6 +263,9 @@ export function isInsideCodeBlock( if (inlineNode) { const inlineContent = inlineNode.text const inlineTree = inlineParser.parse(inlineContent) + if (!inlineTree) { + return false + } const relativePos = position - inlineNode.startIndex // Check if position is inside any code_span diff --git a/src/tree-sitter/segment-generator.ts b/src/tree-sitter/segment-generator.ts index caf88d8..ae286ef 100644 --- a/src/tree-sitter/segment-generator.ts +++ b/src/tree-sitter/segment-generator.ts @@ -379,79 +379,80 @@ export function generateSegments( if (inlineNode && inlineParser) { const inlineContent = inlineNode.text const inlineTree = inlineParser.parse(inlineContent) + if (inlineTree) { + // Count the range in the new portion + const newPortionStart = actualFromIndex - inlineNode.startIndex + const newPortionEnd = actualToIndex - inlineNode.startIndex + const newPortion = inlineContent.substring(Math.max(0, newPortionStart), newPortionEnd) + + // Check for unmatched backtick + if (newPortion.includes('`')) { + const hasCompleteCodeSpan = hasCompleteCodeSpanAt(inlineTree.rootNode, newPortionStart, newPortionEnd) + if (!hasCompleteCodeSpan) { + state.pendingInlineContent = newContent + state.pendingInlineStartIndex = actualFromIndex + state.sourceOffset = actualToIndex + return { segments, state } + } + } - // Count the range in the new portion - const newPortionStart = actualFromIndex - inlineNode.startIndex - const newPortionEnd = actualToIndex - inlineNode.startIndex - const newPortion = inlineContent.substring(Math.max(0, newPortionStart), newPortionEnd) - - // Check for unmatched backtick - if (newPortion.includes('`')) { - const hasCompleteCodeSpan = hasCompleteCodeSpanAt(inlineTree.rootNode, newPortionStart, newPortionEnd) - if (!hasCompleteCodeSpan) { - state.pendingInlineContent = newContent - state.pendingInlineStartIndex = actualFromIndex - state.sourceOffset = actualToIndex - return { segments, state } + // Check for unmatched bold markers + if (newPortion.includes('**')) { + const hasCompleteBold = hasCompleteBoldAt(inlineTree.rootNode, newPortionStart, newPortionEnd) + if (!hasCompleteBold) { + state.pendingInlineContent = newContent + state.pendingInlineStartIndex = actualFromIndex + state.sourceOffset = actualToIndex + return { segments, state } + } } - } - // Check for unmatched bold markers - if (newPortion.includes('**')) { - const hasCompleteBold = hasCompleteBoldAt(inlineTree.rootNode, newPortionStart, newPortionEnd) - if (!hasCompleteBold) { - state.pendingInlineContent = newContent - state.pendingInlineStartIndex = actualFromIndex - state.sourceOffset = actualToIndex - return { segments, state } + // Check for unmatched italic markers (skip if inside code block) + const insideCodeBlock = isInsideCodeBlock(currentTree.rootNode, actualFromIndex, currentTree, inlineParser) + if (!insideCodeBlock) { + const hasUnmatchedItalic = hasUnmatchedItalicMarker(newPortion, inlineParser) + if (hasUnmatchedItalic) { + const hasCompleteItalic = hasCompleteItalicAt(inlineTree.rootNode, newPortionStart, newPortionEnd) + if (!hasCompleteItalic) { + state.pendingInlineContent = newContent + state.pendingInlineStartIndex = actualFromIndex + state.sourceOffset = actualToIndex + return { segments, state } + } + } } - } - // Check for unmatched italic markers (skip if inside code block) - const insideCodeBlock = isInsideCodeBlock(currentTree.rootNode, actualFromIndex, currentTree, inlineParser) - if (!insideCodeBlock) { - const hasUnmatchedItalic = hasUnmatchedItalicMarker(newPortion, inlineParser) - if (hasUnmatchedItalic) { - const hasCompleteItalic = hasCompleteItalicAt(inlineTree.rootNode, newPortionStart, newPortionEnd) - if (!hasCompleteItalic) { + // Check for unmatched strikethrough markers + if (newPortion.includes('~~')) { + const hasCompleteStrikethrough = hasCompleteStrikethroughAt(inlineTree.rootNode, newPortionStart, newPortionEnd) + if (!hasCompleteStrikethrough) { state.pendingInlineContent = newContent state.pendingInlineStartIndex = actualFromIndex state.sourceOffset = actualToIndex return { segments, state } } } - } - - // Check for unmatched strikethrough markers - if (newPortion.includes('~~')) { - const hasCompleteStrikethrough = hasCompleteStrikethroughAt(inlineTree.rootNode, newPortionStart, newPortionEnd) - if (!hasCompleteStrikethrough) { - state.pendingInlineContent = newContent - state.pendingInlineStartIndex = actualFromIndex - state.sourceOffset = actualToIndex - return { segments, state } - } - } - // Check for incomplete link opening [ - if (newPortion.includes('[')) { - const hasCompleteLink = hasCompleteLinkAt(inlineTree.rootNode, newPortionStart, newPortionEnd) - if (!hasCompleteLink && hasIncompleteLinkOpening(newPortion, inlineParser)) { - state.pendingInlineContent = newContent - state.pendingInlineStartIndex = actualFromIndex - state.sourceOffset = actualToIndex - return { segments, state } + // Check for incomplete link opening [ + if (newPortion.includes('[')) { + const hasCompleteLink = hasCompleteLinkAt(inlineTree.rootNode, newPortionStart, newPortionEnd) + if (!hasCompleteLink && hasIncompleteLinkOpening(newPortion, inlineParser)) { + state.pendingInlineContent = newContent + state.pendingInlineStartIndex = actualFromIndex + state.sourceOffset = actualToIndex + return { segments, state } + } } - } - // Check for incomplete image opening ![ - if (newPortion.includes('![')) { - const hasCompleteImage = hasCompleteImageAt(inlineTree.rootNode, newPortionStart, newPortionEnd) - if (!hasCompleteImage && hasIncompleteImageOpening(newPortion, inlineParser)) { - state.pendingInlineContent = newContent - state.pendingInlineStartIndex = actualFromIndex - state.sourceOffset = actualToIndex - return { segments, state } + // Check for incomplete image opening ![ + if (newPortion.includes('![')) { + const hasCompleteImage = hasCompleteImageAt(inlineTree.rootNode, newPortionStart, newPortionEnd) + if (!hasCompleteImage && hasIncompleteImageOpening(newPortion, inlineParser)) { + state.pendingInlineContent = newContent + state.pendingInlineStartIndex = actualFromIndex + state.sourceOffset = actualToIndex + return { segments, state } + } } } } @@ -641,40 +642,42 @@ export function generateSegments( usedInlineContent = hostInlineNode !== null const inlineContent = hostInlineNode?.text ?? processedContent const inlineTree = inlineParser.parse(inlineContent) - const delimiterRanges = collectInlineDelimiterRanges(inlineTree) - const chunkStartInInline = hostInlineNode - ? Math.max(0, actualFromIndex - hostInlineNode.startIndex) - : 0 - const chunkEndInInline = hostInlineNode - ? Math.max(chunkStartInInline, actualToIndex - hostInlineNode.startIndex) - : processedContent.length - - const spanResult = processInlineSpans( - inlineTree, - chunkStartInInline, - chunkEndInInline, - inlineContent, - state, - delimiterRanges, - chunkStartUtf16 - rawToRenderedOffset(chunkStartInInline, delimiterRanges) - ) - opening = spanResult.opening - closing = spanResult.closing - contained = spanResult.contained - state = { ...state, openSpans: spanResult.newOpenSpans } - - // Strip inline markers from the content - strippedContent = getInlineContent( - hostInlineNode ? inlineContent.substring(chunkStartInInline, chunkEndInInline) : processedContent, - inlineTree, - chunkStartInInline, - chunkEndInInline - ) - - if (hostInlineNode && blockInfo.type !== 'header' && actualToIndex > hostInlineNode.endIndex) { - const tailStart = Math.max(actualFromIndex, hostInlineNode.endIndex) - const tailText = content.substring(tailStart, actualToIndex) - strippedContent += stripListSuppressedRanges(tailText, currentTree.rootNode, content, tailStart, actualToIndex) + if (inlineTree) { + const delimiterRanges = collectInlineDelimiterRanges(inlineTree) + const chunkStartInInline = hostInlineNode + ? Math.max(0, actualFromIndex - hostInlineNode.startIndex) + : 0 + const chunkEndInInline = hostInlineNode + ? Math.max(chunkStartInInline, actualToIndex - hostInlineNode.startIndex) + : processedContent.length + + const spanResult = processInlineSpans( + inlineTree, + chunkStartInInline, + chunkEndInInline, + inlineContent, + state, + delimiterRanges, + chunkStartUtf16 - rawToRenderedOffset(chunkStartInInline, delimiterRanges) + ) + opening = spanResult.opening + closing = spanResult.closing + contained = spanResult.contained + state = { ...state, openSpans: spanResult.newOpenSpans } + + // Strip inline markers from the content + strippedContent = getInlineContent( + hostInlineNode ? inlineContent.substring(chunkStartInInline, chunkEndInInline) : processedContent, + inlineTree, + chunkStartInInline, + chunkEndInInline + ) + + if (hostInlineNode && blockInfo.type !== 'header' && actualToIndex > hostInlineNode.endIndex) { + const tailStart = Math.max(actualFromIndex, hostInlineNode.endIndex) + const tailText = content.substring(tailStart, actualToIndex) + strippedContent += stripListSuppressedRanges(tailText, currentTree.rootNode, content, tailStart, actualToIndex) + } } } diff --git a/vitest.config.ts b/vitest.config.ts index bb2b066..ef3f5f2 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -4,6 +4,7 @@ export default defineConfig({ test: { environment: 'node', globals: true, + include: ['src/**/*.test.ts'], }, esbuild: { target: 'node23' From 61722569c3eee2dbafd16cd0687284520eaae342 Mon Sep 17 00:00:00 2001 From: Dmitry Bondarenko Date: Tue, 14 Jul 2026 22:30:40 +0600 Subject: [PATCH 3/4] Fix(prosemirror): harden stream assembly and test fixtures --- PLAN.md | 49 ++ .../lib/prosemirror/stream-assembly.test.ts | 38 ++ .../src/lib/prosemirror/stream-assembly.ts | 512 +++++------------- .../stream-examples.integration.test.ts | 3 +- 4 files changed, 238 insertions(+), 364 deletions(-) diff --git a/PLAN.md b/PLAN.md index d6ce584..7d78391 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,5 +1,54 @@ # Migrate svelte-demo rendering to ProseMirror (LIX-MDSP-20) +## TODO - ProseMirror adjacency review findings (2026-07-14) + +The demo migration is architecturally aligned with Lixpi, but the following issues must be resolved before treating it as a promotion-ready reference implementation. The current Lixpi reference is `/home/dima/Desktop/lixpi/packages/lixpi/prosemirror/src/shared/`; its schema and bounded transaction APIs have evolved since the original plan was written. + +### 1. Preserve sibling and nested list topology (blocker) +`buildContentFromChunks` currently calls `flushLists(list.depth - 1)` before every list item. Because `flushLists` pops frames whose depth is greater than the supplied depth, a same-depth sibling flushes the current list and starts another list node. Nested siblings can likewise become multiple nested list nodes instead of items in one list. + +Fix the depth-stack transition so it: +- keeps the current frame for a sibling with the same depth and list type; +- flushes only deeper frames when returning to a shallower depth; +- flushes and replaces the current frame when the list type changes at the same depth; +- preserves ordered-list starts and attaches nested lists to the correct parent item. + +Add structural tests that assert one list contains multiple sibling `list_item` nodes, nested siblings share one nested list, mixed ordered/unordered transitions produce the intended separate lists, and returning from a nested list continues the original parent list. + +### 2. Define and implement the real Lixpi portability contract (blocker) +The statement that `buildContentFromChunks(schema, chunks)` runs against Lixpi's `createProseMirrorSchema(...)` unchanged is currently false: +- the demo defines `image` as inline, while current Lixpi defines `image` as a block node; +- current Lixpi schemas do not include `bullet_list`, `ordered_list`, `list_item`, or table node types; +- current Lixpi documents use composed roots and target nodes such as `documentTitle`, `aiChatThread`, and `aiResponseMessage`; +- Lixpi streaming publishes bounded transaction steps, while the demo replaces the entire document. + +Decide the canonical contract, then either extend Lixpi's schema builder with the portable list/table/image specs or make assembly capability-aware through an explicit adapter. Add a Lixpi-side adapter that replaces only the target response node content and returns transaction steps compatible with `HeadlessProseMirrorEngine`. Add contract tests using the actual `createProseMirrorSchema(DOCUMENT_TYPE.AI_CHAT_THREAD)` schema. Update this plan's portability claims to describe the resulting contract precisely. + +### 3. Make integration tests reproducible from a clean checkout +`stream-examples.integration.test.ts` reads fixtures from ignored/generated `demo/svelte-demo/static/llm-streams-examples`, but `pnpm --dir demo/svelte-demo run test` does not run `copy-llm-examples`. A clean checkout therefore lacks the test inputs. + +Read fixtures directly from tracked `demo/llm-streams-examples`, or add an explicit test preparation step. Verify the test command succeeds after removing generated Svelte/static output and without first running dev/build/prepack. + +### 4. Align image URL sanitization with the documented policy +`sanitizeImageSrc` accepts `/...`, `./...`, and `../...`, but rejects bare path-relative sources such as `image.png`, despite the plan allowing path-relative URLs. Define the allowed `data:image/*` MIME types rather than accepting every image subtype through one broad regex; explicitly decide whether SVG data URLs are permitted. + +Add tests for bare relative paths, query/hash-only edge cases, protocol-relative URLs, encoded or mixed-case script schemes, raster data URLs, SVG data URLs, malformed data URLs, and non-image data URLs. Unsafe sources must remain plain covered text. + +### 5. Assemble images whose closed span crosses chunk boundaries +`buildTextRuns` currently recognizes an image only when the complete image span exactly matches one run inside one chunk. A closed image span covering multiple chunks remains text instead of being replaced by one image node. + +Build image projection from absolute span ranges across the block, consume all covered text runs once the span closes, and insert exactly one sanitized image node. Keep open image spans as plain text until metadata becomes available. Add tests for contained, multi-chunk closing, unsafe multi-chunk, adjacent, and image-with-surrounding-text cases. + +### 6. Enforce schema validity and make malformed fallback observable +The malformed-table test only asserts that a document exists. `NodeType.create` can construct an empty table or row that violates its content expression without throwing, so the broad catch fallback may never run. + +Validate assembled block nodes or the completed document with `check()`/`validContent`, use `createAndFill` where appropriate, and fall back to a paragraph when partial metadata cannot form a valid node. Do not silently hide schema-contract programming errors: distinguish expected incomplete-stream fallback from unexpected assembly failures. Add assertions that malformed and partial states pass `doc.check()` and that valid partial table states retain all available cells. + +### 7. Complete real browser verification and correct stale completion claims +The current integration tests exercise parser-to-buffer-to-document projection, but they do not exercise `EditorView`, Svelte lifecycle, controls, debug columns, or rendered DOM. The plan also states that the dev server is running, which is transient environment state rather than a completed repository guarantee. + +Add browser E2E coverage for full play, pause/resume, single-step, reset mid-stream, replay after completion, switching examples mid-stream, backtrack correction, and rendered heading/list/code/table/task/strikethrough/image output. Assert the debug columns stay synchronized with the ProseMirror projection. Run this in a container/image that includes the selected browser runtime, then replace transient statements in the completed-findings section with reproducible commands and recorded outcomes. + ## Completed follow-up review findings (2026-07-08) The plan below has been implemented (new module in `demo/svelte-demo/src/lib/prosemirror/`, legacy rendering removed from `+page.svelte`). The post-implementation review items were applied inside the docker container `lixpi-markdown-stream-parser-demo`. diff --git a/demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts index 99f953b..0924e46 100644 --- a/demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts +++ b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts @@ -45,6 +45,13 @@ describe('stream assembly helpers', () => { expect(sanitizeImageSrc('data:image/png;base64,AAAA')).toBe('data:image/png;base64,AAAA') expect(sanitizeImageSrc('//example.com/image.png')).toBeNull() expect(sanitizeImageSrc('data:text/html;base64,AAAA')).toBeNull() + expect(sanitizeImageSrc('image.png')).toBe('image.png') + expect(sanitizeImageSrc('../image.png')).toBe('../image.png') + expect(sanitizeImageSrc('?cache=1')).toBeNull() + expect(sanitizeImageSrc('#fragment')).toBeNull() + expect(sanitizeImageSrc('JaVaScRiPt:alert(1)')).toBeNull() + expect(sanitizeImageSrc('data:image/svg+xml;base64,PHN2Zz4=')).toBeNull() + expect(sanitizeImageSrc('data:image/jpeg;base64,AAAA')).toBe('data:image/jpeg;base64,AAAA') }) it('builds marks and rejects unsafe links', () => { @@ -145,6 +152,36 @@ describe('stream assembly helpers', () => { expect(doc.child(1).type.name).toBe('table') expect(doc.child(2).type.name).toBe('code_block') expect(doc.child(2).attrs.language).toBe('ts') + expect(() => doc.check()).not.toThrow() + }) + + it('preserves same-depth siblings, nested siblings, and the parent list on return', () => { + const list = (text: string, offset: number, depth: number, type: 'ordered' | 'unordered' = 'unordered', ordinal?: number) => chunk({ + text, + offset, + block: { type: 'list_item', list: { type, depth, marker: type === 'ordered' ? '.' : '-', ordinal } }, + }) + const doc = buildDocFromChunks(schema, [ + list('one\n', 0, 0), list('child a\n', 4, 1), list('child b\n', 12, 1), list('two', 20, 0), + list('numbered', 23, 0, 'ordered', 3), + ]) + expect(doc.child(0).type.name).toBe('bullet_list') + expect(doc.child(0).childCount).toBe(2) + const nested = doc.child(0).child(0).lastChild! + expect(nested.type.name).toBe('bullet_list') + expect(nested.childCount).toBe(2) + expect(doc.child(1).type.name).toBe('ordered_list') + expect(doc.child(1).attrs.order).toBe(3) + expect(() => doc.check()).not.toThrow() + }) + + it('projects a closed image span across chunks once while retaining surrounding text', () => { + const nodes = buildInlineContent(schema, [ + chunk({ text: 'before lo', offset: 0 }), + chunk({ text: 'go after', offset: 9, closing: [{ type: 'image', offset: 7, length: 4, src: 'image.png', alt: 'logo' }] }), + ]) + expect(nodes.map(node => node.isText ? node.text : `[${node.type.name}]`).join('')).toBe('before [image] after') + expect(nodes.filter(node => node.type.name === 'image')).toHaveLength(1) }) it('falls back instead of throwing for malformed block states', () => { @@ -154,5 +191,6 @@ describe('stream assembly helpers', () => { expect(doc.type.name).toBe('doc') expect(doc.childCount).toBeGreaterThan(0) + expect(() => doc.check()).not.toThrow() }) }) diff --git a/demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts index 7cb8532..624d20e 100644 --- a/demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts +++ b/demo/svelte-demo/src/lib/prosemirror/stream-assembly.ts @@ -1,465 +1,251 @@ import { Fragment, type Mark, type Node as ProseMirrorNode, type Schema } from 'prosemirror-model' -import type { Chunk, ClosedSpan, OpenSpan, Span, SpanType } from '../../../../../src/markdown-stream-parser.ts' +import type { Chunk, ClosedSpan, OpenSpan, SpanType } from '../../../../../src/markdown-stream-parser.ts' -type BlockType = Chunk['block']['type'] type ListMetadata = NonNullable -type ListFrame = { - type: ListMetadata['type'] - depth: number - order: number - items: ProseMirrorNode[] -} -type CellGroup = { - cellId: string - columnIndex: number - type: 'table_header_cell' | 'table_cell' - align?: 'left' | 'center' | 'right' - chunks: Chunk[] -} -type MarkRange = { - type: SpanType - start: number - end: number -} +type ListFrame = { type: ListMetadata['type']; depth: number; order: number; items: ProseMirrorNode[] } +type CellGroup = { cellId: string; columnIndex: number; type: 'table_header_cell' | 'table_cell'; align?: 'left' | 'center' | 'right'; chunks: Chunk[] } +type MarkRange = { type: Exclude; start: number; end: number } +type TextRun = { start: number; end: number; text: string; image?: ClosedSpan & { type: 'image'; src: string; alt?: string } } export function isChunkBeforeBacktrack(chunk: Chunk, backtrackOffset: number): boolean { return chunk.offset + chunk.length <= backtrackOffset } export function applyStreamingChunkToBuffer(buffer: Chunk[], chunk: Chunk): Chunk[] { - const next = chunk.backtrackOffset === undefined - ? buffer - : buffer.filter(bufferedChunk => isChunkBeforeBacktrack(bufferedChunk, chunk.backtrackOffset!)) + const next = chunk.backtrackOffset === undefined ? buffer : buffer.filter(item => isChunkBeforeBacktrack(item, chunk.backtrackOffset!)) return [...next, chunk] } export function sanitizeLinkHref(rawHref: string): string | null { const href = rawHref.trim() if (!href) return null - try { const url = new URL(href) - return url.protocol === 'http:' || url.protocol === 'https:' || url.protocol === 'mailto:' - ? href - : null - } catch { - return null - } + return ['http:', 'https:', 'mailto:'].includes(url.protocol.toLowerCase()) ? href : null + } catch { return null } } +// SVG data URLs are intentionally excluded: SVG is an active document format, not a raster image. +const safeDataImage = /^data:image\/(?:png|apng|gif|jpe?g|webp|avif);base64,[a-z0-9+/]*={0,2}$/i + export function sanitizeImageSrc(rawSrc: string): string | null { const src = rawSrc.trim() - if (!src || src.startsWith('//')) return null - if (src.startsWith('/') || src.startsWith('./') || src.startsWith('../')) return src - + if (!src || src.startsWith('//') || src.startsWith('?') || src.startsWith('#')) return null + if (src.startsWith('/') || /^(?:\.\.?\/)?[^:/?#][^:]*$/u.test(src)) return src + if (safeDataImage.test(src)) return src try { const url = new URL(src) - if (url.protocol === 'http:' || url.protocol === 'https:') return src - if (url.protocol === 'data:' && /^data:image\/[a-z0-9.+-]+;base64,/i.test(src)) return src - return null - } catch { - return null - } + return ['http:', 'https:'].includes(url.protocol.toLowerCase()) ? src : null + } catch { return null } } export function groupChunksIntoBlocks(chunks: Chunk[]): Chunk[][] { - const blocks: Chunk[][] = [] - let currentBlock: Chunk[] = [] - let lastBlockType: BlockType | undefined - let lastBlockLevel: number | undefined - let lastTableId: string | undefined - let lastListDepth: number | undefined - let lastListType: ListMetadata['type'] | undefined - let lastOffset = -1 - + const result: Chunk[][] = [] + let current: Chunk[] = [] for (const chunk of chunks) { - const blockType = chunk.block.type - const blockLevel = chunk.block.level - const tableId = chunk.block.table?.tableId - const listDepth = chunk.block.list?.depth - const listType = chunk.block.list?.type - let isNewBlock = false - - if ( - blockType !== lastBlockType - && !(isTableCellBlockType(blockType) && isTableCellBlockType(lastBlockType) && tableId === lastTableId) - ) { - isNewBlock = true - } else if (tableId !== lastTableId) { - isNewBlock = true - } else if (blockType === 'heading' && blockLevel !== lastBlockLevel) { - isNewBlock = true - } else if (blockType === 'list_item' && (listDepth !== lastListDepth || listType !== lastListType)) { - isNewBlock = true - } else if (blockType === 'list_item' && lastOffset >= 0) { - const previousChunk = currentBlock[currentBlock.length - 1] - if (previousChunk?.text.endsWith('\n') && chunk.text.trim().length > 0) { - isNewBlock = true - } + const previous = current.at(-1) + const typeChanged = previous && chunk.block.type !== previous.block.type && !(isTableCellBlockType(chunk.block.type) && isTableCellBlockType(previous.block.type) && chunk.block.table?.tableId === previous.block.table?.tableId) + const tableChanged = previous && chunk.block.table?.tableId !== previous.block.table?.tableId + const headingChanged = previous && chunk.block.type === 'heading' && chunk.block.level !== previous.block.level + const listChanged = previous && chunk.block.type === 'list_item' && (chunk.block.list?.depth !== previous.block.list?.depth || chunk.block.list?.type !== previous.block.list?.type) + const nextListItem = previous && chunk.block.type === 'list_item' && previous.text.endsWith('\n') && chunk.text.trim().length > 0 + if ((typeChanged || tableChanged || headingChanged || listChanged || nextListItem) && current.length) { + result.push(current) + current = [] } - - if (isNewBlock && currentBlock.length > 0) { - blocks.push(currentBlock) - currentBlock = [] - } - - currentBlock.push(chunk) - lastBlockType = blockType - lastBlockLevel = blockLevel - lastTableId = tableId - lastListDepth = listDepth - lastListType = listType - lastOffset = chunk.offset + chunk.length + current.push(chunk) } - - if (currentBlock.length > 0) blocks.push(currentBlock) - return blocks + if (current.length) result.push(current) + return result } export function buildInlineContent(schema: Schema, blockChunks: Chunk[]): ProseMirrorNode[] { - const nodes: ProseMirrorNode[] = [] const chunks = trimTrailingNewline(blockChunks) - const closedSpans = chunks.flatMap(chunk => [...chunk.contained, ...chunk.closing]) - const imageSpans = dedupeClosedSpans(closedSpans.filter((span): span is ClosedSpan & { type: 'image'; src: string; alt?: string } => span.type === 'image')) - const markRanges = buildMarkRanges(chunks, closedSpans) - const textRuns = buildTextRuns(chunks, closedSpans, markRanges, imageSpans) - - for (const run of textRuns) { + const closed = dedupeClosedSpans(chunks.flatMap(chunk => [...chunk.contained, ...chunk.closing])) + const marks = buildMarkRanges(chunks, closed) + return buildTextRuns(chunks, closed, marks).flatMap(run => { if (run.image) { const src = sanitizeImageSrc(run.image.src) - if (src) { - nodes.push(schema.nodes.image.create({ src, alt: run.image.alt ?? null })) - continue - } + if (src && schema.nodes.image?.isInline) return [schema.nodes.image.create({ src, alt: run.image.alt ?? null })] } - - if (!run.text) continue - const marks = createMarksForRange(schema, closedSpans, markRanges, run.start, run.end) - nodes.push(marks.length > 0 ? schema.text(run.text, marks) : schema.text(run.text)) - } - - return nodes + if (!run.text) return [] + const active = createMarksForRange(schema, closed, marks, run.start, run.end) + return [schema.text(run.text, active)] + }) } +/** Schema-parameterized content builder. Schemas without optional list/table/image nodes degrade to valid paragraphs/text. */ export function buildContentFromChunks(schema: Schema, chunks: Chunk[]): Fragment { - const blocks = groupChunksIntoBlocks(chunks) const nodes: ProseMirrorNode[] = [] - let listFrames: ListFrame[] = [] - - function flushLists(toDepth = -1): void { - while (listFrames.length > 0 && listFrames[listFrames.length - 1].depth > toDepth) { - const frame = listFrames.pop()! + const frames: ListFrame[] = [] + const flush = (minimumDepth = -1) => { + while (frames.length && frames.at(-1)!.depth >= minimumDepth) { + const frame = frames.pop()! const listNode = createListNode(schema, frame) - if (listFrames.length > 0) { - const parent = listFrames[listFrames.length - 1] - const lastItem = parent.items.pop() - if (lastItem) { - parent.items.push(appendBlockToListItem(schema, lastItem, listNode)) - } else { - parent.items.push(schema.nodes.list_item.create(null, listNode)) - } - } else { - nodes.push(listNode) - } + if (!listNode) { + nodes.push(...frame.items.map(item => schema.nodes.paragraph.create(null, item.textContent ? schema.text(item.textContent) : null))) + } else if (frames.length) { + const parent = frames.at(-1)! + const item = parent.items.pop() + parent.items.push(item ? appendBlockToListItem(schema, item, listNode) : schema.nodes.list_item.create(null, listNode)) + } else nodes.push(listNode) } } - + const blocks = groupChunksIntoBlocks(chunks) for (let index = 0; index < blocks.length; index++) { const block = blocks[index] - const firstChunk = block[0] - const nextBlock = blocks[index + 1] - + const first = block[0] try { - if (firstChunk?.block.type === 'list_item' && firstChunk.block.list) { - const list = firstChunk.block.list - flushLists(list.depth - 1) - let frame = listFrames[listFrames.length - 1] - if (!frame || frame.depth !== list.depth || frame.type !== list.type) { - frame = { - type: list.type, - depth: list.depth, - order: list.ordinal ?? 1, - items: [], - } - listFrames.push(frame) - } - frame.items.push(createListItemNode(schema, block, list)) - if (nextBlock?.[0]?.block.list === undefined) flushLists() + if (first?.block.type === 'list_item' && first.block.list) { + const list = first.block.list + // Keep same-depth siblings; close only descendants. A different list type replaces the frame. + while (frames.length && frames.at(-1)!.depth > list.depth) flush(frames.at(-1)!.depth) + if (frames.at(-1)?.depth === list.depth && frames.at(-1)?.type !== list.type) flush(list.depth) + if (!frames.at(-1) || frames.at(-1)!.depth !== list.depth) frames.push({ type: list.type, depth: list.depth, order: list.ordinal ?? 1, items: [] }) + frames.at(-1)!.items.push(createListItemNode(schema, block, list)) + if (!blocks[index + 1]?.[0]?.block.list) flush(0) continue } - - flushLists() - - if (isTableCellBlockType(firstChunk?.block.type)) { + flush(0) + if (isTableCellBlockType(first?.block.type)) { + const tableId = first.block.table?.tableId const tableBlocks = [block] - const tableId = firstChunk?.block.table?.tableId - while (blocks[index + 1]?.[0]?.block.table?.tableId === tableId) { - tableBlocks.push(blocks[++index]) - } - nodes.push(createTableNode(schema, tableBlocks.flat())) - continue - } - - nodes.push(createBlockNode(schema, block)) - } catch { - flushLists() - nodes.push(schema.nodes.paragraph.create(null, createPlainTextContent(schema, block))) + while (tableId !== undefined && blocks[index + 1]?.[0]?.block.table?.tableId === tableId) tableBlocks.push(blocks[++index]) + const table = createTableNode(schema, tableBlocks.flat()) + nodes.push(table ?? fallbackParagraph(schema, tableBlocks.flat())) + } else nodes.push(createBlockNode(schema, block)) + } catch (error) { + console.warn('Unexpected ProseMirror stream assembly failure; using plain-text fallback.', error) + flush(0) + nodes.push(fallbackParagraph(schema, block)) } } - - flushLists() - return nodes.length > 0 ? Fragment.fromArray(nodes) : Fragment.from(schema.nodes.paragraph.create()) + flush(0) + return nodes.length ? Fragment.fromArray(nodes) : Fragment.from(schema.nodes.paragraph.create()) } export function buildDocFromChunks(schema: Schema, chunks: Chunk[]): ProseMirrorNode { - return schema.nodes.doc.create(null, buildContentFromChunks(schema, chunks)) + const doc = schema.nodes.doc.createAndFill(null, buildContentFromChunks(schema, chunks)) + if (!doc) throw new Error('Schema cannot create a valid document from streaming content') + doc.check() + return doc } function createBlockNode(schema: Schema, block: Chunk[]): ProseMirrorNode { const first = block[0] - const inlineContent = buildInlineContent(schema, block) - - switch (first?.block.type) { - case 'heading': - return schema.nodes.heading.create({ level: first.block.level ?? 1 }, inlineContent) - case 'code_block': - return schema.nodes.code_block.create( - { language: first.block.language ?? '' }, - createTextNodeOrNull(schema, trimTrailingNewline(block).map(chunk => chunk.text).join('')), - ) - case 'blockquote': - return schema.nodes.blockquote.create(null, schema.nodes.paragraph.create(null, inlineContent)) - case 'paragraph': - case undefined: - default: - return schema.nodes.paragraph.create(null, inlineContent) - } + const inline = buildInlineContent(schema, block) + const node = first?.block.type === 'heading' ? schema.nodes.heading?.createAndFill({ level: first.block.level ?? 1 }, inline) + : first?.block.type === 'code_block' ? schema.nodes.code_block?.createAndFill({ language: first.block.language ?? '' }, createTextNodeOrNull(schema, trimTrailingNewline(block).map(c => c.text).join(''))) + : first?.block.type === 'blockquote' ? schema.nodes.blockquote?.createAndFill(null, schema.nodes.paragraph.create(null, inline)) + : schema.nodes.paragraph.createAndFill(null, inline) + return node ?? fallbackParagraph(schema, block) } function createListItemNode(schema: Schema, block: Chunk[], list: ListMetadata): ProseMirrorNode { - const paragraph = schema.nodes.paragraph.create(null, buildInlineContent(schema, block)) - return schema.nodes.list_item.create({ task: list.task ?? null }, paragraph) + return schema.nodes.list_item.createAndFill({ task: list.task ?? null }, schema.nodes.paragraph.create(null, buildInlineContent(schema, block))) ?? fallbackParagraph(schema, block) } -function createListNode(schema: Schema, frame: ListFrame): ProseMirrorNode { +function createListNode(schema: Schema, frame: ListFrame): ProseMirrorNode | null { const type = frame.type === 'ordered' ? schema.nodes.ordered_list : schema.nodes.bullet_list - return frame.type === 'ordered' - ? type.create({ order: frame.order }, frame.items) - : type.create(null, frame.items) + return type?.createAndFill(frame.type === 'ordered' ? { order: frame.order } : null, frame.items) ?? null } function appendBlockToListItem(schema: Schema, item: ProseMirrorNode, block: ProseMirrorNode): ProseMirrorNode { - return schema.nodes.list_item.create(item.attrs, item.content.append(Fragment.from(block))) + return schema.nodes.list_item.createAndFill(item.attrs, item.content.append(Fragment.from(block))) ?? item } -function createTableNode(schema: Schema, chunks: Chunk[]): ProseMirrorNode { +function createTableNode(schema: Schema, chunks: Chunk[]): ProseMirrorNode | null { + if (!schema.nodes.table || !schema.nodes.table_row || !schema.nodes.table_cell || !schema.nodes.table_header_cell) return null const rows = buildTableRows(chunks) + if (!rows.length || rows.some(row => !row.cells.length)) return null const rowNodes = rows.map(row => { - const cellNodes = row.cells.map(cell => { - const cellType = cell.type === 'table_header_cell' - ? schema.nodes.table_header_cell - : schema.nodes.table_cell - return cellType.create({ align: cell.align ?? null }, buildInlineContent(schema, cell.chunks)) - }) - return schema.nodes.table_row.create(null, cellNodes) + const cells = row.cells + .map(cell => (cell.type === 'table_header_cell' ? schema.nodes.table_header_cell : schema.nodes.table_cell).createAndFill({ align: cell.align ?? null }, buildInlineContent(schema, cell.chunks))) + .filter((node): node is ProseMirrorNode => node !== null) + return schema.nodes.table_row.createAndFill(null, cells) }) - return schema.nodes.table.create(null, rowNodes) + if (rowNodes.some(node => !node)) return null + const table = schema.nodes.table.createAndFill(null, rowNodes.filter((node): node is ProseMirrorNode => node !== null)) + if (table) table.check() + return table } function buildTableRows(chunks: Chunk[]): Array<{ rowIndex: number; cells: CellGroup[] }> { const rows = new Map>() - for (const chunk of chunks) { - const table = chunk.block.table - if (!table) continue - - let row = rows.get(table.rowIndex) - if (!row) { - row = new Map() - rows.set(table.rowIndex, row) - } - - let cell = row.get(table.cellId) - if (!cell) { - cell = { - cellId: table.cellId, - columnIndex: table.columnIndex, - type: chunk.block.type === 'table_header_cell' ? 'table_header_cell' : 'table_cell', - align: table.align, - chunks: [], - } - row.set(table.cellId, cell) - } - + const meta = chunk.block.table + if (!meta) continue + let row = rows.get(meta.rowIndex); if (!row) rows.set(meta.rowIndex, row = new Map()) + let cell = row.get(meta.cellId) + if (!cell) row.set(meta.cellId, cell = { cellId: meta.cellId, columnIndex: meta.columnIndex, type: chunk.block.type === 'table_header_cell' ? 'table_header_cell' : 'table_cell', align: meta.align, chunks: [] }) cell.chunks.push(chunk) } - - return [...rows.entries()] - .sort((a, b) => a[0] - b[0]) - .map(([rowIndex, cells]) => ({ - rowIndex, - cells: [...cells.values()].sort((a, b) => a.columnIndex - b.columnIndex), - })) + return [...rows].sort(([a], [b]) => a - b).map(([rowIndex, cells]) => ({ rowIndex, cells: [...cells.values()].sort((a, b) => a.columnIndex - b.columnIndex) })) } -function buildTextRuns( - chunks: Chunk[], - spans: ClosedSpan[], - markRanges: MarkRange[], - imageSpans: Array, -): Array<{ start: number; end: number; text: string; image?: Span & { type: 'image' } }> { - const runs: Array<{ start: number; end: number; text: string; image?: Span & { type: 'image' } }> = [] - - for (const chunk of chunks) { - const chunkStart = chunk.offset - const chunkEnd = chunk.offset + chunk.length - const boundaries = new Set([chunkStart, chunkEnd]) - - for (const span of spans) { - const spanStart = span.offset - const spanEnd = span.offset + span.length - if (spanStart > chunkStart && spanStart < chunkEnd) boundaries.add(spanStart) - if (spanEnd > chunkStart && spanEnd < chunkEnd) boundaries.add(spanEnd) - } - for (const range of markRanges) { - if (range.start > chunkStart && range.start < chunkEnd) boundaries.add(range.start) - if (range.end > chunkStart && range.end < chunkEnd) boundaries.add(range.end) - } - - const sortedBoundaries = [...boundaries].sort((a, b) => a - b) - for (let index = 0; index < sortedBoundaries.length - 1; index++) { - const start = sortedBoundaries[index] - const end = sortedBoundaries[index + 1] - if (start === end) continue - const image = imageSpans.find(span => span.offset === start && span.offset + span.length === end) - runs.push({ - start, - end, - text: chunk.text.slice(start - chunk.offset, end - chunk.offset), - image, - }) - } +function buildTextRuns(chunks: Chunk[], spans: ClosedSpan[], markRanges: MarkRange[]): TextRun[] { + const images = spans.filter((span): span is ClosedSpan & { type: 'image'; src: string; alt?: string } => span.type === 'image' && isFullyCovered(chunks, span.offset, span.offset + span.length)) + const boundaries = new Set() + for (const chunk of chunks) { boundaries.add(chunk.offset); boundaries.add(chunk.offset + chunk.length) } + for (const span of spans) { boundaries.add(span.offset); boundaries.add(span.offset + span.length) } + for (const range of markRanges) { boundaries.add(range.start); boundaries.add(range.end) } + const points = [...boundaries].sort((a, b) => a - b) + const runs: TextRun[] = [] + for (let i = 0; i < points.length - 1;) { + const start = points[i], image = images.find(span => span.offset === start) + const end = image ? image.offset + image.length : points[i + 1] + const text = textForRange(chunks, start, end) + runs.push({ start, end, text, image }) + i = image ? points.findIndex(point => point === end) : i + 1 } - return runs } -function buildMarkRanges(chunks: Chunk[], spans: ClosedSpan[]): MarkRange[] { - const ranges: MarkRange[] = [] - const openSpans: OpenSpan[] = [] - const blockEnd = chunks.reduce((max, chunk) => Math.max(max, chunk.offset + chunk.length), 0) +function textForRange(chunks: Chunk[], start: number, end: number): string { + return chunks.map(chunk => { + const from = Math.max(start, chunk.offset), to = Math.min(end, chunk.offset + chunk.length) + return from < to ? chunk.text.slice(from - chunk.offset, to - chunk.offset) : '' + }).join('') +} - for (const span of spans) { - if (span.type === 'link' || span.type === 'image') continue - ranges.push({ type: span.type, start: span.offset, end: span.offset + span.length }) +function isFullyCovered(chunks: Chunk[], start: number, end: number): boolean { + let cursor = start + for (const chunk of [...chunks].sort((a, b) => a.offset - b.offset)) { + if (chunk.offset > cursor) return false + if (chunk.offset + chunk.length > cursor) cursor = Math.min(end, chunk.offset + chunk.length) + if (cursor === end) return true } + return false +} +function buildMarkRanges(chunks: Chunk[], spans: ClosedSpan[]): MarkRange[] { + const ranges = spans.filter(span => span.type !== 'link' && span.type !== 'image').map(span => ({ type: span.type, start: span.offset, end: span.offset + span.length })) + const open: Array = [], end = Math.max(0, ...chunks.map(chunk => chunk.offset + chunk.length)) for (const chunk of chunks) { - for (const span of chunk.opening) { - if (span.type === 'link' || span.type === 'image') continue - openSpans.push(span) - } - + for (const span of chunk.opening) if (span.type !== 'link' && span.type !== 'image') open.push(span as OpenSpan & { type: MarkRange['type'] }) for (const span of chunk.closing) { if (span.type === 'link' || span.type === 'image') continue - const openIndex = openSpans.findIndex(openSpan => - openSpan.type === span.type && openSpan.openOffset === span.offset - ) - if (openIndex === -1) continue - - ranges.push({ - type: span.type, - start: openSpans[openIndex].openOffset, - end: span.offset + span.length, - }) - openSpans.splice(openIndex, 1) + const index = open.findIndex(item => item.type === span.type && item.openOffset === span.offset) + if (index >= 0) { ranges.push({ type: span.type, start: open[index].openOffset, end: span.offset + span.length }); open.splice(index, 1) } } } - - for (const span of openSpans) { - if (span.openOffset < blockEnd) { - ranges.push({ type: span.type, start: span.openOffset, end: blockEnd }) - } - } - - return ranges + return [...ranges, ...open.filter(span => span.openOffset < end).map(span => ({ type: span.type, start: span.openOffset, end }))] } -function createMarksForRange(schema: Schema, spans: ClosedSpan[], markRanges: MarkRange[], start: number, end: number): Mark[] { +function createMarksForRange(schema: Schema, spans: ClosedSpan[], ranges: MarkRange[], start: number, end: number): Mark[] { const marks: Mark[] = [] - const activeTypes = new Set() - - for (const span of dedupeClosedSpans(spans)) { - if (span.type === 'image') continue - if (span.offset >= end || span.offset + span.length <= start) continue - - if (span.type === 'link') { - const href = sanitizeLinkHref(span.url) - if (href) marks.push(schema.marks.link.create({ href })) - } - } - - for (const range of markRanges) { - if (range.type === 'link' || range.type === 'image') continue - if (range.start >= end || range.end <= start) continue - - if (activeTypes.has(range.type)) continue - activeTypes.add(range.type) - const mark = createMarkForSpanType(schema, range.type) - if (mark) marks.push(mark) - } - + for (const span of spans) if (span.type === 'link' && span.offset < end && span.offset + span.length > start) { const href = sanitizeLinkHref(span.url); if (href && schema.marks.link) marks.push(schema.marks.link.create({ href })) } + for (const range of ranges) if (range.start < end && range.end > start) { const mark = createMarkForSpanType(schema, range.type); if (mark && !marks.some(item => item.type === mark.type)) marks.push(mark) } return marks } function createMarkForSpanType(schema: Schema, type: SpanType): Mark | null { - switch (type) { - case 'bold': - return schema.marks.strong.create() - case 'italic': - return schema.marks.em.create() - case 'code': - return schema.marks.code.create() - case 'strikethrough': - return schema.marks.strikethrough.create() - default: - return null - } + const name = type === 'bold' ? 'strong' : type === 'italic' ? 'em' : type + return name === 'code' || name === 'strikethrough' || name === 'strong' || name === 'em' ? schema.marks[name]?.create() ?? null : null } -function dedupeClosedSpans(spans: T[]): T[] { - const seen = new Set() - return spans.filter(span => { - const key = `${span.type}:${span.offset}:${span.length}:${'url' in span ? span.url : ''}:${'src' in span ? span.src : ''}` - if (seen.has(key)) return false - seen.add(key) - return true - }) -} - -function trimTrailingNewline(chunks: Chunk[]): Chunk[] { - if (chunks.length === 0) return chunks - const trimmed = [...chunks] - const last = trimmed[trimmed.length - 1] - if (!last.text.endsWith('\n')) return trimmed - - trimmed[trimmed.length - 1] = { - ...last, - text: last.text.slice(0, -1), - length: Math.max(0, last.length - 1), - } - return trimmed -} - -function createTextNodeOrNull(schema: Schema, text: string): ProseMirrorNode | null { - return text ? schema.text(text) : null -} - -function createPlainTextContent(schema: Schema, block: Chunk[]): ProseMirrorNode[] { - const text = trimTrailingNewline(block).map(chunk => chunk.text).join('') - return text ? [schema.text(text)] : [] -} - -function isTableCellBlockType(blockType: string | undefined): boolean { - return blockType === 'table_header_cell' || blockType === 'table_cell' -} +function dedupeClosedSpans(spans: T[]): T[] { const seen = new Set(); return spans.filter(span => { const key = `${span.type}:${span.offset}:${span.length}:${'url' in span ? span.url : ''}:${'src' in span ? span.src : ''}`; if (seen.has(key)) return false; seen.add(key); return true }) } +function trimTrailingNewline(chunks: Chunk[]): Chunk[] { const copy = [...chunks], last = copy.at(-1); if (!last?.text.endsWith('\n')) return copy; copy[copy.length - 1] = { ...last, text: last.text.slice(0, -1), length: Math.max(0, last.length - 1) }; return copy } +function createTextNodeOrNull(schema: Schema, text: string): ProseMirrorNode | null { return text ? schema.text(text) : null } +function fallbackParagraph(schema: Schema, chunks: Chunk[]): ProseMirrorNode { const text = trimTrailingNewline(chunks).map(chunk => chunk.text).join(''); return schema.nodes.paragraph.create(null, text ? schema.text(text) : null) } +function isTableCellBlockType(type: string | undefined): boolean { return type === 'table_header_cell' || type === 'table_cell' } diff --git a/demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts b/demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts index 4ed0819..5cfa677 100644 --- a/demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts +++ b/demo/svelte-demo/src/lib/prosemirror/stream-examples.integration.test.ts @@ -9,7 +9,8 @@ import { applyStreamingChunkToBuffer, buildDocFromChunks } from './stream-assemb const __filename = fileURLToPath(import.meta.url) const __dirname = dirname(__filename) const repoRoot = join(__dirname, '../../../../..') -const examplesDir = join(repoRoot, 'demo/svelte-demo/static/llm-streams-examples') +// Tests must not depend on generated Svelte static output. +const examplesDir = join(repoRoot, 'demo/llm-streams-examples') const wasmDir = join(repoRoot, 'demo/svelte-demo/static') type ParsedExample = { From 49a172e23e8ce0d2ee6f2a81cb809109d476b4a4 Mon Sep 17 00:00:00 2001 From: Dmitry Bondarenko Date: Thu, 16 Jul 2026 21:29:38 +0600 Subject: [PATCH 4/4] Updates documentation --- PLAN.md | 216 ------------------------------------------------------ README.md | 32 ++++++++ 2 files changed, 32 insertions(+), 216 deletions(-) delete mode 100644 PLAN.md diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 7d78391..0000000 --- a/PLAN.md +++ /dev/null @@ -1,216 +0,0 @@ -# Migrate svelte-demo rendering to ProseMirror (LIX-MDSP-20) - -## TODO - ProseMirror adjacency review findings (2026-07-14) - -The demo migration is architecturally aligned with Lixpi, but the following issues must be resolved before treating it as a promotion-ready reference implementation. The current Lixpi reference is `/home/dima/Desktop/lixpi/packages/lixpi/prosemirror/src/shared/`; its schema and bounded transaction APIs have evolved since the original plan was written. - -### 1. Preserve sibling and nested list topology (blocker) -`buildContentFromChunks` currently calls `flushLists(list.depth - 1)` before every list item. Because `flushLists` pops frames whose depth is greater than the supplied depth, a same-depth sibling flushes the current list and starts another list node. Nested siblings can likewise become multiple nested list nodes instead of items in one list. - -Fix the depth-stack transition so it: -- keeps the current frame for a sibling with the same depth and list type; -- flushes only deeper frames when returning to a shallower depth; -- flushes and replaces the current frame when the list type changes at the same depth; -- preserves ordered-list starts and attaches nested lists to the correct parent item. - -Add structural tests that assert one list contains multiple sibling `list_item` nodes, nested siblings share one nested list, mixed ordered/unordered transitions produce the intended separate lists, and returning from a nested list continues the original parent list. - -### 2. Define and implement the real Lixpi portability contract (blocker) -The statement that `buildContentFromChunks(schema, chunks)` runs against Lixpi's `createProseMirrorSchema(...)` unchanged is currently false: -- the demo defines `image` as inline, while current Lixpi defines `image` as a block node; -- current Lixpi schemas do not include `bullet_list`, `ordered_list`, `list_item`, or table node types; -- current Lixpi documents use composed roots and target nodes such as `documentTitle`, `aiChatThread`, and `aiResponseMessage`; -- Lixpi streaming publishes bounded transaction steps, while the demo replaces the entire document. - -Decide the canonical contract, then either extend Lixpi's schema builder with the portable list/table/image specs or make assembly capability-aware through an explicit adapter. Add a Lixpi-side adapter that replaces only the target response node content and returns transaction steps compatible with `HeadlessProseMirrorEngine`. Add contract tests using the actual `createProseMirrorSchema(DOCUMENT_TYPE.AI_CHAT_THREAD)` schema. Update this plan's portability claims to describe the resulting contract precisely. - -### 3. Make integration tests reproducible from a clean checkout -`stream-examples.integration.test.ts` reads fixtures from ignored/generated `demo/svelte-demo/static/llm-streams-examples`, but `pnpm --dir demo/svelte-demo run test` does not run `copy-llm-examples`. A clean checkout therefore lacks the test inputs. - -Read fixtures directly from tracked `demo/llm-streams-examples`, or add an explicit test preparation step. Verify the test command succeeds after removing generated Svelte/static output and without first running dev/build/prepack. - -### 4. Align image URL sanitization with the documented policy -`sanitizeImageSrc` accepts `/...`, `./...`, and `../...`, but rejects bare path-relative sources such as `image.png`, despite the plan allowing path-relative URLs. Define the allowed `data:image/*` MIME types rather than accepting every image subtype through one broad regex; explicitly decide whether SVG data URLs are permitted. - -Add tests for bare relative paths, query/hash-only edge cases, protocol-relative URLs, encoded or mixed-case script schemes, raster data URLs, SVG data URLs, malformed data URLs, and non-image data URLs. Unsafe sources must remain plain covered text. - -### 5. Assemble images whose closed span crosses chunk boundaries -`buildTextRuns` currently recognizes an image only when the complete image span exactly matches one run inside one chunk. A closed image span covering multiple chunks remains text instead of being replaced by one image node. - -Build image projection from absolute span ranges across the block, consume all covered text runs once the span closes, and insert exactly one sanitized image node. Keep open image spans as plain text until metadata becomes available. Add tests for contained, multi-chunk closing, unsafe multi-chunk, adjacent, and image-with-surrounding-text cases. - -### 6. Enforce schema validity and make malformed fallback observable -The malformed-table test only asserts that a document exists. `NodeType.create` can construct an empty table or row that violates its content expression without throwing, so the broad catch fallback may never run. - -Validate assembled block nodes or the completed document with `check()`/`validContent`, use `createAndFill` where appropriate, and fall back to a paragraph when partial metadata cannot form a valid node. Do not silently hide schema-contract programming errors: distinguish expected incomplete-stream fallback from unexpected assembly failures. Add assertions that malformed and partial states pass `doc.check()` and that valid partial table states retain all available cells. - -### 7. Complete real browser verification and correct stale completion claims -The current integration tests exercise parser-to-buffer-to-document projection, but they do not exercise `EditorView`, Svelte lifecycle, controls, debug columns, or rendered DOM. The plan also states that the dev server is running, which is transient environment state rather than a completed repository guarantee. - -Add browser E2E coverage for full play, pause/resume, single-step, reset mid-stream, replay after completion, switching examples mid-stream, backtrack correction, and rendered heading/list/code/table/task/strikethrough/image output. Assert the debug columns stay synchronized with the ProseMirror projection. Run this in a container/image that includes the selected browser runtime, then replace transient statements in the completed-findings section with reproducible commands and recorded outcomes. - -## Completed follow-up review findings (2026-07-08) - -The plan below has been implemented (new module in `demo/svelte-demo/src/lib/prosemirror/`, legacy rendering removed from `+page.svelte`). The post-implementation review items were applied inside the docker container `lixpi-markdown-stream-parser-demo`. - -### 1. Fixed the demo test run -Added `demo/svelte-demo/vitest.config.ts` so the demo runner includes only `src/**/*.test.ts` and excludes generated `.svelte-kit/**` output. `pnpm --dir demo/svelte-demo run test` now passes with the real ProseMirror unit/integration tests only. - -### 2. Restored strict demo typechecking -Restored `"strict": true` in `demo/svelte-demo/tsconfig.json` and kept `"allowImportingTsExtensions": true`. Strict-mode errors came from the root parser source imported by the demo, so nullable tree-sitter parse results now have explicit guards in `src/tree-sitter/inline-detection.ts` and `src/tree-sitter/segment-generator.ts`. `pnpm --dir demo/svelte-demo run check` now reports 0 errors. - -### 3. Styled open spans during streaming -`buildInlineContent` now tracks open non-link/image spans across block chunks and applies live marks for bold/italic/code/strikethrough until a matching closing span or the current buffer end. Link/image behavior stays closed-only for URL metadata safety. Added unit tests for a multi-chunk bold span and an unclosed bold span at the end of the current buffer. - -### 4. Reviewed out-of-scope edits -- Kept the type-only `Edit` cast in `src/tree-sitter-markdown-stream-parser.ts` because current `web-tree-sitter` types require `editPoint`/`editRange` even though the runtime accepts the existing edit shape. Removing it breaks strict demo typechecking. -- Reverted the root Vitest expansion so root tests stay scoped to `src/**/*.test.ts`; demo tests run through `pnpm --dir demo/svelte-demo run test`. - -### 5. Verified examples and controls path -The dev server is running at `http://localhost:5173` and responds with HTTP 200. The container has no Chromium/Firefox/Playwright/Puppeteer binary, so real browser automation could not be executed there. Added `stream-examples.integration.test.ts` to replay the real JSON token streams through the parser and ProseMirror assembly, covering heading/list/emphasis examples, code blocks, backtracking self-correction with no stale active text, strikethrough, tables, task-list metadata, reset, replay after completion, and switching examples at the buffer level. - -Everything else was verified as conforming: module layering and function signatures (including the `buildContentFromChunks(schema, chunks): Fragment` core required for Lixpi portability), schema node/mark parity with Lixpi, URL sanitization rules and plain-text fallback, editor wiring, legacy-code deletion in `+page.svelte` with the shared `isChunkBeforeBacktrack` used by both debug and renderer paths, dependency set, CSS, and `svelte-check` (0 errors). - -## Context - -The demo at `demo/svelte-demo` currently renders the parser's streaming output with an ad-hoc, hand-written rendering layer inside `src/routes/+page.svelte` (~898 lines): a reactive block-grouping state machine (`parsedBlocks`), manual open-span tracking, table reconstruction (`buildTableRows`), and a giant `{#each}/{#if}` markup tree with Tailwind span classes. This is the "legacy state-machine code" to replace. - -Goal: replace that rendering layer with ProseMirror, following the architecture of the Lixpi main repo (local checkout at `/home/dima/Desktop/lixpi`; reference files in `packages/lixpi/prosemirror/src/`): a framework-free ProseMirror module (schema + pure stream-assembly functions) driving an `EditorView` via transactions. - -**Key adaptation vs Lixpi:** Lixpi's `packages/lixpi/prosemirror/src/stream-assembly.ts` consumes the OLD parser segment shape (`{segment, styles[], type, isBlockDefining}`). This repo's tree-sitter parser emits a new offset-based `Chunk` shape (`src/tree-sitter/types.ts`): `{text, offset, length, block:{type, level?, language?, list?, table?}, opening/closing/contained spans, backtrackOffset?, recovery?}` wrapped in `StreamingChunk` (`START_STREAM | STREAMING | END_STREAM`). Lixpi's schema also lacks list/table nodes, which this parser emits. So we replicate the *architecture*, not the code verbatim. - -**User-approved decisions:** -- **Rebuild projection** strategy: keep a chunk buffer; on each chunk rebuild the whole doc via a pure `chunks → doc` function and dispatch one replace transaction. Backtracking = filter buffer; reset/replay = clear buffer. Demo-scale docs make O(n) rebuild imperceptible. -- **Location**: `demo/svelte-demo/src/lib/prosemirror/` (framework-free TS, promotable to a package later; repo stays single-package). -- Hand-written NodeSpecs for lists/tables (no `prosemirror-schema-list`/`prosemirror-tables` — those provide editing commands we don't need for a read-only view; Lixpi hand-writes all specs too). -- All commands run inside docker container `lixpi-markdown-stream-parser-demo`. - -## New files (all under `demo/svelte-demo/src/lib/prosemirror/`) - -### 1. `schema.ts` -Adapt Lixpi's `base-schema.ts` (`/home/dima/Desktop/lixpi/packages/lixpi/prosemirror/src/base-schema.ts`), extend with lists/tables, drop lixpi-only nodes. Read-only view ⇒ `parseDOM` optional. - -Nodes: -- `doc` (`block+`), `paragraph` (`inline*` → `['p', 0]`), `heading` (attr `level`, → `h1..h6`), `code_block` (attr `language`, `content:'text*'`, `marks:''`, `code:true`, → `['pre', {'data-language': language}, ['code', 0]]`), `blockquote` (`block+`), `text` -- `bullet_list` (`list_item+` → `['ul', 0]`), `ordered_list` (attr `order` → `['ol', {start}, 0]`), `list_item` (attr `task: null|{checked}`, `content:'block+'`, → `['li', {'data-task': 'checked'|'unchecked'}, 0]`; attribute omitted entirely for non-task items) -- `table` (`table_row+` → `['table', ['tbody', 0]]`), `table_row` (`(table_header_cell|table_cell)+` → `['tr', 0]`), `table_header_cell`/`table_cell` (attr `align`, `content:'inline*'`, → `['th'|'td', {style:'text-align: ...'}, 0]`) -- `image` — **inline** (`inline:true`, group `inline`, attrs `src`, `alt`) since the parser emits images as inline spans - -Marks (per Lixpi's `createStreamingMark` mapping): `strong`, `em`, `code`, `strikethrough` (→ ``), `link` (attr `href`, `inclusive:false`, render with `rel="noopener noreferrer"`). - -Security: parser span metadata is untrusted text. Before creating link marks or image nodes, sanitize URLs with a single helper used by stream assembly: -- Links: allow `http:`, `https:`, and `mailto:` only; reject empty, malformed, `javascript:`, `vbscript:`, and `data:` URLs. -- Images: allow `http:`, `https:`, root-relative/path-relative URLs, and safe `data:image/*` URLs only; reject protocol-relative URLs, scriptable URLs, and non-image data URLs. -- Rejected links/images render as their plain covered text, not as clickable links or image nodes. - -### 2. `stream-assembly.ts` -Pure, no DOM/Svelte. Import types from `../../../../../src/markdown-stream-parser.ts` (same relative-source style `+page.svelte` already uses). - -```ts -// backtrack semantics: if chunk.backtrackOffset set, drop buffered chunks with -// (offset + length) > backtrackOffset, then append (proven predicate, +page.svelte:180-188) -applyStreamingChunkToBuffer(buffer: Chunk[], chunk: Chunk): Chunk[] - -// shared predicate so editor buffer and +page.svelte debug parsedSegments cannot drift -isChunkBeforeBacktrack(chunk: Chunk, backtrackOffset: number): boolean - -sanitizeLinkHref(rawHref: string): string | null -sanitizeImageSrc(rawSrc: string): string | null - -// port of the parsedBlocks state machine (+page.svelte:382-441): boundaries on -// block.type change (except same-tableId cells), tableId change, heading level change, -// list newline heuristic; ADD: list depth/type change also starts a new group -groupChunksIntoBlocks(chunks: Chunk[]): Chunk[][] - -// slice chunk text at span boundaries (absolute UTF-16 offsets); active spans = -// carried-open ∪ opening ∪ covering-contained − closed; image spans → inline image -// node replacing covered text; others → marks (bold→strong, italic→em, code→code, -// strikethrough→strikethrough, link→link{href:sanitizedUrl}). Rejected unsafe URLs -// render as plain covered text. Skip empty runs. -buildInlineContent(schema: Schema, blockChunks: Chunk[]): Node[] - -// fold groups into nodes: paragraph/heading{level}/code_block{language}/ -// blockquote(paragraph)/list depth-stack (nested bullet_list/ordered_list, ordinal→order, -// task attr)/table grouping by tableId with rowIndex/columnIndex ordering + cellId dedup -// (port buildTableRows, +page.svelte:334-373). Empty buffer → doc(paragraph). -// try/catch per block → fallback plain paragraph so mid-stream states never throw. -buildDocFromChunks(schema: Schema, chunks: Chunk[]): Node -``` - -Notes: trim one trailing `\n` per non-code block group; tolerate partial table rows mid-stream; open link/image spans have no url/src until closed — render as plain text until closure (rebuild fixes retroactively). - -### 3. `editor.ts` -```ts -createStreamRenderer(mount: HTMLElement): StreamRenderer -// StreamRenderer: { handleStreamingChunk(parsed: StreamingChunk): void; reset(): void; destroy(): void } -``` -- `new EditorView(mount, { state: EditorState.create({schema, doc: emptyDoc}), editable: () => false })` -- Private non-reactive `buffer: Chunk[]`. On `STREAMING`: update buffer, `nextDoc = buildDocFromChunks(...)`, skip if `nextDoc.eq(state.doc)`, else `dispatch(tr.replaceWith(0, doc.content.size, nextDoc.content))` -- On `START_STREAM`: internal `reset()` (restart per stream, not append across runs). `console.warn` on `recovery.type === 'window_overflow'`. - -### 4. `ProseMirrorRenderer.svelte` -Mirrors Lixpi's `ProseMirror.svelte` mount pattern: `bind:this={mountEl}` div with `class="prose prose-sm max-w-none"`, `onMount` → `createStreamRenderer`, `onDestroy` → `destroy()`. Exports `handleStreamingChunk` / `reset` for `bind:this` use from the page. - -### 5. `prosemirror.css` -- `.ProseMirror { outline: none; word-wrap: break-word; }` (no global `pre-wrap`; trim newlines in assembly instead) -- Task-list checkboxes via `li[data-task="unchecked"]::before` (☐) / `li[data-task="checked"]::before` (☑) plus `list-style: none` on task items, code-block language badge via `pre[data-language]::after { content: attr(data-language) }`, table `th/td` borders to match old look. - -## Modified files - -### `demo/svelte-demo/src/routes/+page.svelte` -- Replace the `{#each parsedBlocks ...}` markup (lines ~596–829) with `` in the same card div. -- Subscription callback (~135–198): add `pmRenderer?.handleStreamingChunk(parsed)`; keep `parsedSegments` accumulation + backtrack filtering (feeds debug columns) and the backtrack `console.warn`. Replace the page's inline backtrack predicate (`seg.chunk.offset + seg.chunk.length <= chunk.backtrackOffset`, line ~183) with the shared `isChunkBeforeBacktrack` import so debug and renderer paths cannot drift. -- `resetParser()` (~302): add `pmRenderer?.reset()`. -- Delete dead code: `parsedBlocks` reactive block, `buildTableRows`, `getTableAlignClass`, `getTableCellAlignClass`, `isTableCellBlockType`, `getSpanClasses`, `hasCodeStyle`, `getActiveSpanTypes`, `updateOpenSpans` + `openSpans` state, `TableCellGroup`/`TableRowGroup`/`TableAlign` types. -- Keep: example picker, delay slider, play/pause/step/reset, and all debug columns (Current Token, Parsed Chunks JSON, raw tokens, concatenated txt). - -### `demo/svelte-demo/package.json` and `demo/svelte-demo/pnpm-lock.yaml` -Add direct dependencies actually imported by the implementation: `prosemirror-model`, `prosemirror-state`, and `prosemirror-view`. Add `prosemirror-transform` only if implementation code imports it directly. Add `vitest` as a devDependency plus a `test` script because the demo package currently has `check` but no test runner. - -### `demo/svelte-demo/src/lib/prosemirror/stream-assembly.test.ts` -Add focused unit tests for the pure assembly layer: backtrack filtering shared by buffer/debug paths, nested lists, ordered-list start attrs, task lists, tables, link/image URL sanitization and rejection, open/closed/contained spans, zero-length text skipping, and malformed mid-stream states falling back instead of throwing. Test style reference: `/home/dima/Desktop/lixpi/packages/lixpi/prosemirror/src/stream-assembly.test.ts`. - -### `demo/svelte-demo/src/app.css` -Add `@import './lib/prosemirror/prosemirror.css';` (Tailwind v4 CSS-first; typography plugin already loaded). - -## Implementation order - -1. `docker compose up -d`; then `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo add prosemirror-model prosemirror-state prosemirror-view` (add `prosemirror-transform` only if directly imported) -2. Add the demo test runner: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo add -D vitest`, then add a `"test": "vitest run"` script (non-watch mode; `pnpm add` will also trigger the demo's `postinstall` WASM copy — expected). -3. Implement `schema.ts` -4. Implement `stream-assembly.ts` (port grouping/table logic from `+page.svelte`) -5. Add `stream-assembly.test.ts` for the pure assembly layer. -6. Implement `editor.ts` + `ProseMirrorRenderer.svelte` + `prosemirror.css` + `app.css` import -7. Wire into `+page.svelte`, delete legacy rendering -8. Run tests and typecheck: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run test` and `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run check` - -## Verification (all inside the container) - -1. Unit tests from implementation step 5 pass: `docker exec lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run test`. -2. `docker exec -d lixpi-markdown-stream-parser-demo pnpm --dir demo/svelte-demo run dev` (predev regenerates the manifest; binds 0.0.0.0:5173 → host 5173). Open `http://localhost:5173`. -3. Exercise examples from `demo/svelte-demo/static/llm-streams-examples/`: - - headings/paragraphs/bold/italic/lists: `claude-3.5-1-quantum-physics`, `gpt-4.o-history-of-cats` - - fenced code blocks + language: `claude-3.7-happy-number-5-programs`, `gpt-4.5-cat-coding` - - **backtracking**: `claude-3.7-markdown-with-nested-code-block`, `test-error-recovery` — watch for `⚠️ BACKTRACK` console warning; PM doc must self-correct with no stale/duplicated text - - strikethrough: `test-strikethrough`; find table/task-list examples via `grep -l '|' static/llm-streams-examples/*.txt` and `grep -l '\- \['` -4. Controls: full play to END_STREAM; pause + single-step (doc updates chunk-by-chunk); reset mid-stream (doc clears); replay after completion (restarts, doesn't append); switch example mid-stream. -5. Debug columns still behave identically. -6. `pnpm --dir demo/svelte-demo run check` passes. - -## Portability to main Lixpi - -This parser is a tool for main Lixpi, which still consumes the deprecated legacy parser shape (`{segment, styles[], isBlockDefining}`). This demo's module is the reference implementation for the new `Chunk` shape → ProseMirror mapping, so preserve these guarantees during implementation: - -- `stream-assembly.ts` stays framework-free and schema-parameterized (no import of the demo schema instance) so it runs against Lixpi's `createProseMirrorSchema(...)` schemas unchanged. -- Node/mark names stay identical to Lixpi's `base-schema.ts` (`paragraph`, `heading`, `code_block`, `blockquote`, `strong`, `em`, `code`, `strikethrough`, `link`); new list/table NodeSpecs are plain spec objects portable into Lixpi's `node-specs.ts`. -- `buildDocFromChunks` must be a thin wrapper over a `buildContentFromChunks(schema, chunks): Fragment` core. The demo replaces the whole doc; main Lixpi will instead rebuild the content of a target node (e.g. `aiResponseMessage`) and emit one bounded ReplaceStep per update — compatible with its `HeadlessProseMirrorEngine` + step-publishing pipeline without whole-doc steps. -- URL sanitizers remain pure/exported for reuse in Lixpi's server-side assembler. - -## Risks / notes - -- Backtrack filter predicate `chunk.offset + chunk.length <= backtrackOffset` is the proven one from the legacy code — expose it once and reuse it for both the ProseMirror chunk buffer and debug `parsedSegments`. -- ProseMirror `toDOM` specs are nested arrays; strings like `table>tbody` are documentation shorthand only and must not be used as literal tag names. -- Sanitization belongs before ProseMirror mark/node creation; do not rely on DOM escaping to make `href`/`src` safe. Implement it as exported pure helpers so behavior is unit-testable without a browser. -- Legacy list-item newline grouping heuristic is imperfect; keep for parity plus the added depth/type boundary rule. -- ProseMirror text nodes can't be empty — skip zero-length runs. -- `+page.svelte` uses Svelte legacy syntax under Svelte 5 (`$:`/`on:click`) — keep new code consistent (onMount/bind:this), don't convert the page to runes. diff --git a/README.md b/README.md index 10548a9..ebfb2ff 100644 --- a/README.md +++ b/README.md @@ -413,6 +413,38 @@ flowchart LR Tree-sitter changed ranges and syntax errors identify source positions that may invalidate emitted output. Recovery selects a stored checkpoint, reconstructs generator state, and replays source from that checkpoint. Public offsets remain in rendered UTF-16 coordinates even though recovery decisions use raw source positions internally. +## ProseMirror Demo Integration + +The Svelte demo renders streaming chunks with a read-only ProseMirror `EditorView`. Its framework-free assembly layer lives in `demo/svelte-demo/src/lib/prosemirror/` and accepts a ProseMirror schema plus `Chunk[]`, so the projection logic can be reused with compatible schemas. + +The demo keeps a chunk buffer. For each `STREAMING` event, it removes buffered chunks whose rendered range crosses `backtrackOffset`, appends the replacement chunk, rebuilds the document, and replaces the editor content. `START_STREAM` and reset clear the buffer. A bounded recovery emits a console warning with the parser's recovery metadata. + +The demo schema represents paragraphs, headings, code blocks, blockquotes, ordered and unordered lists, task items, tables, inline images, and the `strong`, `em`, `code`, `strikethrough`, and `link` marks. List assembly preserves same-depth siblings, nested siblings, ordered-list start values, and the parent list when returning from a nested list. Table cells are grouped by the parser's table, row, and cell identifiers. + +Inline spans use rendered offsets across the entire block. Closed image spans replace their covered text with one image node even when the span crosses chunk boundaries; open image spans remain text until metadata arrives. The assembler validates completed documents and falls back to plain paragraphs for incomplete or unsupported structures. Unexpected assembly failures are logged before that fallback is used. + +### URL Policy in the Demo + +Link marks accept absolute `http:`, `https:`, and `mailto:` URLs. Image nodes accept absolute `http:` and `https:` URLs, root-relative and path-relative paths, and base64-encoded PNG, APNG, GIF, JPEG, WebP, and AVIF data URLs. Protocol-relative URLs, query- and fragment-only sources, scriptable schemes, non-image data URLs, malformed values, and SVG data URLs render as ordinary text. + +## Demo Verification + +The demo unit and integration tests run through the container: + +```bash +docker exec -it lixpi-markdown-stream-parser-demo \ + pnpm --dir demo/svelte-demo run test +``` + +The integration tests read the tracked JSON fixtures in `demo/llm-streams-examples`; development commands copy those fixtures into the demo's static directory for the example picker. + +Type-check the demo with: + +```bash +docker exec -it lixpi-markdown-stream-parser-demo \ + pnpm --dir demo/svelte-demo run check +``` + ## Development The repository's Docker service installs the root and demo dependencies. Start it from the repository root: