diff --git a/.gitignore b/.gitignore index 6232ac9..5f363d3 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,8 @@ build/ # Node modules node_modules/ .pnpm-store/ +.npm-cache/ +package-lock.json # Logs npm-debug.log* diff --git a/Dockerfile b/Dockerfile index b5791f5..8b848b3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -5,10 +5,22 @@ ARG NODE_VERSION=23 FROM node:${NODE_VERSION}-alpine # Install necessary packages -RUN apk add --update --no-cache curl - -# Install pnpm globally -RUN npm install -g pnpm +# tree-sitter needs C/C++ compiler (g++, make) and python3 +# cargo is needed to install tree-sitter-cli from source because npm install fails due to network/SSL issues with GitHub releases in this environment +RUN apk add --update --no-cache curl python3 make g++ cargo + +# Match the packageManager version declared in package.json. Newer pnpm versions +# no longer read the demo's pnpm.onlyBuiltDependencies setting. +RUN npm install -g pnpm@9.15.0 + +# Install tree-sitter-cli from source via cargo (bypassing GitHub releases download issue) +# Pin version to 0.25.0 to avoid dependency on libloading 0.9.0 which requires newer Rust than available in node:23-alpine +RUN cargo install --locked --version 0.25.0 tree-sitter-cli +ENV PATH="/root/.cargo/bin:${PATH}" + +# Set environment variables for C++ compilation +ENV CXXFLAGS="-std=c++20 -fexceptions" +ENV CXX="g++ -std=c++20 -fexceptions" # Set the working directory WORKDIR /usr/src/service diff --git a/MAINTAINING-DOCUMENTATION.md b/MAINTAINING-DOCUMENTATION.md new file mode 100644 index 0000000..75d1f85 --- /dev/null +++ b/MAINTAINING-DOCUMENTATION.md @@ -0,0 +1,207 @@ +--- +title: Maintaining Documentation +description: How to keep Lixpi's developer documentation accurate, readable, Markdoc-compatible, and easy to navigate as the product and architecture change. +--- + +# Maintaining Documentation + +Lixpi documentation should be useful to a human developer first. It can help agents too, but it should not read like agent scaffolding, a checklist dump, or a frozen snapshot of the repo tree. + +Use this guide when creating, moving, deleting, or reorganizing documentation. + +## Start by Discovering the Live Shape + +Do not assume folders, page names, or architecture boundaries are permanent. Before changing documentation: + +1. Read the docs index at the root of the documentation tree. +2. Check the generated docs-site navigation or list the Markdown files. +3. Read the pages around the area you are changing. +4. Read nearby source-code READMEs for the implementation area. +5. Fact-check behavior against the live code before repeating or rewriting it. + +The docs index is a map, not a contract. If the product shape changes, update the map to match the new shape. Avoid adding tiny "read this folder first" files whose only job is routing; put real guidance in this guide, in the relevant domain page, or in the docs index. + +## Document the Live System, Not a Timeline + +Hard rule: product and developer docs describe how the system works. They are +not a history record, migration diary, before/after report, or commentary on +what changed. + +Never frame normal documentation with phrases like: + +- "Current Responsibilities" +- "Current State" +- "now" +- "previously" +- "used to" +- "no longer" +- "old behavior" +- "new behavior" +- "deprecated path" +- "legacy path" + +Write the contract directly: + +- Use "Responsibilities", not "Current Responsibilities". +- Use "Input Flow", not "Current Flow". +- Use "Schema", "Runtime Wiring", "Files", "Transaction Meta", and similar direct headings. +- Say what the code does, not what it replaced. + +Mention removed or replaced behavior only in an explicit archive, migration +plan, changelog, or compatibility section where that history is the subject. If +a symbol remains for compatibility, document the live compatibility contract: +"parses `aiUserInput` and removes it in `appendTransaction()`", not "this used +to be the composer." + +## Keep the Structure Flexible + +Organize by stable product or engineering concerns, not by whatever filenames happen to exist today. Good documentation domains usually answer one of these questions: + +- What is this part of the product? +- What data does it persist? +- How does the runtime path work? +- How does a user flow move through the system? +- How is it deployed or operated? +- What conventions must implementation code follow? + +When the architecture changes, the documentation shape should change with it. Moving a page is fine. Splitting a page is fine. Deleting a page is fine if the content was moved or is false. + +Before deleting or replacing docs, compare against the existing version and account for every important concept: + +- Keep still-true product behavior. +- Drop false behavior. +- Keep history out of normal docs unless the page is explicitly an archive, migration plan, changelog, or compatibility note. +- Preserve useful rationale, constraints, and gotchas. +- Remove stale route-finding breadcrumbs. + +## Keep the Docs Honest + +Every factual claim should be easy to defend from live code, infrastructure, tests, or linked external source. + +Prefer durable statements over brittle ones: + +- Say "application tables" instead of freezing a table count. +- Say "configured by the deployment" instead of hardcoding a task count unless the exact number is the point. +- Say "configured default" when a setting can change. +- Say "computed and logged" if the code does not publish or persist something. +- Say "future split needs worker subscription code" if the boundary exists but the implementation is not wired. + +Avoid broad absolute claims unless the code enforces them: + +- "all" +- "every" +- "never" +- "guarantees" +- "only source" +- "production-ready" +- "no code changes" + +If the claim is a benchmark, capacity estimate, market comparison, legal/compliance statement, or vendor capability, either cite an up-to-date source or make it clear that it is a hypothesis that needs validation. + +## Write Like a Developer + +Use direct, natural language. Prefer the plain sentence that explains the thing. + +Avoid bureaucratic filler: + +- "source of truth" when "covers" or "explains" works +- "owned by" when "covered in" works +- "delta" when "what is specific to this page" works +- "leverage" when "use" works +- "robust solution" without saying what failure it handles + +Documentation should sound like a senior engineer explaining the system to another engineer: precise, calm, and not puffed up. + +## Keep Markdoc Compatibility + +These docs are Markdown that must render through the static Markdoc site. + +Use this authoring shape: + +```markdown +--- +title: Page Title +description: One sentence about what this page covers. +--- + +# Page Title +``` + +Frontmatter is not mandatory for the renderer, but human-facing pages should have it. + +Use standard Markdown whenever possible: + +- Relative links to documentation pages should point at `.md` files. +- Links to source code outside the documentation tree should be normal relative repo links. +- Use fenced code blocks with a language tag. +- Use Mermaid only inside fenced `mermaid` blocks. +- Use Markdoc callouts for notes, warnings, important details, and tips. + +```markdoc +{% callout type="warning" %} +Explain the risk and what to do about it. +{% /callout %} +``` + +Avoid: + +- Raw framework components. +- JSX/Svelte syntax. +- Inline HTML that Markdoc may parse differently from GitHub. +- Unclosed `{% callout %}` tags. +- Mermaid diagrams that depend on unsupported runtime plugins. +- Anchor links guessed by hand. Prefer linking to the page when you cannot verify a heading fragment. + +The docs build can validate heading IDs and anchor fragments when a human explicitly asks for that check. Do not run it as a default agent step. + +## Moving or Renaming Pages + +Do not delete documentation files silently. If cleanup, reverting agent edits, moving content, or replacing docs would delete files, ask the user to confirm the exact file path(s) first. If the user does not confirm, keep the files and report them as cleanup candidates. + +When reorganizing documentation: + +1. Map old pages to their new homes before deleting anything. +2. Search for old paths and old page titles across the repo. +3. Update links in docs, source comments, package READMEs, and tests. +4. Use static link review unless the user explicitly asks for the docs build. +5. If a source-shape test asserts a documentation path, update the test with the new path. + +Do not leave references to deleted pages. Keep links defensible from static review unless a requested docs build validates the rendered site. + +## Updating the Docs Index + +The docs index should help readers choose a starting point. It does not need to list every file forever. + +Keep the index useful by: + +- Linking to the main entry points for each active domain. +- Describing what each domain is for. +- Letting the generated site sidebar provide the exhaustive file inventory. +- Removing links to pages that became archives, implementation memory, or stale planning notes. + +When a domain changes shape, update the index at the same time as the pages. Do not add a separate "using this directory" page just to tell agents to inspect a folder. + +## Verification + +Do not run the docs build after documentation changes unless the user explicitly asks for it. Use static review by default. + +When a docs build is explicitly requested, run it through the documented Docker-only workflow. Never run `pnpm docs:build` on the host. + +If documentation changes a tested source assertion, run the relevant test +through the allowed project test command only when the user explicitly asks for +tests in the current thread. For web UI tests, use Dockerized Vitest. Do not use +`svelte-check`, browsers, screenshots, or manual visual inspection as +substitutes for permitted tests. + +## Before Calling It Done + +Check these: + +- The docs describe the live code as the actual system. +- Normal docs do not use before/after framing, "current" headings, or old-vs-new commentary. +- Historical behavior appears only when the page is explicitly an archive, migration plan, changelog, or compatibility note. +- Links work in the generated site, not only on GitHub. +- Page names and headings are human-readable. +- The docs index still gives a good starting point. +- No tiny routing-only guide was added. +- No brittle counts, capacity promises, or exact file inventories were added unless they are intentionally part of the subject. diff --git a/README.md b/README.md index c72f502..10548a9 100644 --- a/README.md +++ b/README.md @@ -1,485 +1,469 @@ -# @lixpi/markdown-stream-parser - -A library designed to incrementally parse Markdown text from a stream of tokens. +--- +title: Markdown Stream Parser +description: Incrementally parse streamed Markdown into render-agnostic text chunks, block context, inline spans, and recovery instructions. +--- -It's built to handle the ambiguities of LLM-generated streams, which often produce imperfect or invalid Markdown. It combines a finite state machine with regex patterns to determine the best match for each segment. +# Markdown Stream Parser -## This project is an **open-ended** research on how to incrementally parse LLM-streams. +`@lixpi/markdown-stream-parser` incrementally parses Markdown from token streams. It emits plain rendered text with block context, inline span metadata, and correction offsets that a consumer can apply to its output buffer. -### ⚠️ ***Please note that this project is in an early stage of development, so there are MANY bugs and missing features.*** +The parser uses the block and inline grammars from `tree-sitter-markdown`. It does not render HTML and does not depend on a UI framework. -### 🛑 All feature development is blocked by this *[research task](https://github.com/Lixpi/markdown-stream-parser/issues/5)* which would bring a complete re-imagining of the code. Stay tuned... +The project is under active development. Review [Supported Markdown](#supported-markdown) and [Limitations](#limitations) before using it in a production rendering path. -
+- [Live demo](https://markdown-stream-parser.lixpi.org) +- [Repository](https://github.com/Lixpi/markdown-stream-parser) -### DEMO: [markdown-stream-parser.lixpi.org](https://markdown-stream-parser.lixpi.org) +![Markdown stream parser demo](https://github.com/user-attachments/assets/6e3525f7-9082-46e9-853b-90ee20447fe5) -
+## Installation -![sample](https://github.com/user-attachments/assets/6e3525f7-9082-46e9-853b-90ee20447fe5) +Install the package with your package manager: +```bash +pnpm add @lixpi/markdown-stream-parser +``` -## Installation +```bash +npm install @lixpi/markdown-stream-parser +``` -NPM: ```bash -pnpm i @lixpi/markdown-stream-parser -npm i @lixpi/markdown-stream-parser yarn add @lixpi/markdown-stream-parser ``` -Or just clone the repository and import it directly from the source. +ES modules and CommonJS entry points are declared by the package: -### Importing - -The parser supports both ES6 module and CommonJS (Node.js) import styles. - -**ES6 import:** ```typescript import { MarkdownStreamParser } from '@lixpi/markdown-stream-parser' ``` -**CommonJS require:** -```typescript +```javascript const { MarkdownStreamParser } = require('@lixpi/markdown-stream-parser') ``` -Can be used on a backend or frontend, there's no rendering logic involved. +## WASM Assets +The parser needs three WASM files at runtime: -### Basic Concepts +- `tree-sitter.wasm` from `web-tree-sitter` +- `tree-sitter-markdown.wasm` +- `tree-sitter-markdown-inline.wasm` -- **Singleton Pattern:** - Use `MarkdownStreamParser.getInstance(instanceId)` to ensure one parser per logical stream/session. +The package does not copy these assets into an application. Copy or serve them as part of your deployment before creating a parser instance. -- **Parsing Lifecycle:** - - `startParsing()`: Begin parsing and set up subscriptions. - - `parseToken(chunk: string)`: Feed incoming text chunks. - - `stopParsing()`: Flush buffers, reset state, and notify listeners of stream end. +In a browser, `web-tree-sitter` resolves its runtime as `/tree-sitter.wasm`. The Markdown grammars default to `/tree-sitter-markdown.wasm` and `/tree-sitter-markdown-inline.wasm`. -- **Subscribing to Output:** - Use `subscribeToTokenParse(listener)` to receive parsed segments as soon as they are available. Returns an unsubscribe function. - The unsubscribe function takes no arguments. +Call `configureWasmPath()` before the first call to `getInstance()` when the grammar files use different paths: +```typescript +MarkdownStreamParser.configureWasmPath( + '/parsers/tree-sitter-markdown.wasm', + '/parsers/tree-sitter-markdown-inline.wasm' +) +``` -## How to Use +If the second argument is omitted, the inline path is derived by replacing `.wasm` with `-inline.wasm` in the Markdown grammar path. -There are several ways to use the parser. It is quite modular. You can initialize it in one place and consume the parsed stream elsewhere, thanks to the singleton pattern. +In Node.js, the grammar defaults are `./wasm/tree-sitter-markdown.wasm` and `./wasm/tree-sitter-markdown-inline.wasm`. Pass filesystem paths to `configureWasmPath()` when the assets are stored elsewhere. -## Subscribing to the Parser +## Quick Start -Before you can parse the stream, you must subscribe to the parser. If you do not subscribe in advance, the parser will likely stop and terminate before you receive the first segment. +Each logical stream uses an instance ID. `getInstance()` is asynchronous because the first call initializes `web-tree-sitter` and loads both grammars. -First import the parser and initialize it with an `instance-id`. (you can have as many parallel parsers as you want, just make sure to use different `instance-id`s) +Subscribe before calling `startParsing()`, feed string chunks with `parseToken()`, and finish with `stopParsing()` so buffered content is flushed. ```typescript -import { MarkdownStreamParser } from '@lixpi/markdown-stream-parser' +import { + MarkdownStreamParser, + type Chunk, +} from '@lixpi/markdown-stream-parser' -// Get a parser instance (singleton per ID) -const parser = MarkdownStreamParser.getInstance('session-1') -``` +const instanceId = 'response-42' +const parser = await MarkdownStreamParser.getInstance(instanceId) -#### Approach 1: The Simplest +let renderedText = '' -```typescript -// Subscribe to parsed output -parser.subscribeToTokenParse((parsedSegment, unsubscribe) => { - console.log(parsedSegment) // Happy little parsed segment - - // Clean up when the stream ends - if (parsedSegment.status === 'END_STREAM') { - unsubscribe() - MarkdownStreamParser.removeInstance('session-1') +const unsubscribe = parser.subscribeToTokenParse((event) => { + if (event.status === 'START_STREAM') { + renderedText = '' + return } -}) -``` -#### Approach 2: Customizable + if (event.status === 'END_STREAM') { + return + } -```typescript -// Subscribe to the parser service -const parserUnsubscribe = parser.subscribeToTokenParse(parsedSegment => { - console.log(parsedSegment) // Happy little parsed segment + applyChunk(event.chunk) }) -// When the stream has ended stop the parser to avoid issues and memory leaks. -// You can decide when to terminate the parser. -// For example, using your own logic or rely on the `parser.parsing` flag. -if (!parser.parsing) { - parserUnsubscribe() // Unsubscribe from the parser service - MarkdownStreamParser.removeInstance('session-1') // Dispose of the parser instance -} -``` - -## Parsing the Stream - -Regardless of which subscription method you choose, feeding the stream into the parser does not change. -Once the subscription to the parser is initialized, you can start parsing the stream. - -Again, this can be done in the same file or in a different part of your application. Just make sure to refer to the same parser `instance-id`. - -```typescript -import { MarkdownStreamParser } from '@lixpi/markdown-stream-parser' +function applyChunk(chunk: Chunk): void { + if (chunk.backtrackOffset !== undefined) { + renderedText = renderedText.slice(0, chunk.backtrackOffset) + } -// Get a parser instance (singleton per ID) -const parser = MarkdownStreamParser.getInstance('session-1') + renderedText += chunk.text +} -// Start the parser parser.startParsing() -// Your iterator function here -for await (const chunk of ["Hello", " ~~world~~", "!", " \n"]) { - parser.parseToken(chunk) +for (const token of ['## ', 'Hello ', '**world**', '\n']) { + parser.parseToken(token) } -// Make sure to stop the parser at the end of the stream. It will flush any remaining content from the buffer. parser.stopParsing() +unsubscribe() +MarkdownStreamParser.removeInstance(instanceId) ``` -The output is a series of objects containing the content of a parsed segment, the type of segment, and any possible inline styles. - -```javascript -{ - status: 'STREAMING', - segment: { - segment: 'Hello ', - styles: [], - type: 'paragraph', - isBlockDefining: true, // Indicates beginning of a new block, e.g. paragraph, heading, list etc... - isProcessingNewLine: true - } -} -{ - status: 'STREAMING', - segment: { - segment: 'world', - styles: [ 'strikethrough' ], - type: 'paragraph', - isBlockDefining: false, - isProcessingNewLine: false - } -} -{ - status: 'STREAMING', - segment: { - segment: '! ', - styles: [], - type: 'paragraph', - isBlockDefining: false, - isProcessingNewLine: false - } -} -{ status: 'END_STREAM' } -``` - +`getInstance()` returns the same parser for repeated calls with the same instance ID. Use different IDs for independent streams, and call `removeInstance()` when a stream no longer needs to be retained. -## Is that it? What am I supposed to do with that? +## Stream Lifecycle -Good question. You can use this stream to render styled content in your application in real time. Having a `segment type` and `inline styles` is enough to style it however you want. +The subscriber receives a discriminated union: -It will **always remain `render-agnostic`** - whatever you use to render your styled text is entirely up to you. +```typescript +type StreamingChunk = + | { status: 'START_STREAM' } + | { status: 'STREAMING'; chunk: Chunk } + | { status: 'END_STREAM' } +``` +The lifecycle is: -## Features +1. `subscribeToTokenParse(listener)` registers a listener and returns an unsubscribe function. +2. `startParsing()` resets accumulated parser state and emits `START_STREAM`. +3. `parseToken(chunk)` adds a string to the input buffer. Complete buffered segments may produce `STREAMING` events. +4. `stopParsing()` flushes buffered input and incomplete inline content, emits `END_STREAM`, and stops the session. +5. `removeInstance(instanceId)` stops and removes the retained parser instance. -- [x] Headers (`# H1`, `## H2`, etc.) -- [x] Paragraphs -- [x] Inline styles - - [x] Inline Italic (`*text*`) - - [x] Inline Bold (`**text**`) - - [x] Inline Bold & Italic (`***text***`) - - [x] Inline Strikethrough (`~~text~~`) - - [x] Inline Code (`` `code` ``) -- [x] Code Blocks (```` ```code-block``` ````) with language detection -- [ ] Blockquotes (`> quote`) [Iusse #2](https://github.com/Lixpi/markdown-stream-parser/issues/2) -- [ ] //TODO: PRIORITY: Ordered Lists (`1. item`) [Iusse #3](https://github.com/Lixpi/markdown-stream-parser/issues/3) -- [ ] //TODO: PRIORITY: Unordered Lists (`- item`, `* item`, `+ item`) *BLOCKED BY:* [Iusse #3](https://github.com/Lixpi/markdown-stream-parser/issues/3) -- [ ] //TODO: Task Lists (`- [ ] item`) *BLOCKED BY:* [Iusse #3](https://github.com/Lixpi/markdown-stream-parser/issues/3) -- [ ] //TODO: PRIORITY: Tables [Iusse #7](https://github.com/Lixpi/markdown-stream-parser/issues/7) -- [ ] //TODO: PRIORITY: Links (`[text](url)`) -- [ ] //TODO: PRIORITY: Images (`![alt](url)`) -- [ ] //TODO: Horizontal Rules (`---`, `***`, `___`) -- [ ] //TODO: Footnotes -- [ ] //TODO: HTML blocks -- [ ] //TODO: Escaping (`\*literal asterisks\*`) -- [ ] //TODO: Automatic Links (``) -- [ ] //TODO: Emoji (`:smile:`) -- [ ] //TODO: Superscript (`x^2^`) -- [ ] //TODO: Subscript (`H~2~O`) +Calling `parseToken()` before `startParsing()` returns an `Error`. Calling `startParsing()` while the parser is running leaves the active session in place. +Multiple listeners can subscribe to one parser instance. Each listener receives the event and an unsubscribe function as arguments: -## Running examples +```typescript +const unsubscribe = parser.subscribeToTokenParse((event, unsubscribeListener) => { + if (event.status === 'END_STREAM') { + unsubscribeListener() + } +}) +``` -To try out the parser with example streams, look inside the `llm-streams-examples` directory. This folder contains real LLM responses collected from various providers. Each response has two versions: +## Output Model -- `*.json`: An array of items used for streaming -- `*.txt`: The same stream combined into a single file +A `STREAMING` event contains a `Chunk`: -Having the `*.txt` version is handy for visual comparison and debugging the parser. +```typescript +type Chunk = { + text: string + offset: number + length: number + block: BlockContext + opening: OpenSpan[] + closing: ClosedSpan[] + contained: ClosedSpan[] + backtrackOffset?: number + recovery?: RecoveryInfo + original?: string +} +type RecoveryInfo = { + type: 'window_overflow' + windowSize: number + fullBacktrackOffset: number + appliedBacktrackOffset: number +} +``` -Inside the repository root dir run: +`text` contains rendered text with recognized Markdown syntax removed. `offset`, `length`, span positions, and `backtrackOffset` use UTF-16 code units in the rendered output coordinate space. This matches JavaScript string indexing and `String.prototype.slice()`. -1. Start the Docker container: - ```bash - docker compose up -d - ``` +`block` describes the surrounding block: -2. Run the debug parser inside the container: - ```bash - docker exec -it lixpi-markdown-stream-parser-demo pnpm run debug-parser --file= - ``` +```typescript +type BlockContext = { + type: + | 'paragraph' + | 'heading' + | 'code_block' + | 'list_item' + | 'table' + | 'table_row' + | 'table_header_cell' + | 'table_cell' + | 'blockquote' + level?: number + language?: string + list?: { + type: 'ordered' | 'unordered' + depth: number + marker: '-' | '+' | '*' | '.' | ')' + ordinal?: number + task?: { checked: boolean } + } + table?: { + tableId: string + rowIndex: number + columnIndex: number + cellId: string + align?: 'left' | 'center' | 'right' + } +} +``` - Replace `` with the relative path to any `.json` file. Examples: - - For files in `llm-streams-examples`: `--file=demo/llm-streams-examples/claude-3.5-1-quantum-physics.json` - - For manually created files: `--file=demo/llm-stream-examples-manually-simulated/long-consecutive-sequence.json` +`level` applies to headings. `language` contains the info string detected on a fenced code block. -3. **Creating custom test streams**: You can also create your own chunked streams from arbitrary text files using the `split-sample-into-chunks.ts` script: - ```bash - docker exec -it lixpi-markdown-stream-parser-demo pnpm run split-sample-into-chunks -- --file= --chunkSize= --outputPath= - ``` +`list` is present when the chunk is inside a list item, including nested blocks such as fenced code blocks contained by a list item. `depth` is zero-based: top-level items use `0`, and nested items use `1` or greater. - Example: - ```bash - docker exec -it lixpi-markdown-stream-parser-demo pnpm run split-sample-into-chunks -- --file=demo/llm-input-examples-raw-text/long-consecutive-sequence.txt --chunkSize=2 --outputPath=demo/llm-stream-examples-manually-simulated/long-consecutive-sequence.json - ``` +For unordered items, `marker` is the bullet character from the source: `-`, `+`, or `*`. For ordered items, `marker` is the delimiter only: `.` or `)`. The list number is exposed separately as `ordinal` when it is safely representable as a JavaScript number, so `10.` becomes `{ ordinal: 10, marker: '.' }`. -This will execute the parser against the selected example stream and print parsed segments to the console. +Task list items omit the checkbox marker and following space from rendered text. `[x]` and `[X]` produce `task: { checked: true }`; `[ ]` produces `task: { checked: false }`. -## Running tests +`table` is present when the chunk is inside a table cell. `tableId` identifies the enclosing table, `rowIndex` is zero-based with the header row at `0`, `columnIndex` is zero-based within the row, `cellId` is a stable `${tableId}:${rowIndex}:${columnIndex}` grouping key, and `align` reflects the parsed delimiter row when specified. -The project includes comprehensive test coverage with 187 tests across all core functionality. To run the tests: +Chunks are streamed at content boundaries, not at Markdown table-cell boundaries. One Markdown cell can produce multiple chunks. Consumers that need to rebuild a visual table should group chunks by `block.table.tableId`, then by `rowIndex`, then by `cellId`. -1. Start the Docker container: - ```bash - docker compose up -d - ``` +Compatibility note: header cell chunks now use `block.type === 'table_header_cell'`. Consumers that previously treated all table header content as `table_cell` should update that branch. -2. Run all tests: - ```bash - docker exec -it lixpi-markdown-stream-parser-demo pnpm test:run - ``` +When `includeRawStreamedToken` is enabled, `original` contains the raw Markdown source associated with the emitted chunk. It is separate from the rendered UTF-16 coordinate space. -3. Run tests in watch mode during development: - ```bash - docker exec -it lixpi-markdown-stream-parser-demo pnpm test - ``` +## Inline Spans -4. Run tests with coverage reporting: - ```bash - docker exec -it lixpi-markdown-stream-parser-demo pnpm test:coverage - ``` +Chunks and inline spans are independent. A span can be contained by one chunk or cross chunk boundaries. -**Note:** 3 tests are intentionally designed to fail to prove the existence of the known bug with long consecutive character sequences. All other tests should pass. +```typescript +type Span = + | { type: 'bold' } + | { type: 'italic' } + | { type: 'code' } + | { type: 'strikethrough' } + | { type: 'link'; url: string } + | { type: 'image'; src: string; alt?: string } + +type OpenSpan = { + type: Span['type'] + openOffset: number +} ---- +type ClosedSpan = Span & { + offset: number + length: number +} +``` +- `opening` lists spans that start in the chunk and remain open. +- `closing` lists spans that started in an earlier chunk and close in this chunk. +- `contained` lists complete spans represented within the chunk. -## How It Works +Consumers that maintain active span state can match a closing span to an opening span by `type` and by comparing `OpenSpan.openOffset` with `ClosedSpan.offset`. -#### The core of the parser is built around several key concepts: +```typescript +import type { Chunk, OpenSpan } from '@lixpi/markdown-stream-parser' -#### 1: Buffers +let openSpans: OpenSpan[] = [] -The parser uses a two-level buffering system: +function updateSpanState(chunk: Chunk): void { + if (chunk.backtrackOffset !== undefined) { + openSpans = openSpans.filter( + (span) => span.openOffset < chunk.backtrackOffset! + ) + } -1. **L1 Buffer (TokensStreamBuffer)**: Accumulates tokens until a complete segment (word, whitespace, punctuation) forms -2. **L2 Buffer (inside the Parser)**: Analyzes segments to detect markdown patterns and apply styles + openSpans.push(...chunk.opening) -This approach ensures style detection even when markdown syntax is split across multiple incoming chunks. + for (const closed of chunk.closing) { + const index = openSpans.findIndex( + (open) => + open.type === closed.type && + open.openOffset === closed.offset + ) -```mermaid -flowchart LR - A[Token] --> B[TokensStreamBuffer] - B --> C[Emit Complete Segment] - C --> D[MarkdownStreamParser] - D --> F((•)) + if (index !== -1) { + openSpans.splice(index, 1) + } + } +} ``` -#### 2: Blocks and Inline Elements +Links include their URL when closed. Images include `src` and may include `alt`. -Markdown consists of two fundamental components: +## Error Recovery -1. *Block-level elements* (paragraphs, headings, lists) which define document structure and cannot be nested within each other -2. *Inline elements* (bold, italic, code spans) which apply styling within blocks +Markdown structure can change as more source arrives. A line initially emitted as a paragraph can become part of a table or fenced code block, for example. -This distinction is central to our parsing approach, as it allows us to process markdown streams with predictable patterns. Block elements establish context, while inline styles modify content within that context. +When previously emitted output is affected, the first replacement chunk includes `backtrackOffset`. The consumer must: +1. Remove rendered output from `backtrackOffset` onward. +2. Remove derived block and span state at or after that offset. +3. Apply the replacement chunk and subsequent chunks in order. -#### 3: Routing aka State Machine +```typescript +if (chunk.backtrackOffset !== undefined) { + output = output.slice(0, chunk.backtrackOffset) + openSpans = openSpans.filter( + (span) => span.openOffset < chunk.backtrackOffset! + ) +} -The `MarkdownStreamParser` implements a state machine that processes text chunks from the `TokensStreamBuffer`. It utilizes pattern-matching evaluations based on regular expressions to determine the appropriate state transitions. +output += chunk.text +``` -The core of this architecture is the routing mechanism, which: +A correction may contain an empty `text` value when stale output must be deleted without replacement. Apply `backtrackOffset` even when `chunk.length` is zero. -1. Receives buffered segments from the stream processor -2. Executes pattern-matching evaluations against incoming content (partial or full matches) -3. Triggers corresponding actions based on matched patterns -4. Transitions the parser into the appropriate state (block-level or inline) +## Configuration -This consistent routing approach handles both high-level block elements (headings, paragraphs, code blocks) and inline styling (bold, italic, code spans) using the same underlying mechanism. +Pass configuration when creating an instance or merge it into an existing instance with `setConfig()`: -Below is a **simplified diagram** for parsing a Markdown stream containing a paragraph with inline styles (italic, bold, etc.). This example omits other block types for clarity. +```typescript +const parser = await MarkdownStreamParser.getInstance('response-42', { + includeRawStreamedToken: true, + windowSize: 500, +}) -```mermaid -stateDiagram-v2 - [*] --> idle - idle --> routing: startParsing() - routing --> processParagraph: paragraph detected - processParagraph --> emit: emit parsed segment - emit --> routing: next segment - - routing --> processInlineStylesGroup: inline style detected - processInlineStylesGroup --> emit: emit styled segment - emit --> routing: next segment - - routing --> handleMalformedSyntax: malformed style (e.g., missing closing marker + new line symbol that denotes beginning of a new block) - handleMalformedSyntax --> emit: emit unstyled segment - emit --> routing: next segment - - routing --> [*]: end of stream - - %% Notes: - %% - "emit" represents emitting a parsed segment to subscribers. - %% - The state machine loops through routing and processing states for each segment. - %% - If an inline style is opened but not closed before a new line, the unstyled content is flushed via emit and the state machine returns to routing. - %% - Only paragraph and inline style states are shown for simplicity. +parser.setConfig({ windowSize: 1000 }) +const config = parser.getConfig() ``` -Alternatively parser state transitions can be represented like this: +| Option | Type | Default | Behavior | +| --- | --- | --- | --- | +| `includeRawStreamedToken` | `boolean` | `false` | Adds the associated raw Markdown source to `chunk.original`. | +| `windowSize` | `number` | `undefined` | Requests a maximum correction distance in rendered UTF-16 code units. See the recovery limitation below. | -```mermaid -stateDiagram-v2 - direction LR - receiveToken --> buffer - buffer --> splitWords - splitWords --> routing +`windowSize` must be finite and greater than or equal to `0`; invalid values throw `RangeError`. `setConfig()` performs a shallow merge, so omitted properties retain their values. - routing --> processingHeader: header pattern detected - routing --> processingCodeBlock: code block detected - routing --> processingParagraph: default +## Supported Markdown - processingHeader --> emit: emit parsed segment - processingHeader --> routing: inline style detected +The parser handles these structures in its exercised parsing paths: - processingParagraph --> emit: emit parsed segment - processingParagraph --> routing: inline style detected +- Paragraphs and ATX headings (`#` through `######`) +- Fenced code blocks with language detection +- Ordered list items with `.` and `)` delimiters +- Unordered list items with `-`, `+`, and `*` markers +- Nested and loose list items +- Task list items with checked and unchecked state +- Bold, italic, bold-italic, strikethrough, and inline code spans +- Pipe tables with header-cell detection, delimiter suppression, alignment metadata, and stable per-cell grouping keys for covered table forms - %% Inline styles can only be entered from header or paragraph - routing --> processingItalicText: italic detected - routing --> processingBoldText: bold detected - routing --> processingBoldItalicText: bold+italic detected - routing --> processingStrikethroughText: strikethrough detected - routing --> processingInlineCode: inline code detected +Link and image span extraction is implemented, including URL and image metadata, but dedicated coverage is still needed for those paths. - processingItalicText --> emit: emit parsed segment - processingItalicText --> routing: next segment +### Lists - processingBoldText --> emit - processingBoldText --> routing +List marker syntax is removed from `text`. Consumers should use `block.list` to render bullets, ordered numbers, nesting, and task state instead of parsing the original Markdown source. - processingBoldItalicText --> emit - processingBoldItalicText --> routing +List metadata is attached to all chunks emitted inside a list item. A fenced code block inside a list keeps `block.type === 'code_block'` and also receives `block.list`, so consumers can preserve list indentation while rendering the nested block with its natural block type. - processingStrikethroughText --> emit - processingStrikethroughText --> routing +The parser removes structural list indentation from rendered output. Rendered list item text keeps meaningful content newlines, including blank lines in loose lists and the trailing newline at the end of an item. - processingInlineCode --> emit - processingInlineCode --> routing +These structures are incomplete or unsupported: - processingCodeBlock --> emit - processingCodeBlock --> routing +- Blockquote marker stripping and nested blockquotes +- Full coverage for every valid Markdown table shape +- Horizontal rules +- Footnotes +- HTML blocks +- Autolinks +- Emoji shortcodes +- Superscript and subscript extensions - emit --> routing: next segment - emit --> [*]: end of stream -``` +Escaped inline markers pass through the delimiter logic, but escaping behavior does not yet have complete feature coverage. -#### 4: Publish/Subscribe Pattern +## Limitations -The parser uses a *publish/subscribe* pattern to decouple the flow of data from its consumption. This design enables you to feed data into the parser and independently subscribe to a stream of parsed output. +### Recovery Window -```mermaid -flowchart TD - A[Input Tokens] -->|buffer| B(TokensStreamBuffer) - B -->|segment| C(MarkdownStreamParserStateMachine) - C -->|buffer| D(MarkdownStreamParser) - D -->|notify parsed segment| E[Subscribers] +`windowSize` is measured in rendered UTF-16 code units. When a complete correction would require replay before `lastEmittedOffset - windowSize`, the parser chooses the earliest stored checkpoint inside the configured window and attaches recovery metadata to the first replacement chunk: + +```typescript +if (chunk.recovery?.type === 'window_overflow') { + console.warn('Correction was truncated to the configured window', chunk.recovery) +} ``` -##### Benefits of this approach: -- Enables real-time, event-driven processing of Markdown streams -- Cleanly separates parsing logic from rendering or further processing -- Supports multiple independent subscribers per parser instance +`recovery.fullBacktrackOffset` is where complete replay would have started. `recovery.appliedBacktrackOffset` equals `chunk.backtrackOffset` and is the bounded offset consumers should apply. Output before `appliedBacktrackOffset` may remain stale because the parser did not ask the consumer to discard outside the configured window. -To receive parsed segments, simply subscribe to the parser before feeding data. Each subscriber is notified as soon as a new segment is available, and can unsubscribe at any time. +Leave `windowSize` undefined when a consumer requires complete recovery. -#### 5. Singleton Pattern +### Recovery Coverage -The parser utilizes a singleton pattern for instance management. Associate each logical stream with a unique `instanceId`. Use `MarkdownStreamParser.getInstance(instanceId)` to retrieve or create the parser for that stream, and `MarkdownStreamParser.removeInstance(instanceId)` for cleanup when the stream ends. +Recovery is covered for table and code-fence reclassification, rendered offsets, raw source output, bounded lookback behavior, and deletion-only corrections. Dedicated cases are still needed for: -This design enables *parallel processing* of multiple independent streams (e.g., concurrent user sessions or documents). By isolating each stream's state within its dedicated instance, the library ensures consistent state management. +- Inline delimiter replay with opening and closing spans -#### 6. Regex-driven Parsing +### Long Streams -This project uses a **regex-driven approach** for parsing segments, which while sometimes **debated** so far allowed to achieve the more stable result than previous attempts. -**There's still HUGE number of bugs. Refer to some examples in demo.** +Checkpoint history is copied as segments are emitted and searched linearly during recovery. Error detection walks only errored syntax subtrees, and replaced tree-sitter trees are released as the document changes. These paths can still accumulate disproportionate work as a document grows. -For each markup type, we define a set of regex rules to detect both full matches (e.g., single-word styles) and partial matches, which indicate the start or end of a style applied across multiple words. +Long uninterrupted input is emitted in bounded chunks by the token buffer, but output can still be delayed until the buffer reaches its internal threshold or receives whitespace. ---- +## Runtime Design +The parser maintains one block syntax tree and one inline parser per instance. Incoming strings pass through a token buffer before incremental parsing. Generated chunks carry rendered offsets, while internal state also retains source offsets for replay. -## Known issues +When incremental parsing succeeds, the parser compares changed ranges against the prior syntax tree, then releases the replaced tree. If parsing does not produce a replacement tree, the parser keeps the existing tree so the next chunk can continue from a valid parser state. -- **Delayed processing for extremely long sequences of characters without whitespace**: This is a downside of using the L1 buffer. Given the speed of modern LLMs, it's not a significant issue. The only time it becomes visually noticeable is when an LLM generates an **extremely long** regex, causing the output to freeze until receiving the final sequence. While this may be inconvenient, it's a rare edge case and not a high priority to fix. +```mermaid +flowchart LR + A[Input strings] --> B[Token buffer] + B --> C[Incremental block parse] + C --> D[Block and inline analysis] + D --> E[Rendered chunks] + C --> F[Changed ranges and errors] + F --> G[Checkpoint replay] + G --> E +``` -- **Inline styles for headings** are not implemented yet. Therefore, when a stream contains something like `### Title **with bold word**`, only the heading part will be detected. This should be fixed in the near future. +Tree-sitter changed ranges and syntax errors identify source positions that may invalidate emitted output. Recovery selects a stored checkpoint, reconstructs generator state, and replays source from that checkpoint. Public offsets remain in rendered UTF-16 coordinates even though recovery decisions use raw source positions internally. ---- +## Development +The repository's Docker service installs the root and demo dependencies. Start it from the repository root: -## Future Plans and Directions +```bash +docker compose up -d +``` -### Exploration of Alternative Parsing Architectures +Run the test suite in the service container: -While our current regex-based approach provides good results for LLM-generated Markdown streams, we recognize that established parsing libraries may offer additional benefits for long-term scalability and maintenance. We're evaluating: +```bash +docker exec -it lixpi-markdown-stream-parser-demo pnpm test:run +``` -- **Tree-sitter** - - A mature incremental parsing system adopted by Neovim and formerly by Atom - - Offers syntax recovery parsing with efficient incremental updates - - References: - - [Tree-sitter Documentation](https://tree-sitter.github.io/) - - [Tree-sitter repo](https://github.com/tree-sitter/tree-sitter) - - [Node.js Tree-sitter repo](https://github.com/tree-sitter/node-tree-sitter) - - [A Markdown parser for tree-sitter](https://github.com/tree-sitter-grammars/tree-sitter-markdown) +Run the package build: -- **Lezer** - - Modern incremental parser system developed by the authors of **ProseMirror** && **CodeMirror**... - - Designed specifically for editor use cases, also supports syntax recovery, though not sure if as advanced as `Tree-sitter` - - References: - - [Lezer Documentation](https://lezer.codemirror.net) - - [Lezer Markdown Grammar](https://github.com/lezer-parser/markdown) +```bash +docker exec -it lixpi-markdown-stream-parser-demo pnpm run build +``` -1. These parsers can effectively handle partial Markdown syntax across stream chunks -2. Our L1 buffer concept could be integrated with these parsers to maintain the current user experience +### Debug a Recorded Stream +Recorded streams live under `demo/llm-streams-examples`. JSON files preserve chunk boundaries; matching text files provide the combined Markdown for comparison. -**Community feedback and contributions are especially welcome regarding these architectural considerations, as diverse use cases will help inform the best approach.** -Please feel free to share your thoughts in **[discussions](https://github.com/Lixpi/markdown-stream-parser/discussions)**. +```bash +docker exec -it lixpi-markdown-stream-parser-demo \ + pnpm run debug-parser-tree-sitter \ + --file=claude-3.5-long-regex.json +``` +Create a chunked JSON stream from a text fixture: -## Contributions and Roadmap +```bash +docker exec -it lixpi-markdown-stream-parser-demo \ + pnpm run split-sample-into-chunks -- \ + --file=demo/llm-input-examples-raw-text/long-consecutive-sequence.txt \ + --chunkSize=2 \ + --outputPath=demo/llm-stream-examples-manually-simulated/long-consecutive-sequence.json +``` -- **Contributions:** - PRs and issues are *welcome*! +## Development Priorities +Recovery work focuses on a strict `windowSize` overflow contract and inline-span replay coverage. -- **Roadmap:** - - Support for the missing markdown features listed earlier. - - Performance optimizations - - Build an AST (abstract syntax tree) model to represent the parsed stream in memory +Scaling work focuses on stable-boundary checkpoints, indexed lookup, parser-internal checkpoint storage, tracked unresolved errors, and long-stream benchmarks. ---- +Markdown coverage work focuses on the incomplete structures listed in [Supported Markdown](#supported-markdown). + +## Contributing + +Bug reports, implementation proposals, and pull requests are welcome. Use [GitHub Discussions](https://github.com/Lixpi/markdown-stream-parser/discussions) for design questions and the repository issue tracker for reproducible defects. ## License diff --git a/demo/debug-scripts/char-streamer.ts b/demo/debug-scripts/char-streamer.ts deleted file mode 100644 index a242afe..0000000 --- a/demo/debug-scripts/char-streamer.ts +++ /dev/null @@ -1,75 +0,0 @@ -import fs from 'fs' - -// import { log, info, infoStr, warn, err } from './debug-tools.ts' - -import { MarkdownStreamParser } from '../../src/markdown-stream-parser.ts' - -// Parse CLI arguments -const args = process.argv.slice(2); -let DELAY = 0; -let filePath = ''; - -for (const arg of args) { - if (arg.startsWith('--interval=')) { - const val = parseInt(arg.split('=')[1], 10); - if (!isNaN(val)) DELAY = val; - } - if (arg.startsWith('--file=')) { - filePath = arg.split('=')[1]; - } -} - -if (!filePath) { - throw new Error('Missing required argument: --file='); -} - -const sourceFile = `/usr/src/service/${filePath}`; - -const markdownStreamParser = MarkdownStreamParser.getInstance(filePath) - -type JSONChunk = string | object; // Adjust as needed for your JSON structure - -async function* streamJSONinChunks(jsonArray: JSONChunk[]): AsyncGenerator { - const delay = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); - - // Iterate over each object in the json array - for (const item of jsonArray) { - if (item !== '') { - yield item; - await delay(DELAY); - } - } -} - - -;(async () => { - console.log('\n') - - const jsonContent: string = fs.readFileSync(sourceFile, { encoding: 'utf-8' }); - const parsedJson: JSONChunk[] = JSON.parse(jsonContent); - const textStream = streamJSONinChunks(parsedJson); - - markdownStreamParser.startParsing() // Parser has to be started before the stream is created - - for await (const chunk of textStream) { - markdownStreamParser.parseToken(chunk); - } - - markdownStreamParser.stopParsing() // At the end of the stream, it flushes any remaining content - -})() - - -type UnsubscribeFn = () => void; - -markdownStreamParser.subscribeToTokenParse( - (parsedSegment: any, unsubscribe: UnsubscribeFn) => { - console.log('parsedSegment', parsedSegment) // Happy little parsed segment - - // At the end of the stream, unsubscribe from the parser service - if (parsedSegment.status === 'END_STREAM') { - unsubscribe() - MarkdownStreamParser.removeInstance(filePath) - } - } -) diff --git a/demo/debug-scripts/debug-tools.ts b/demo/debug-scripts/debug-tools.ts deleted file mode 100644 index 8d5614c..0000000 --- a/demo/debug-scripts/debug-tools.ts +++ /dev/null @@ -1,51 +0,0 @@ -'use strict' - -import util from 'util' -import chalk from 'chalk' - -let safeInspect = (val) => { - try { - return util.inspect(val, { showHidden: false, depth: null, colors: true }) - } catch (err) { - return JSON.stringify(val, null, 2) - } -} - -// Iterating over the arguments and formatting them, if they are objects, using safeInspect, otherwise just returning them as they are -const formatArgs = (args) => args.map(arg => typeof arg === 'string' ? arg : safeInspect(arg)) - -export const log = (...args) => { - if (typeof args[0] === 'string' && args.length > 1) { - console.log(chalk.green(args[0]), ...formatArgs(args.slice(1))) - } else { - console.log(...formatArgs(args)) - } -} - -export const info = (...args) => { - if (typeof args[0] === 'string' && args.length > 1) { - console.info(chalk.blue(args[0]), ...formatArgs(args.slice(1))) - } else { - console.info(...formatArgs(args)) - } -} - -export const infoStr = (args: string[]) => { - console.info(args.join('')) -} - -export const warn = (...args) => { - if (typeof args[0] === 'string' && args.length > 1) { - console.warn(chalk.yellow(args[0]), ...formatArgs(args.slice(1))) - } else { - console.warn(...formatArgs(args)) - } -} - -export const err = (...args) => { - if (typeof args[0] === 'string' && args.length > 1) { - console.error(chalk.red(args[0]), ...formatArgs(args.slice(1))) - } else { - console.error(...formatArgs(args)) - } -} diff --git a/demo/llm-streams-examples/test-error-recovery.json b/demo/llm-streams-examples/test-error-recovery.json new file mode 100644 index 0000000..c77e834 --- /dev/null +++ b/demo/llm-streams-examples/test-error-recovery.json @@ -0,0 +1,12 @@ +[ + "## Table Test\n\n", + "| Col A | Col B |\n", + "| --- | --- |\n", + "| cell1 | cell2 |\n\n", + "## Code Fence Test\n\n", + "Some paragraph text here.\n", + "```python\n", + "x = 1\n", + "```\n\n", + "After code.\n" +] \ No newline at end of file diff --git a/demo/llm-streams-examples/test-error-recovery.txt b/demo/llm-streams-examples/test-error-recovery.txt new file mode 100644 index 0000000..c7d255b --- /dev/null +++ b/demo/llm-streams-examples/test-error-recovery.txt @@ -0,0 +1,14 @@ +## Table Test + +| Col A | Col B | +| --- | --- | +| cell1 | cell2 | + +## Code Fence Test + +Some paragraph text here. +```python +x = 1 +``` + +After code. diff --git a/demo/llm-streams-examples/test-strikethrough.json b/demo/llm-streams-examples/test-strikethrough.json new file mode 100644 index 0000000..68dbcd4 --- /dev/null +++ b/demo/llm-streams-examples/test-strikethrough.json @@ -0,0 +1,7 @@ +[ + "This is ", + "~~deleted~~", + " text and ", + "~~another strikethrough~~", + " here.\n" +] diff --git a/demo/llm-streams-examples/test-strikethrough.txt b/demo/llm-streams-examples/test-strikethrough.txt new file mode 100644 index 0000000..7883c38 --- /dev/null +++ b/demo/llm-streams-examples/test-strikethrough.txt @@ -0,0 +1 @@ +This is ~~deleted~~ text and ~~another strikethrough~~ here. diff --git a/demo/svelte-demo/package.json b/demo/svelte-demo/package.json index 2071606..2d1da56 100644 --- a/demo/svelte-demo/package.json +++ b/demo/svelte-demo/package.json @@ -14,7 +14,8 @@ "generate-manifest": "ts-node scripts/generate-llm-examples-manifest.ts", "copy-llm-examples": "ts-node scripts/copy-llm-examples.ts", "predev": "pnpm copy-llm-examples && pnpm generate-manifest", - "prebuild": "pnpm copy-llm-examples && pnpm generate-manifest" + "prebuild": "pnpm copy-llm-examples && pnpm generate-manifest", + "postinstall": "cp node_modules/web-tree-sitter/tree-sitter.wasm static/ && cp src/assets/parsers/tree-sitter-markdown.wasm static/" }, "files": [ "dist", @@ -52,7 +53,8 @@ "tailwindcss": "^4.0.0", "ts-node": "^10.9.2", "typescript": "^5.0.0", - "vite": "^6.2.6" + "vite": "^6.2.6", + "web-tree-sitter": "*" }, "keywords": [ "svelte" diff --git a/demo/svelte-demo/pnpm-lock.yaml b/demo/svelte-demo/pnpm-lock.yaml index 4909255..71cdb65 100644 --- a/demo/svelte-demo/pnpm-lock.yaml +++ b/demo/svelte-demo/pnpm-lock.yaml @@ -60,6 +60,9 @@ importers: vite: specifier: ^6.2.6 version: 6.3.3(@types/node@22.15.3)(jiti@2.4.2)(lightningcss@1.29.2) + web-tree-sitter: + specifier: '*' + version: 0.25.10 packages: @@ -1083,6 +1086,14 @@ packages: vite: optional: true + web-tree-sitter@0.25.10: + resolution: {integrity: sha512-Y09sF44/13XvgVKgO2cNDw5rGk6s26MgoZPXLESvMXeefBf7i6/73eFurre0IsTW6E14Y0ArIzhUMmjoc7xyzA==} + peerDependencies: + '@types/emscripten': ^1.40.0 + peerDependenciesMeta: + '@types/emscripten': + optional: true + whatwg-encoding@2.0.0: resolution: {integrity: sha512-p41ogyeMUrw3jWclHWTQg1k05DSVXPLcVxRTYsXUk+ZooOCZLcoYgPZ/HL/D/N+uQPOtcp1me1WhBEaX02mhWg==} engines: {node: '>=12'} @@ -1950,6 +1961,8 @@ snapshots: optionalDependencies: vite: 6.3.3(@types/node@22.15.3)(jiti@2.4.2)(lightningcss@1.29.2) + web-tree-sitter@0.25.10: {} + whatwg-encoding@2.0.0: dependencies: iconv-lite: 0.6.3 diff --git a/demo/svelte-demo/src/assets/parsers/tree-sitter-markdown.wasm b/demo/svelte-demo/src/assets/parsers/tree-sitter-markdown.wasm new file mode 100755 index 0000000..e6aee07 Binary files /dev/null and b/demo/svelte-demo/src/assets/parsers/tree-sitter-markdown.wasm differ diff --git a/demo/svelte-demo/src/routes/+page.svelte b/demo/svelte-demo/src/routes/+page.svelte index fdea488..e7cfa81 100644 --- a/demo/svelte-demo/src/routes/+page.svelte +++ b/demo/svelte-demo/src/routes/+page.svelte @@ -1,239 +1,529 @@
-
-

@lixpi/markdown-stream-parser demo

-

This is just a quick and dirty showcase of the @lixpi/markdown-stream-parser, the parser itself has nothing to do with rendering !!! Please keep that in mind...

-

This demo is entirely `vibe-coded`, while the parser is painstakingly created by a human being 👩‍💻 :)

-
+
+

+ @lixpi/markdown-stream-parser demo +

+

+ This is just a quick and dirty showcase of the + @lixpi/markdown-stream-parser, the + parser itself has nothing to do with rendering + !!! Please keep that in mind... +

+

+ This parser setup example is just an AI slop, its only goal is to + visually showcase the parser. + pls refer to the readme file for better instruction on how to user + parser API +

+
+
- {#if paused} - {:else} - {/if} - -
@@ -285,188 +594,235 @@ $: { // Reactive block to parse jsonContent when it changes
-
+

Parsed Stream

{#each parsedBlocks as block} -
- {#each block as seg} - {#if seg.segment?.type === 'header'} - {#if seg.segment?.level === 1} -

- {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {@const blockType = block[0]?.block.type} + {@const blockLevel = block[0]?.block.level} + {@const blockLanguage = block[0]?.block.language} + {@const hasTableCells = + blockType === "table_header_cell" || blockType === "table_cell"} + {@const tableRows = hasTableCells ? buildTableRows(block) : []} + +
+ {#if blockType === "heading"} + {#if blockLevel === 1} +

+ {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} {/if} -

- {:else if seg.segment?.level === 2} -

- {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {/each} +

+ {:else if blockLevel === 2} +

+ {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} {/if} -

- {:else if seg.segment?.level === 3} -

- {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {/each} +

+ {:else if blockLevel === 3} +

+ {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} {/if} -

- {:else if seg.segment?.level === 4} -

- {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {/each} +

+ {:else if blockLevel === 4} +

+ {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} {/if} -

- {:else if seg.segment?.level === 5} -
- {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {/each} +
+ {:else if blockLevel === 5} +
+ {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} {/if} -
- {:else if seg.segment?.level === 6} -
- {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {/each} +
+ {:else if blockLevel === 6} +
+ {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} {/if} -
- {:else} - - {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {/each} +

+ {:else} + + {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} {/if} - - {/if} - {:else if seg.segment?.type === 'codeBlock'} - 
{seg.segment?.segment}
- {:else if seg.segment?.type === 'blockQuote'} - {seg.segment?.segment} - {:else} - - {#if seg.segment?.styles?.length} - - {#if seg.segment.styles.includes('strikethrough')} - {seg.segment?.segment} - {:else if seg.segment.styles.includes('code')} - {seg.segment?.segment} - {:else} - {seg.segment?.segment} - {/if} - + {/each} + + {/if} + {:else if blockType === "code_block"} + 
{#each block as chunk}{chunk.text}{/each}
+ {#if blockLanguage} + {blockLanguage} + {/if} + {:else if blockType === "blockquote"} + + {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} {:else} - {seg.segment?.segment} + {chunk.text} + {/if} + {/each} + + {:else if blockType === "list_item"} + {@const list = block[0]?.block.list} + + + {#if list?.type === "ordered"} + {list.ordinal ?? ""}{list.marker} + {:else} + • {/if} - {/if} - {/each} + {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} + {:else} + {chunk.text} + {/if} + {/each} + + {:else if blockType === "table_header_cell" || blockType === "table_cell"} +
+ + {#each tableRows as row} + + {#each row.cells as cell} + + {#each cell.chunks as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} + {:else} + {chunk.text} + {/if} + {/each} + + {/each} + + {/each} +
+
+ {:else} + + + {#each block as chunk} + {@const styles = [ + ...chunk.contained.map((s) => s.type), + ...chunk.opening.map((s) => s.type), + ]} + {#if hasCodeStyle(styles)} + {chunk.text} + {:else} + {chunk.text} + {/if} + {/each} + + {/if}
{/each}
@@ -476,19 +832,33 @@ $: { // Reactive block to parse jsonContent when it changes

Current Token

- 
{JSON.stringify(currentToken, null, 2)}
+ 
{JSON.stringify(
+            currentToken,
+            null,
+            2,
+          )}

Parsed Chunks

{#if currentParsedChunks.length > 0} {#each currentParsedChunks as chunk, index}
-
{index + 1} of {currentParsedChunks.length}
- 
{JSON.stringify(chunk, null, 2)}
+
+ {index + 1} of {currentParsedChunks.length} +
+ 
{JSON.stringify(
+                  chunk,
+                  null,
+                  2,
+                )}
{/each} {:else} -
No parsed chunks for this token
+
+ No parsed chunks for this token +
{/if}
@@ -496,11 +866,16 @@ $: { // Reactive block to parse jsonContent when it changes

Raw array of streamed tokens

-
+
{#each jsonItems as item, index}
{JSON.stringify(item)}
@@ -511,7 +886,8 @@ $: { // Reactive block to parse jsonContent when it changes

Concatenated raw LLM output

- 
{txtContent}
+ 
{txtContent}
diff --git a/demo/svelte-demo/static/grammar.js b/demo/svelte-demo/static/grammar.js new file mode 100644 index 0000000..7db82c2 --- /dev/null +++ b/demo/svelte-demo/static/grammar.js @@ -0,0 +1,474 @@ +// This grammar only concerns the inline structure according to the CommonMark Spec +// (https://spec.commonmark.org/0.30/#inlines) +// For more information see README.md + +/// + +const common = require('../common/common'); + +// Levels used for dynmic precedence. Ideally +// n * PRECEDENCE_LEVEL_EMPHASIS > PRECEDENCE_LEVEL_LINK for any n, so maybe the +// maginuted of these values should be increased in the future +const PRECEDENCE_LEVEL_EMPHASIS = 1; +const PRECEDENCE_LEVEL_LINK = 10; +const PRECEDENCE_LEVEL_HTML = 100; + +// Punctuation characters as specified in +// https://github.github.com/gfm/#ascii-punctuation-character +const PUNCTUATION_CHARACTERS_REGEX = '!-/:-@\\[-`\\{-~'; + + +// !!! +// Notice the call to `add_inline_rules` which generates some additional rules related to parsing +// inline contents in different contexts. +// !!! +module.exports = grammar(add_inline_rules({ + name: 'markdown_inline', + + externals: $ => [ + // An `$._error` token is never valid and gets emmited to kill invalid parse branches. Concretely + // this is used to decide wether a newline closes a paragraph and together and it gets emitted + // when trying to parse the `$._trigger_error` token in `$.link_title`. + $._error, + $._trigger_error, + + // Opening and closing delimiters for code spans. These are sequences of one or more backticks. + // An opening token does not mean the text after has to be a code span if there is no closing token + $._code_span_start, + $._code_span_close, + + // Opening and closing delimiters for emphasis. + $._emphasis_open_star, + $._emphasis_open_underscore, + $._emphasis_close_star, + $._emphasis_close_underscore, + + // For emphasis we need to tell the parser if the last character was a whitespace (or the + // beginning of a line) or a punctuation. These tokens never actually get emitted. + $._last_token_whitespace, + $._last_token_punctuation, + + $._strikethrough_open, + $._strikethrough_close, + + // Opening and closing delimiters for latex. These are sequences of one or more dollar signs. + // An opening token does not mean the text after has to be latex if there is no closing token + $._latex_span_start, + $._latex_span_close, + + // Token emmited when encountering opening delimiters for a leaf span + // e.g. a code span, that does not have a matching closing span + $._unclosed_span + ], + precedences: $ => [ + // [$._strong_emphasis_star, $._inline_element_no_star], + [$._strong_emphasis_star_no_link, $._inline_element_no_star_no_link], + // [$._strong_emphasis_underscore, $._inline_element_no_underscore], + [$._strong_emphasis_underscore_no_link, $._inline_element_no_underscore_no_link], + [$.hard_line_break, $._whitespace], + [$.hard_line_break, $._text_base], + ], + // More conflicts are defined in `add_inline_rules` + conflicts: $ => [ + + [$._closing_tag, $._text_base], + [$._open_tag, $._text_base], + [$._html_comment, $._text_base], + [$._processing_instruction, $._text_base], + [$._declaration, $._text_base], + [$._cdata_section, $._text_base], + + [$._link_text_non_empty, $._inline_element], + [$._link_text_non_empty, $._inline_element_no_star], + [$._link_text_non_empty, $._inline_element_no_underscore], + [$._link_text_non_empty, $._inline_element_no_tilde], + [$._link_text, $._inline_element], + [$._link_text, $._inline_element_no_star], + [$._link_text, $._inline_element_no_underscore], + [$._link_text, $._inline_element_no_tilde], + + [$._image_description, $._image_description_non_empty, $._text_base], + // [$._image_description, $._image_description_non_empty, $._text_inline], + // [$._image_description, $._image_description_non_empty, $._text_inline_no_star], + // [$._image_description, $._image_description_non_empty, $._text_inline_no_underscore], + + [$._image_shortcut_link, $._image_description], + [$.shortcut_link, $._link_text], + [$.link_destination, $.link_title], + [$._link_destination_parenthesis, $.link_title], + + [$.wiki_link, $._inline_element], + [$.wiki_link, $._inline_element_no_star], + [$.wiki_link, $._inline_element_no_underscore], + [$.wiki_link, $._inline_element_no_tilde], + ], + extras: $ => [], + + rules: { + inline: $ => seq(optional($._last_token_whitespace), $._inline), + + ...common.rules, + + + // A lot of inlines are defined in `add_inline_rules`, including: + // + // * collections of inlines + // * emphasis + // * textual content + // + // This is done to reduce code duplication, as some inlines need to be parsed differently + // depending on the context. For example inlines in ATX headings may not contain newlines. + + code_span: $ => seq( + alias($._code_span_start, $.code_span_delimiter), + repeat(choice($._text_base, '[', ']', $._soft_line_break, $._html_tag)), + alias($._code_span_close, $.code_span_delimiter) + ), + + latex_block: $ => seq( + alias($._latex_span_start, $.latex_span_delimiter), + repeat(choice($._text_base, '[', ']', $._soft_line_break, $._html_tag)), + alias($._latex_span_close, $.latex_span_delimiter), + ), + + // Different kinds of links: + // * inline links (https://github.github.com/gfm/#inline-link) + // * full reference links (https://github.github.com/gfm/#full-reference-link) + // * collapsed reference links (https://github.github.com/gfm/#collapsed-reference-link) + // * shortcut links (https://github.github.com/gfm/#shortcut-reference-link) + // + // Dynamic precedence is distributed as granular as possible to help the parser decide + // while parsing which branch is the most important. + // + // https://github.github.com/gfm/#links + _link_text: $ => prec.dynamic(PRECEDENCE_LEVEL_LINK, choice( + $._link_text_non_empty, + seq('[', ']') + )), + _link_text_non_empty: $ => seq('[', alias($._inline_no_link, $.link_text), ']'), + shortcut_link: $ => prec.dynamic(PRECEDENCE_LEVEL_LINK, $._link_text_non_empty), + full_reference_link: $ => prec.dynamic(2 * PRECEDENCE_LEVEL_LINK, seq( + $._link_text, + $.link_label + )), + collapsed_reference_link: $ => prec.dynamic(PRECEDENCE_LEVEL_LINK, seq( + $._link_text, + '[', + ']' + )), + inline_link: $ => prec.dynamic(PRECEDENCE_LEVEL_LINK, seq( + $._link_text, + '(', + repeat(choice($._whitespace, $._soft_line_break)), + optional(seq( + choice( + seq( + $.link_destination, + optional(seq( + repeat1(choice($._whitespace, $._soft_line_break)), + $.link_title + )) + ), + $.link_title, + ), + repeat(choice($._whitespace, $._soft_line_break)), + )), + ')' + )), + + wiki_link: $ => prec.dynamic(2 * PRECEDENCE_LEVEL_LINK, seq( + '[', '[', + alias($._wiki_link_destination, $.link_destination), + optional(seq( + '|', + alias($._wiki_link_text, $.link_text) + )), + ']', ']' + ) + ), + + _wiki_link_destination: $ => repeat1(choice( + $._word, + common.punctuation_without($, ['[',']', '|']), + $._whitespace, + )), + + _wiki_link_text: $ => repeat1(choice( + $._word, + common.punctuation_without($, ['[',']']), + $._whitespace, + )), + + // Images work exactly like links with a '!' added in front. + // + // https://github.github.com/gfm/#images + image: $ => choice( + $._image_inline_link, + $._image_shortcut_link, + $._image_full_reference_link, + $._image_collapsed_reference_link + ), + _image_inline_link: $ => prec.dynamic(PRECEDENCE_LEVEL_LINK, seq( + $._image_description, + '(', + repeat(choice($._whitespace, $._soft_line_break)), + optional(seq( + choice( + seq( + $.link_destination, + optional(seq( + repeat1(choice($._whitespace, $._soft_line_break)), + $.link_title + )) + ), + $.link_title, + ), + repeat(choice($._whitespace, $._soft_line_break)), + )), + ')' + )), + _image_shortcut_link: $ => prec.dynamic(3 * PRECEDENCE_LEVEL_LINK, $._image_description_non_empty), + _image_full_reference_link: $ => prec.dynamic(PRECEDENCE_LEVEL_LINK, seq($._image_description, $.link_label)), + _image_collapsed_reference_link: $ => prec.dynamic(PRECEDENCE_LEVEL_LINK, seq($._image_description, '[', ']')), + _image_description: $ => prec.dynamic(3 * PRECEDENCE_LEVEL_LINK, choice($._image_description_non_empty, seq('!', '[', prec(1, ']')))), + _image_description_non_empty: $ => seq('!', '[', alias($._inline, $.image_description), prec(1, ']')), + + // Autolinks. Uri autolinks actually accept protocolls of arbitrary length which does not + // align with the spec. This is because the binary for the grammar gets to large if done + // otherwise as tree-sitters code generation is not very concise for this type of regex. + // + // Email autolinks do not match every valid email (emails normally should not be parsed + // using regexes), but this is how they are defined in the spec. + // + // https://github.github.com/gfm/#autolinks + uri_autolink: $ => /<[a-zA-Z][a-zA-Z0-9+\.\-][a-zA-Z0-9+\.\-]*:[^ \t\r\n<>]*>/, + email_autolink: $ => + /<[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*>/, + + // Raw html. As with html blocks we do not emit additional information as this is best done + // by a proper html tree-sitter grammar. + // + // https://github.github.com/gfm/#raw-html + _html_tag: $ => choice($._open_tag, $._closing_tag, $._html_comment, $._processing_instruction, $._declaration, $._cdata_section), + _open_tag: $ => prec.dynamic(PRECEDENCE_LEVEL_HTML, seq('<', $._tag_name, repeat($._attribute), repeat(choice($._whitespace, $._soft_line_break)), optional('/'), '>')), + _closing_tag: $ => prec.dynamic(PRECEDENCE_LEVEL_HTML, seq('<', '/', $._tag_name, repeat(choice($._whitespace, $._soft_line_break)), '>')), + _tag_name: $ => seq($._word_no_digit, repeat(choice($._word_no_digit, $._digits, '-'))), + _attribute: $ => seq(repeat1(choice($._whitespace, $._soft_line_break)), $._attribute_name, repeat(choice($._whitespace, $._soft_line_break)), '=', repeat(choice($._whitespace, $._soft_line_break)), $._attribute_value), + _attribute_name: $ => /[a-zA-Z_:][a-zA-Z0-9_\.:\-]*/, + _attribute_value: $ => choice( + /[^ \t\r\n"'=<>`]+/, + seq("'", repeat(choice($._word, $._whitespace, $._soft_line_break, common.punctuation_without($, ["'"]))), "'"), + seq('"', repeat(choice($._word, $._whitespace, $._soft_line_break, common.punctuation_without($, ['"']))), '"'), + ), + _html_comment: $ => prec.dynamic(PRECEDENCE_LEVEL_HTML, seq( + '' + )), + _processing_instruction: $ => prec.dynamic(PRECEDENCE_LEVEL_HTML, seq( + '' + )), + _declaration: $ => prec.dynamic(PRECEDENCE_LEVEL_HTML, seq( + /']), + ))), + '>' + )), + _cdata_section: $ => prec.dynamic(PRECEDENCE_LEVEL_HTML, seq( + '' + )), + + // A hard line break. + // + // https://github.github.com/gfm/#hard-line-breaks + hard_line_break: $ => seq(choice('\\', $._whitespace_ge_2), $._soft_line_break), + _text: $ => choice($._word, common.punctuation_without($, []), $._whitespace), + + // Whitespace is divided into single whitespaces and multiple whitespaces as wee need this + // information for hard line breaks. + _whitespace_ge_2: $ => /\t| [ \t]+/, + _whitespace: $ => seq(choice($._whitespace_ge_2, / /), optional($._last_token_whitespace)), + + // Other than whitespace we tokenize into strings of digits, punctuation characters + // (handled by `common.punctuation_without`) and strings of any other characters. This way the + // lexer does not have to many different states, which makes it a lot easier to make + // conflicts work. + _word: $ => choice($._word_no_digit, $._digits), + _word_no_digit: $ => new RegExp('[^' + PUNCTUATION_CHARACTERS_REGEX + ' \\t\\n\\r0-9]+(_+[^' + PUNCTUATION_CHARACTERS_REGEX + ' \\t\\n\\r0-9]+)*'), + _digits: $ => /[0-9][0-9_]*/, + _soft_line_break: $ => seq($._newline_token, optional($._last_token_whitespace)), + + _inline_base: $ => prec.right(repeat1(choice( + $.image, + $._soft_line_break, + $.backslash_escape, + $.hard_line_break, + $.uri_autolink, + $.email_autolink, + $.entity_reference, + $.numeric_character_reference, + (common.EXTENSION_LATEX ? $.latex_block : choice()), + $.code_span, + alias($._html_tag, $.html_tag), + $._text_base, + common.EXTENSION_TAGS ? $.tag : choice(), + $._unclosed_span, + ))), + _text_base: $ => choice( + $._word, + common.punctuation_without($, ['[', ']']), + $._whitespace, + '