Skip to content

test: add generated line-mode lexer Markdown fixture#705

Merged
dowdiness merged 2 commits into
mainfrom
feat/701-line-mode-e2e
Jul 12, 2026
Merged

test: add generated line-mode lexer Markdown fixture#705
dowdiness merged 2 commits into
mainfrom
feat/701-line-mode-e2e

Conversation

@dowdiness

@dowdiness dowdiness commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Closes #701

Adds the compile-and-run acceptance fixture for the #561 line-mode lexer work stream:

  • Adds fixtures/line_lexer_regression to the MoonBit workspace.
  • Generates and commits line_lexer.g.mbt from annotated token/term source with loomgen --target native.
  • Runs the generated line lexer against real Markdown LF input and CRLF input.
  • Asserts exact token payloads, lengths, modes, positions, and EOF behavior.
  • Keeps regeneration deterministic with an executable regenerate.sh.

#561 is already closed by the implementation PRs; this closes its #701 end-to-end follow-up. #699 skeleton replacement remains separate.

Verification:

  • moon check --target native
  • moon test --target native loomgen (173/173)
  • moon test --target native (3427/3427)
  • moon test --target native fixtures/line_lexer_regression (2/2)
  • Regeneration SHA-256 unchanged

Bounded pre-PR reviewers: moonbit-reviewer PASS; reviewer PASS.

Summary by CodeRabbit

  • New Features

    • Added support for line-based lexer modes, including inline and line-start processing.
    • Added recognition of headings, list items, block quotes, text, blank lines, errors, and end-of-file tokens.
    • Added handling for LF, CR, and CRLF line endings with accurate token positions.
  • Tests

    • Added regression coverage for line-oriented Markdown tokenization and Windows-style line endings.
  • Chores

    • Added tooling to regenerate lexer output fixtures.

@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@dowdiness, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 50 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 55c1968c-c39d-4c90-88ca-16f149b001e3

📥 Commits

Reviewing files that changed from the base of the PR and between 144a8e4 and 29b37f7.

📒 Files selected for processing (1)
  • fixtures/line_lexer_regression/line_lexer_wbtest.mbt
📝 Walkthrough

Walkthrough

Adds an end-to-end fixture for generated line-mode lexers, including lexer contracts, generated and handwritten runtime code, regeneration tooling, package metadata, and tests for Markdown markers, token positions, EOF, and CRLF handling.

Changes

Line lexer regression fixture

Layer / File(s) Summary
Fixture contracts and generation setup
fixtures/line_lexer_regression/src/..., fixtures/line_lexer_regression/*_def.mbt, fixtures/line_lexer_regression/pkg.generated.mbti, fixtures/line_lexer_regression/moon.*, fixtures/line_lexer_regression/regenerate.sh
Defines lexer tokens, modes, terms, generated interfaces, package metadata, and the script used to regenerate the lexer output.
Line-mode lexer runtime
fixtures/line_lexer_regression/line_lexer.g.mbt, fixtures/line_lexer_regression/line_lexer_support.mbt
Implements line-start marker recognition, inline and blank-line tokenization, heading marker handling, and mode-based dispatch.
Tokenization regression coverage
fixtures/line_lexer_regression/line_lexer_wbtest.mbt
Checks Markdown token sequences and positions, EOF behavior, and CRLF-preserved text and offsets.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Test
  participant lex
  participant lex_line_start
  participant lex_inline
  Test->>lex: tokenize source with LineStart mode
  lex->>lex_line_start: dispatch current position
  lex_line_start->>lex_inline: process non-marker line content
  lex_line_start-->>Test: return marker LexStep and LineStart
  lex_inline-->>Test: return Text or BlankLine LexStep and next mode
Loading

Possibly related issues

Possibly related PRs

  • dowdiness/loom#29 — Introduces the LexStep API used by the fixture’s generated lexer contract.
  • dowdiness/loom#698 — Adds the line-pattern and line-mode generation behavior exercised by this fixture.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Linked Issues check ❓ Inconclusive Most acceptance criteria are covered, but workspace/build inclusion via moon.work cannot be verified because that file was filtered out. Provide the unfiltered moon.work diff or confirm the fixture is added to the workspace so moon check coverage can be verified.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed Title is concise and accurately describes the new generated line-mode lexer Markdown fixture.
Description check ✅ Passed Summary and verification are present and the purpose is clear, though the template's checklist section is omitted.
Out of Scope Changes check ✅ Passed The changes stay focused on the line-mode lexer fixture, generated artifacts, tests, and regeneration script.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/701-line-mode-e2e

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@fixtures/line_lexer_regression/moon.mod`:
- Line 9: Resolve the broken fixture reference in moon.mod by either adding the
missing README.md file under the same fixture directory or removing the readme
field. Ensure the fixture passes validation without referencing an absent file.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 343cbd56-eb54-42fa-92a3-61694192f3e2

📥 Commits

Reviewing files that changed from the base of the PR and between 6e4cd12 and 144a8e4.

⛔ Files ignored due to path filters (1)
  • moon.work is excluded by !**/*.work
📒 Files selected for processing (10)
  • fixtures/line_lexer_regression/lexmode_def.mbt
  • fixtures/line_lexer_regression/line_lexer.g.mbt
  • fixtures/line_lexer_regression/line_lexer_support.mbt
  • fixtures/line_lexer_regression/line_lexer_wbtest.mbt
  • fixtures/line_lexer_regression/moon.mod
  • fixtures/line_lexer_regression/moon.pkg
  • fixtures/line_lexer_regression/pkg.generated.mbti
  • fixtures/line_lexer_regression/regenerate.sh
  • fixtures/line_lexer_regression/src/line_lexer_src.mbt
  • fixtures/line_lexer_regression/token_def.mbt

"dowdiness/loom@0.1.0",
}

readme = "README.md"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if README.md exists alongside moon.mod
fd -t f "README.md" fixtures/line_lexer_regression/

Repository: dowdiness/loom

Length of output: 152


Add the missing README.md fixture or remove the readme field
fixtures/line_lexer_regression/moon.mod references README.md, but that file is absent from fixtures/line_lexer_regression/, so validation can fail.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@fixtures/line_lexer_regression/moon.mod` at line 9, Resolve the broken
fixture reference in moon.mod by either adding the missing README.md file under
the same fixture directory or removing the readme field. Ensure the fixture
passes validation without referencing an absent file.

@dowdiness dowdiness merged commit 6932b71 into main Jul 12, 2026
38 checks passed
@dowdiness dowdiness deleted the feat/701-line-mode-e2e branch July 12, 2026 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

loomgen: end-to-end test for generated line-mode lexer with real Markdown input (#561 follow-up)

1 participant