feat(loomgen): #loom.line_pattern annotation for line-oriented block-level token production#698
Conversation
…production (#561) Adds #loom.line_pattern (on token variants) and #loom.line_mode (on #loom.lexmode term variants) annotations for generating line-mode lexer functions. Each #loom.line_mode mode gets a generated lex function that matches #loom.line_pattern regexes against the current line (pos→newline) and produces tokens via #loom.custom_lex payload extraction. New: - loomgen/emit_line_lexer.mbt — line-mode lexer code generation - loomgen/fixtures/line_pattern_fixture.mbt — test fixture Modified: - parse_annotations.mbt — parse/validate line_pattern + line_mode - emit_lexer.mbt — exclude line_pattern tokens from char-level lexer - emit_syntax_kind.mbt — handle None-role line_pattern tokens in kind entries - emit_token_impls.mbt — handle None-role line_pattern tokens in Show/ToRawKind - main.mbt — --line-lexer CLI option, preflight, and write Tests: 161 loomgen tests, 3315 project-wide — all pass.
|
Warning Review limit reached
Next review available in: 18 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughAdds ChangesLine-pattern lexer
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant LineLexerGenerator
participant EnumMetadata
participant GeneratedLineLexer
CLI->>LineLexerGenerator: request --line-lexer output
LineLexerGenerator->>EnumMetadata: read line_pattern and line_mode annotations
EnumMetadata-->>LineLexerGenerator: return token and mode metadata
LineLexerGenerator-->>CLI: return generated lexer content
CLI->>GeneratedLineLexer: write line_lexer.g.mbt
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
loomgen/emit_lexer.mbt (1)
57-70: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winLine-pattern
custom_lexfunctions leak into character-level lexer hooks.The first loop in
collect_lexer_inputsiterates over all variants (includingline_patternones) and adds everycustom_lexfunction name tocustom_lex_fns. That array is returned unfiltered at line 154, soemit_lexer(lines 257–269) emits character-level hooks for line-pattern custom_lex functions — calling them with(source, pos)instead of the(source, pos, len)signature used byemit_line_lexer, and expecting aLexStep[T]return instead of(LexStep[T], LexMode). This produces a type error in the generated character-level lexer whenever both--lexerand--line-lexerare passed and aline_patternvariant carries#loom.custom_lex.The second loop (lines 72–76) already skips
line_patternvariants; the first loop should do the same.🐛 Proposed fix: skip `line_pattern` variants in the first loop
for v in variants { + // `#loom.line_pattern` variants are produced by the line-mode lexer, not the + // character-level lexer — skip them entirely (mirrors the second loop). + if v.line_pattern is Some(_) { + continue + } match v.role { Some(Ident) => if v.custom_lex is None { ident_count = ident_count + 1 } Some(Keyword(kw)) => keywords.push((kw, v.name)) _ => () } if v.custom_lex is Some(fn_name) { custom_lex_fns.push(fn_name) } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@loomgen/emit_lexer.mbt` around lines 57 - 70, Update the first variant loop in collect_lexer_inputs so custom_lex_fns only collects functions from non-line_pattern variants, matching the filtering already used by the second loop. Preserve ident counting and keyword collection behavior, and ensure line-pattern custom lex functions remain available only to emit_line_lexer rather than character-level hooks.
🧹 Nitpick comments (1)
loomgen/emit_lexer_wbtest.mbt (1)
795-843: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider adding a negative test for the custom_lex requirement on line_pattern variants with constructor arguments.
The upstream validation rejects
#loom.line_patternvariants that have constructor arguments but no#loom.custom_lex(parse_annotations.mbt:370-377). The fixture covers the positive case (HeadingMarker(Int)withlex_heading_marker), but no test asserts the rejection whencustom_lexis missing. Since custom lex for payload extraction is a key PR feature, a negative test would close this coverage gap.🧪 Suggested test
///| test "parse_annotations rejects `#loom.line_pattern` with constructor args but no custom_lex" { let src = #|`#loom.token` #|pub(all) enum T { #| `#loom.line_pattern`("#{1,6} ") #| HeadingMarker(Int) #| `#loom.error` #| Error(String) #| `#loom.eof` #| E #|} derive(Eq) match parse_annotations(src) { Err(msg) => inspect(msg.contains("`#loom.custom_lex`"), content="true") Ok(_) => abort("expected rejection") } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@loomgen/emit_lexer_wbtest.mbt` around lines 795 - 843, Add a negative test alongside the existing parse_annotations validation tests for a `#loom.line_pattern` variant with constructor arguments but no `#loom.custom_lex`. Use a payload variant such as HeadingMarker(Int), include the required error and EOF variants, and assert parse_annotations returns Err with a message containing "`#loom.custom_lex`".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@loomgen/emit_line_lexer.mbt`:
- Around line 54-87: Update emit_mode_line_lexer to emit each line-pattern
candidate only when its lexmode_name matches the enclosing mode. In the nullary
token branch, return the candidate’s declared transition target when present
instead of always returning mode_name; preserve mode_name as the fallback when
no target is declared.
---
Outside diff comments:
In `@loomgen/emit_lexer.mbt`:
- Around line 57-70: Update the first variant loop in collect_lexer_inputs so
custom_lex_fns only collects functions from non-line_pattern variants, matching
the filtering already used by the second loop. Preserve ident counting and
keyword collection behavior, and ensure line-pattern custom lex functions remain
available only to emit_line_lexer rather than character-level hooks.
---
Nitpick comments:
In `@loomgen/emit_lexer_wbtest.mbt`:
- Around line 795-843: Add a negative test alongside the existing
parse_annotations validation tests for a `#loom.line_pattern` variant with
constructor arguments but no `#loom.custom_lex`. Use a payload variant such as
HeadingMarker(Int), include the required error and EOF variants, and assert
parse_annotations returns Err with a message containing "`#loom.custom_lex`".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9dc2de76-6835-416f-ba33-c2f2cf6477c6
📒 Files selected for processing (8)
loomgen/emit_lexer.mbtloomgen/emit_lexer_wbtest.mbtloomgen/emit_line_lexer.mbtloomgen/emit_syntax_kind.mbtloomgen/emit_token_impls.mbtloomgen/fixtures/line_pattern_fixture.mbtloomgen/main.mbtloomgen/parse_annotations.mbt
- emit_line_lexer.mbt: filter line-pattern candidates by mode via lexmode_name; nullary tokens use declared transition target - emit_lexer.mbt: exclude line_pattern custom_lex fns from char-level lexer collection - emit_lexer_wbtest.mbt: add negative test for payload+line_pattern without #loom.custom_lex
- fixtures/line_pattern_fixture.g.mbt: regenerated golden output - emit_lexer_wbtest.mbt: golden snapshot, positive test for custom_lex+pattern, per-mode filtering test with BlockQuoteLineStart/LineStart mode isolation - regenerate_fixtures.mbt: add line_pattern fixture regeneration - README.md: document --line-lexer flag - emit_line_lexer.mbt: per-mode candidate filtering via lexmode_name, nullary tokens use declared transition target
1 participant
Summary
Implements #561:
#loom.line_patternannotation for line-oriented block-level token production.What
#loom.line_pattern("regex")on token variants: marks the variant as a line-level pattern candidate. Matched against the rest-of-line (pos→newline), not character-by-character.#loom.line_modeon#loom.lexmodeterm variants: declares the mode as line-oriented. Each such mode gets a generated lexer function that tries#loom.line_patterntokens in declaration order.#loom.custom_lex: pattern+payload tokens likeHeadingMarker(Int)use a custom_lex function to construct the token from the match text.Files
New:
loomgen/emit_line_lexer.mbt— line-mode lexer code generatorloomgen/fixtures/line_pattern_fixture.mbt— test fixtureModified:
parse_annotations.mbt— parse/validate#loom.line_pattern+#loom.line_modeemit_lexer.mbt— exclude line_pattern tokens from character-level lexeremit_syntax_kind.mbt— handle None-role line_pattern tokensemit_token_impls.mbt— handle None-role line_pattern tokensmain.mbt—--line-lexerCLI flag, preflight, and writeCLI Usage
The generated
line_lexer.g.mbtcontains per-mode functions (e.g.,lex_line_start) that follow the same signature as thelexer_skeleton.g.mbtstubs.Verification
Closes #561
Summary by CodeRabbit
New Features
--line-lexergeneration for producing line-based lexer output.Bug Fixes
Tests